mmdeploy/docs/en/03-benchmark/benchmark_edge.md

Test on embedded device

Here are the test conclusions of our edge devices. You can directly obtain the results of your own environment with model profiling.

Software and hardware environment

• host OS ubuntu 18.04
• backend SNPE-1.59
• device Mi11 (qcom 888)

mmcls

model	dataset	spatial	fp32 top-1 (%)	snpe gpu hybrid fp32 top-1 (%)	latency (ms)
ShuffleNetV2	ImageNet-1k	224x224	69.55	69.83*	20±7
MobilenetV2	ImageNet-1k	224x224	71.86	72.14*	15±6

tips:

1. The ImageNet-1k dataset is too large to test, only part of the dataset is used (8000/50000)
2. The heating of device will downgrade the frequency, so the time consumption will actually fluctuate. Here are the stable values after running for a period of time. This result is closer to the actual demand.

mmocr detection

model	dataset	spatial	fp32 hmean	snpe gpu hybrid hmean	latency(ms)
PANet	ICDAR2015	1312x736	0.795	0.785 @thr=0.9	3100±100

mmpose

model	dataset	spatial	snpe hybrid AR@IoU=0.50	snpe hybrid AP@IoU=0.50	latency(ms)
pose_hrnet_w32	Animalpose	256x256	0.997	0.989	630±50

tips:

• Test pose_hrnet using AnimalPose's test dataset instead of val dataset.

mmseg

model	dataset	spatial	mIoU	latency(ms)
fcn	Cityscapes	512x1024	71.11	4915±500

tips:

• fcn works fine with 512x1024 size. Cityscapes dataset uses 1024x2048 resolution which causes device to reboot.

Notes

• We needs to manually split the mmdet model into two parts. Because
  • In snpe source code, onnx_to_ir.py can only parse onnx input while ir_to_dlc.py does not support topk operator
  • UDO (User Defined Operator) does not work with snpe-onnx-to-dlc
• mmedit model
  • srcnn requires cubic resize which snpe does not support
  • esrgan converts fine, but loading the model causes the device to reboot
• mmrotate depends on e2cnn and needs to be installed manually its Python3.6 compatible branch