update doc about codebase deployment (#1051)
* update doc about mmclassfication deployment * update install mmdeploy part * correct mmcls deployment commands * reformat supported models * add deployed model specification * update doc about mmdetection deployment * fix according to reviewer comments * fix according to reviewer comments * fix according to reviewer comments * fix according to reviewer comments * fix according to reviewer comments * fix according to reviewer comments * update doc about mmsegmentation deployment * update doc about mmocr deployment * check in cityscapes.png as an input image for converting mmsegmentation model * update mmocr deployment * update mmseg.md * update mmseg.md * update mmocr.md * update sdk model inference for mmocr deployment * update according to reviewer comments * update * update * update mmedit * update mmpose deployment * check in face.png for mmedit deployment * update * fix according to reviewer comments * remove duplicate doc * update docs in english * update codebase documents in chinese * fix according to reviewerss comments * update according reviewer comments * ObjectDetection -> Object Detection * InstanceSegmentation -> Instance Segmentationpull/1120/head
lvhan028
GitHub
parent
20b915dc1f
commit
11409d7c98
Binary file not shown.
|
After Width: | Height: | Size: 301 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 254 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 115 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
|
|
@ -1,20 +1,183 @@
|
|||
# MMClassification Support
|
||||
# MMClassification Deployment
|
||||
|
||||
[MMClassification](https://github.com/open-mmlab/mmclassification) is an open-source image classification toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com) project.
|
||||
- [Installation](#installation)
|
||||
- [Install mmcls](#install-mmcls)
|
||||
- [Install mmdeploy](#install-mmdeploy)
|
||||
- [Convert model](#convert-model)
|
||||
- [Model Specification](#model-specification)
|
||||
- [Model inference](#model-inference)
|
||||
- [Backend model inference](#backend-model-inference)
|
||||
- [SDK model inference](#sdk-model-inference)
|
||||
- [Supported models](#supported-models)
|
||||
|
||||
## MMClassification installation tutorial
|
||||
______________________________________________________________________
|
||||
|
||||
Please refer to [install.md](https://github.com/open-mmlab/mmclassification/blob/master/docs/en/install.md) for installation.
|
||||
[MMClassification](https://github.com/open-mmlab/mmclassification) aka `mmcls` is an open-source image classification toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com) project.
|
||||
|
||||
## List of MMClassification models supported by MMDeploy
|
||||
## Installation
|
||||
|
||||
| Model | TorchScript | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :---------------- | :---------: | :----------: | :------: | :--: | :---: | :------: | :---------------------------------------------------------------------------------------------: |
|
||||
| ResNet | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnet) |
|
||||
| ResNeXt | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnext) |
|
||||
| SE-ResNet | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/seresnet) |
|
||||
| MobileNetV2 | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/mobilenet_v2) |
|
||||
| ShuffleNetV1 | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v1) |
|
||||
| ShuffleNetV2 | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v2) |
|
||||
| VisionTransformer | Y | Y | Y | Y | ? | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/vision_transformer) |
|
||||
| SwinTransformer | Y | Y | Y | N | ? | N | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/swin_transformer) |
|
||||
### Install mmcls
|
||||
|
||||
Please follow this [quick guide](https://github.com/open-mmlab/mmclassification/tree/1.x#installation) to install mmcls.
|
||||
|
||||
### Install mmdeploy
|
||||
|
||||
There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.
|
||||
|
||||
**Method I:** Install precompiled package
|
||||
|
||||
> **TODO**. MMDeploy hasn't released based on dev-1.x branch.
|
||||
|
||||
**Method II:** Build using scripts
|
||||
|
||||
If your target platform is **Ubuntu 18.04 or later version**, we encourage you to run
|
||||
[scripts](../01-how-to-build/build_from_script.md). For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
**Method III:** Build from source
|
||||
|
||||
If neither **I** nor **II** meets your requirements, [building mmdeploy from source](../01-how-to-build/build_from_source.md) is the last option.
|
||||
|
||||
## Convert model
|
||||
|
||||
You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) to convert mmcls models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
The command below shows an example about converting `resnet18` model to onnx model that can be inferred by ONNX Runtime.
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
|
||||
# download resnet18 model from mmcls model zoo
|
||||
mim download mmcls --config resnet18_8xb32_in1k --dest .
|
||||
|
||||
# convert mmcls model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmcls/classification_onnxruntime_dynamic.py \
|
||||
resnet18_8xb32_in1k.py \
|
||||
resnet18_8xb32_in1k_20210831-fbbb1da6.pth \
|
||||
tests/data/tiger.jpeg \
|
||||
--work-dir mmdeploy_models/mmcls/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmcls) of all supported backends for mmclassification. The config filename pattern is:
|
||||
|
||||
```
|
||||
classification_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml and etc.
|
||||
- **{precision}:** fp16, int8. When it's empty, it means fp32
|
||||
- **{static | dynamic}:** static shape or dynamic shape
|
||||
- **{shape}:** input shape or shape range of a model
|
||||
|
||||
Therefore, in the above example, you can also convert `resnet18` to other backend models by changing the deployment config file `classification_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmcls), e.g., converting to tensorrt-fp16 model by `classification_tensorrt-fp16_dynamic-224x224-224x224.py`.
|
||||
|
||||
```{tip}
|
||||
When converting mmcls models to tensorrt models, --device should be set to "cuda"
|
||||
```
|
||||
|
||||
## Model Specification
|
||||
|
||||
Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.
|
||||
|
||||
The converted model locates in the working directory like `mmdeploy_models/mmcls/ort` in the previous example. It includes:
|
||||
|
||||
```
|
||||
mmdeploy_models/mmcls/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
in which,
|
||||
|
||||
- **end2end.onnx**: backend model which can be inferred by ONNX Runtime
|
||||
- \***.json**: the necessary information for mmdeploy SDK
|
||||
|
||||
The whole package **mmdeploy_models/mmcls/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.
|
||||
|
||||
## Model inference
|
||||
|
||||
### Backend model inference
|
||||
|
||||
Take the previous converted `end2end.onnx` model as an example, you can use the following code to inference the model.
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmcls/classification_onnxruntime_dynamic.py'
|
||||
model_cfg = './resnet18_8xb32_in1k.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmcls/ort/end2end.onnx']
|
||||
image = 'tests/data/tiger.jpeg'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_classification.png')
|
||||
```
|
||||
|
||||
### SDK model inference
|
||||
|
||||
You can also perform SDK model inference like following,
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Classifier
|
||||
import cv2
|
||||
|
||||
img = cv2.imread('tests/data/tiger.jpeg')
|
||||
|
||||
# create a classifier
|
||||
classifier = Classifier(model_path='./mmdeploy_models/mmcls/ort', device_name='cpu', device_id=0)
|
||||
# perform inference
|
||||
result = classifier(img)
|
||||
# show inference result
|
||||
for label_id, score in result:
|
||||
print(label_id, score)
|
||||
```
|
||||
|
||||
Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo).
|
||||
|
||||
## Supported models
|
||||
|
||||
| Model | TorchScript | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :--------------------------------------------------------------------------------------------------------- | :---------: | :----------: | :------: | :--: | :---: | :------: |
|
||||
| [ResNet](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnet) | Y | Y | Y | Y | Y | Y |
|
||||
| [ResNeXt](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnext) | Y | Y | Y | Y | Y | Y |
|
||||
| [SE-ResNet](https://github.com/open-mmlab/mmclassification/tree/master/configs/seresnet) | Y | Y | Y | Y | Y | Y |
|
||||
| [MobileNetV2](https://github.com/open-mmlab/mmclassification/tree/master/configs/mobilenet_v2) | Y | Y | Y | Y | Y | Y |
|
||||
| [ShuffleNetV1](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v1) | Y | Y | Y | Y | Y | Y |
|
||||
| [ShuffleNetV2](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v2) | Y | Y | Y | Y | Y | Y |
|
||||
| [VisionTransformer](https://github.com/open-mmlab/mmclassification/tree/master/configs/vision_transformer) | Y | Y | Y | Y | ? | Y |
|
||||
| [SwinTransformer](https://github.com/open-mmlab/mmclassification/tree/master/configs/swin_transformer) | Y | Y | Y | N | ? | N |
|
||||
|
|
|
|||
|
|
@ -1,30 +1,210 @@
|
|||
# MMDetection Support
|
||||
# MMDetection Deployment
|
||||
|
||||
MMDetection is an open source object detection toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
- [Installation](#installation)
|
||||
- [Install mmdet](#install-mmdet)
|
||||
- [Install mmdeploy](#install-mmdeploy)
|
||||
- [Convert model](#convert-model)
|
||||
- [Model specification](#model-specification)
|
||||
- [Model inference](#model-inference)
|
||||
- [Backend model inference](#backend-model-inference)
|
||||
- [SDK model inference](#sdk-model-inference)
|
||||
- [Supported models](#supported-models)
|
||||
|
||||
## MMDetection installation tutorial
|
||||
______________________________________________________________________
|
||||
|
||||
Please refer to [get_started.md](https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md) for installation.
|
||||
[MMDetection](https://github.com/open-mmlab/mmdetection) aka `mmdet` is an open source object detection toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
## List of MMDetection models supported by MMDeploy
|
||||
## Installation
|
||||
|
||||
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----------------: | :------------------: | :---------: | :------: | :--: | :---: | :------: | :----------------------------------------------------------------------------------: |
|
||||
| ATSS | ObjectDetection | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/atss) |
|
||||
| FCOS | ObjectDetection | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/fcos) |
|
||||
| FoveaBox | ObjectDetection | Y | N | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/foveabox) |
|
||||
| FSAF | ObjectDetection | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/fsaf) |
|
||||
| RetinaNet | ObjectDetection | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/retinanet) |
|
||||
| SSD | ObjectDetection | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/ssd) |
|
||||
| VFNet | ObjectDetection | N | N | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/vfnet) |
|
||||
| YOLOv3 | ObjectDetection | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/yolo) |
|
||||
| YOLOX | ObjectDetection | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/yolox) |
|
||||
| Cascade R-CNN | ObjectDetection | Y | Y | N | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) |
|
||||
| Faster R-CNN | ObjectDetection | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) |
|
||||
| Faster R-CNN + DCN | ObjectDetection | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) |
|
||||
| GFL | ObjectDetection | Y | Y | N | ? | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/gfl) |
|
||||
| RepPoints | ObjectDetection | N | Y | N | ? | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/reppoints) |
|
||||
| DETR | ObjectDetection | Y | Y | N | ? | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/detr) |
|
||||
| Cascade Mask R-CNN | InstanceSegmentation | Y | N | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) |
|
||||
| Mask R-CNN | InstanceSegmentation | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn) |
|
||||
| Swin Transformer | InstanceSegmentation | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/swin) |
|
||||
### Install mmdet
|
||||
|
||||
Please follow the [installation guide](https://mmdetection.readthedocs.io/en/3.x/get_started.html) to install mmdet.
|
||||
|
||||
### Install mmdeploy
|
||||
|
||||
There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.
|
||||
|
||||
**Method I:** Install precompiled package
|
||||
|
||||
> **TODO**. MMDeploy hasn't released based on dev-1.x branch.
|
||||
|
||||
**Method II:** Build using scripts
|
||||
|
||||
If your target platform is **Ubuntu 18.04 or later version**, we encourage you to run
|
||||
[scripts](../01-how-to-build/build_from_script.md). For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
**Method III:** Build from source
|
||||
|
||||
If neither **I** nor **II** meets your requirements, [building mmdeploy from source](../01-how-to-build/build_from_source.md) is the last option.
|
||||
|
||||
## Convert model
|
||||
|
||||
You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) to convert mmdet models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
The command below shows an example about converting `Faster R-CNN` model to onnx model that can be inferred by ONNX Runtime.
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download faster r-cnn model from mmdet model zoo
|
||||
mim download mmdet --config faster-rcnn_r50_fpn_1x_coco --dest .
|
||||
# convert mmdet model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmdet/detection/detection_onnxruntime_dynamic.py \
|
||||
faster-rcnn_r50_fpn_1x_coco.py \
|
||||
faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
|
||||
demo/resources/det.jpg \
|
||||
--work-dir mmdeploy_models/mmdet/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmdet) of all supported backends for mmdetection, under which the config file path follows the pattern:
|
||||
|
||||
```
|
||||
{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
- **{task}:** task in mmdetection.
|
||||
|
||||
There are two of them. One is `detection` and the other is `instance-seg`, indicating instance segmentation.
|
||||
|
||||
mmdet models like `RetinaNet`, `Faster R-CNN` and `DETR` and so on belongs to `detection` task. While `Mask R-CNN` is one of `instance-seg` models. You can find more of them in chapter [Supported models](#supported-models).
|
||||
|
||||
**DO REMEMBER TO USE** `detection/detection_*.py` deployment config file when trying to convert detection models and use `instance-seg/instance-seg_*.py` to deploy instance segmentation models.
|
||||
|
||||
- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
|
||||
|
||||
- **{precision}:** fp16, int8. When it's empty, it means fp32
|
||||
|
||||
- **{static | dynamic}:** static shape or dynamic shape
|
||||
|
||||
- **{shape}:** input shape or shape range of a model
|
||||
|
||||
Therefore, in the above example, you can also convert `faster r-cnn` to other backend models by changing the deployment config file `detection_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmdet/detection), e.g., converting to tensorrt-fp16 model by `detection_tensorrt-fp16_dynamic-320x320-1344x1344.py`.
|
||||
|
||||
```{tip}
|
||||
When converting mmdet models to tensorrt models, --device should be set to "cuda"
|
||||
```
|
||||
|
||||
## Model specification
|
||||
|
||||
Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.
|
||||
|
||||
The converted model locates in the working directory like `mmdeploy_models/mmdet/ort` in the previous example. It includes:
|
||||
|
||||
```
|
||||
mmdeploy_models/mmdet/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
in which,
|
||||
|
||||
- **end2end.onnx**: backend model which can be inferred by ONNX Runtime
|
||||
- \***.json**: the necessary information for mmdeploy SDK
|
||||
|
||||
The whole package **mmdeploy_models/mmdet/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.
|
||||
|
||||
## Model inference
|
||||
|
||||
### Backend model inference
|
||||
|
||||
Take the previous converted `end2end.onnx` model as an example, you can use the following code to inference the model and visualize the results.
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmdet/detection/detection_onnxruntime_dynamic.py'
|
||||
model_cfg = './faster-rcnn_r50_fpn_1x_coco.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmdet/ort/end2end.onnx']
|
||||
image = './demo/resources/det.jpg'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_detection.png')
|
||||
```
|
||||
|
||||
### SDK model inference
|
||||
|
||||
You can also perform SDK model inference like following,
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Detector
|
||||
import cv2
|
||||
|
||||
img = cv2.imread('./demo/resources/det.jpg')
|
||||
|
||||
# create a detector
|
||||
detector = Detector(model_path='./mmdeploy_models/mmdet/ort', device_name='cpu', device_id=0)
|
||||
# perform inference
|
||||
bboxes, labels, masks = detector(img)
|
||||
|
||||
# visualize inference result
|
||||
indices = [i for i in range(len(bboxes))]
|
||||
for index, bbox, label_id in zip(indices, bboxes, labels):
|
||||
[left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
|
||||
if score < 0.3:
|
||||
continue
|
||||
|
||||
cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))
|
||||
|
||||
cv2.imwrite('output_detection.png', img)
|
||||
```
|
||||
|
||||
Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo).
|
||||
|
||||
## Supported models
|
||||
|
||||
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :----------------------------------------------------------------------------------------------: | :-------------------: | :---------: | :------: | :--: | :---: | :------: |
|
||||
| [ATSS](https://github.com/open-mmlab/mmdetection/tree/master/configs/atss) | Object Detection | Y | Y | N | N | Y |
|
||||
| [FCOS](https://github.com/open-mmlab/mmdetection/tree/master/configs/fcos) | Object Detection | Y | Y | Y | N | Y |
|
||||
| [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/master/configs/foveabox) | Object Detection | Y | N | N | N | Y |
|
||||
| [FSAF](https://github.com/open-mmlab/mmdetection/tree/master/configs/fsaf) | Object Detection | Y | Y | Y | Y | Y |
|
||||
| [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/master/configs/retinanet) | Object Detection | Y | Y | Y | Y | Y |
|
||||
| [SSD](https://github.com/open-mmlab/mmdetection/tree/master/configs/ssd) | Object Detection | Y | Y | Y | N | Y |
|
||||
| [VFNet](https://github.com/open-mmlab/mmdetection/tree/master/configs/vfnet) | Object Detection | N | N | N | N | Y |
|
||||
| [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/master/configs/yolo) | Object Detection | Y | Y | Y | N | Y |
|
||||
| [YOLOX](https://github.com/open-mmlab/mmdetection/tree/master/configs/yolox) | Object Detection | Y | Y | Y | N | Y |
|
||||
| [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) | Object Detection | Y | Y | N | Y | Y |
|
||||
| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) | Object Detection | Y | Y | Y | Y | Y |
|
||||
| [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) | Object Detection | Y | Y | Y | Y | Y |
|
||||
| [GFL](https://github.com/open-mmlab/mmdetection/tree/master/configs/gfl) | Object Detection | Y | Y | N | ? | Y |
|
||||
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/master/configs/reppoints) | Object Detection | N | Y | N | ? | Y |
|
||||
| [DETR](https://github.com/open-mmlab/mmdetection/tree/master/configs/detr) | Object Detection | Y | Y | N | ? | Y |
|
||||
| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) | Instance Segmentation | Y | N | N | N | Y |
|
||||
| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn) | Instance Segmentation | Y | Y | N | N | Y |
|
||||
| [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/master/configs/swin) | Instance Segmentation | Y | Y | N | N | N |
|
||||
|
|
|
|||
|
|
@ -1,42 +1,33 @@
|
|||
# MMDetection3d Support
|
||||
# MMDetection3d Deployment
|
||||
|
||||
MMDetection3d is a next-generation platform for general 3D object detection. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
- [Installation](#installation)
|
||||
- [Install mmdet3d](#install-mmdet3d)
|
||||
- [Install mmdeploy](#install-mmdeploy)
|
||||
- [Convert model](#convert-model)
|
||||
- [Model specification](#model-specification)
|
||||
- [Model inference](#model-inference)
|
||||
- [Backend model inference](#backend-model-inference)
|
||||
- [SDK model inference](#sdk-model-inference)
|
||||
- [Supported models](#supported-models)
|
||||
|
||||
## MMDetection3d installation tutorial
|
||||
______________________________________________________________________
|
||||
|
||||
Please refer to [getting_started.md](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/getting_started.md) for installation.
|
||||
[MMDetection3d](https://github.com/open-mmlab/mmdetection3d) aka `mmdet3d` is an open source object detection toolbox based on PyTorch, towards the next-generation platform for general 3D detection. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
## Example
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
export MODEL_PATH=https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth
|
||||
### Install mmdet3d
|
||||
|
||||
python tools/deploy.py \
|
||||
configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic.py \
|
||||
${MMDET3D_DIR}/configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py \
|
||||
${MODEL_PATH} \
|
||||
${MMDET3D_DIR}/demo/data/kitti/kitti_000008.bin \
|
||||
--work-dir \
|
||||
work_dir \
|
||||
--show \
|
||||
--device \
|
||||
cuda:0
|
||||
```
|
||||
### Install mmdeploy
|
||||
|
||||
## List of MMDetection3d models supported by MMDeploy
|
||||
## Convert model
|
||||
|
||||
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----------: | :------------: | :---------: | :------: | :--: | :---: | :------: | :------------------------------------------------------------------------------------: |
|
||||
| PointPillars | VoxelDetection | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/pointpillars) |
|
||||
## Model specification
|
||||
|
||||
## Reminder
|
||||
## Model inference
|
||||
|
||||
Voxel detection onnx model excludes model.voxelize layer and model post process, and you can use python api to call these func.
|
||||
### Backend model inference
|
||||
|
||||
Example:
|
||||
### SDK model inference
|
||||
|
||||
```python
|
||||
from mmdeploy.codebase.mmdet3d.deploy import VoxelDetectionModel
|
||||
VoxelDetectionModel.voxelize(...)
|
||||
VoxelDetectionModel.post_process(...)
|
||||
```
|
||||
## Supported models
|
||||
|
|
|
|||
|
|
@ -1,20 +1,195 @@
|
|||
# MMEditing Support
|
||||
# MMEditing Deployment
|
||||
|
||||
[MMEditing](https://github.com/open-mmlab/mmediting) is an open-source image and video editing toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
- [Installation](#installation)
|
||||
- [Install mmedit](#install-mmedit)
|
||||
- [Install mmdeploy](#install-mmdeploy)
|
||||
- [Convert model](#convert-model)
|
||||
- [Convert super resolution model](#convert-super-resolution-model)
|
||||
- [Model specification](#model-specification)
|
||||
- [Model inference](#model-inference)
|
||||
- [Backend model inference](#backend-model-inference)
|
||||
- [SDK model inference](#sdk-model-inference)
|
||||
- [Supported models](#supported-models)
|
||||
|
||||
## MMEditing installation tutorial
|
||||
______________________________________________________________________
|
||||
|
||||
Please refer to [official installation guide](https://mmediting.readthedocs.io/en/latest/install.html#installation) to install the codebase.
|
||||
[MMEditing](https://github.com/open-mmlab/mmediting/tree/1.x) aka `mmedit` is an open-source image and video editing toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
## MMEditing models support
|
||||
## Installation
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :---------- | :--------------- | :----------: | :------: | :--: | :---: | :------: | :--------------------------------------------------------------------------------------------: |
|
||||
| SRCNN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srcnn) |
|
||||
| ESRGAN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/esrgan) |
|
||||
| ESRGAN-PSNR | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/esrgan) |
|
||||
| SRGAN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srresnet_srgan) |
|
||||
| SRResNet | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srresnet_srgan) |
|
||||
| Real-ESRGAN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/real_esrgan) |
|
||||
| EDSR | super-resolution | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/edsr) |
|
||||
| RDN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/rdn) |
|
||||
### Install mmedit
|
||||
|
||||
Please follow the [installation guide](https://github.com/open-mmlab/mmediting/tree/1.x#installation) to install mmedit.
|
||||
|
||||
### Install mmdeploy
|
||||
|
||||
There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.
|
||||
|
||||
**Method I:** Install precompiled package
|
||||
|
||||
> **TODO**. MMDeploy hasn't released based on dev-1.x branch.
|
||||
|
||||
**Method II:** Build using scripts
|
||||
|
||||
If your target platform is **Ubuntu 18.04 or later version**, we encourage you to run
|
||||
[scripts](../01-how-to-build/build_from_script.md). For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
**Method III:** Build from source
|
||||
|
||||
If neither **I** nor **II** meets your requirements, [building mmdeploy from source](../01-how-to-build/build_from_source.md) is the last option.
|
||||
|
||||
## Convert model
|
||||
|
||||
You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) to convert mmedit models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
When using `tools/deploy.py`, it is crucial to specify the correct deployment config. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmedit) of all supported backends for mmedit, under which the config file path follows the pattern:
|
||||
|
||||
```
|
||||
{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
- **{task}:** task in mmedit.
|
||||
|
||||
MMDeploy supports models of one task in mmedit, i.e., `super resolution`. Please refer to chapter [supported models](#supported-models) for task-model organization.
|
||||
|
||||
**DO REMEMBER TO USE** the corresponding deployment config file when trying to convert models of different tasks.
|
||||
|
||||
- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
|
||||
|
||||
- **{precision}:** fp16, int8. When it's empty, it means fp32
|
||||
|
||||
- **{static | dynamic}:** static shape or dynamic shape
|
||||
|
||||
- **{shape}:** input shape or shape range of a model
|
||||
|
||||
### Convert super resolution model
|
||||
|
||||
The command below shows an example about converting `ESRGAN` model to onnx model that can be inferred by ONNX Runtime.
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download esrgan model from mmedit model zoo
|
||||
mim download mmedit --config esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k --dest .
|
||||
# convert esrgan model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py \
|
||||
esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py \
|
||||
esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth \
|
||||
demo/resources/face.png \
|
||||
--work-dir mmdeploy_models/mmedit/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
You can also convert the above model to other backend models by changing the deployment config file `*_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmedit), e.g., converting to tensorrt model by `super-resolution/super-resolution_tensorrt-_dynamic-32x32-512x512.py`.
|
||||
|
||||
```{tip}
|
||||
When converting mmedit models to tensorrt models, --device should be set to "cuda"
|
||||
```
|
||||
|
||||
## Model specification
|
||||
|
||||
Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.
|
||||
|
||||
The converted model locates in the working directory like `mmdeploy_models/mmedit/ort` in the previous example. It includes:
|
||||
|
||||
```
|
||||
mmdeploy_models/mmedit/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
in which,
|
||||
|
||||
- **end2end.onnx**: backend model which can be inferred by ONNX Runtime
|
||||
- \***.json**: the necessary information for mmdeploy SDK
|
||||
|
||||
The whole package **mmdeploy_models/mmedit/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.
|
||||
|
||||
## Model inference
|
||||
|
||||
### Backend model inference
|
||||
|
||||
Take the previous converted `end2end.onnx` model as an example, you can use the following code to inference the model and visualize the results.
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py'
|
||||
model_cfg = 'esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmedit/ort/end2end.onnx']
|
||||
image = './demo/resources/face.png'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_restorer.bmp')
|
||||
```
|
||||
|
||||
### SDK model inference
|
||||
|
||||
You can also perform SDK model inference like following,
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Restorer
|
||||
import cv2
|
||||
|
||||
img = cv2.imread('./demo/resources/face.png')
|
||||
|
||||
# create a classifier
|
||||
restorer = Restorer(model_path='./mmdeploy_models/mmedit/ort', device_name='cpu', device_id=0)
|
||||
# perform inference
|
||||
result = restorer(img)
|
||||
|
||||
# visualize inference result
|
||||
# convert to BGR
|
||||
result = result[..., ::-1]
|
||||
cv2.imwrite('output_restorer.bmp', result)
|
||||
```
|
||||
|
||||
Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo).
|
||||
|
||||
## Supported models
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :----------------------------------------------------------------------------------------------- | :--------------- | :----------: | :------: | :--: | :---: | :------: |
|
||||
| [SRCNN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srcnn) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [ESRGAN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/esrgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [ESRGAN-PSNR](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/esrgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [SRGAN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srresnet_srgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [SRResNet](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srresnet_srgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [Real-ESRGAN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/real_esrgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [EDSR](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/edsr) | super-resolution | Y | Y | Y | N | Y |
|
||||
| [RDN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/rdn) | super-resolution | Y | Y | Y | Y | Y |
|
||||
|
|
|
|||
|
|
@ -1,163 +1,240 @@
|
|||
# MMOCR Support
|
||||
# MMOCR Deployment
|
||||
|
||||
MMOCR is an open-source toolbox based on PyTorch and mmdetection for text detection, text recognition, and the corresponding downstream tasks including key information extraction. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
- [Installation](#installation)
|
||||
- [Install mmocr](#install-mmocr)
|
||||
- [Install mmdeploy](#install-mmdeploy)
|
||||
- [Convert model](#convert-model)
|
||||
- [Convert text detection model](#convert-text-detection-model)
|
||||
- [Convert text recognition model](#convert-text-recognition-model)
|
||||
- [Model specification](#model-specification)
|
||||
- [Model Inference](#model-inference)
|
||||
- [Backend model inference](#backend-model-inference)
|
||||
- [SDK model inference](#sdk-model-inference)
|
||||
- [Supported models](#supported-models)
|
||||
|
||||
## MMOCR installation tutorial
|
||||
______________________________________________________________________
|
||||
|
||||
Please refer to [install.md](https://mmocr.readthedocs.io/en/latest/install.html) for installation.
|
||||
[MMOCR](https://github.com/open-mmlab/mmocr/tree/1.x) aka `mmocr` is an open-source toolbox based on PyTorch and mmdetection for text detection, text recognition, and the corresponding downstream tasks including key information extraction. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
## List of MMOCR models supported by MMDeploy
|
||||
## Installation
|
||||
|
||||
| Model | Task | TorchScript | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----- | :--------------- | :---------: | :---------: | :------: | :--: | :---: | :------: | :-----------------------------------------------------------------------------: |
|
||||
| DBNet | text-detection | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/dbnet) |
|
||||
| PSENet | text-detection | Y | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/psenet) |
|
||||
| PANet | text-detection | Y | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/panet) |
|
||||
| CRNN | text-recognition | Y | Y | Y | Y | Y | N | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/crnn) |
|
||||
| SAR | text-recognition | N | Y | N | N | N | N | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/sar) |
|
||||
| SATRN | text-recognition | Y | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/satrn) |
|
||||
### Install mmocr
|
||||
|
||||
## Reminder
|
||||
Please follow the [installation guide](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html) to install mmocr.
|
||||
|
||||
Note that ncnn, pplnn, and OpenVINO only support the configs of DBNet18 for DBNet.
|
||||
### Install mmdeploy
|
||||
|
||||
For the PANet with the [checkpoint](https://download.openmmlab.com/mmocr/textdet/panet/panet_r18_fpem_ffm_sbn_600e_icdar2015_20210219-42dbe46a.pth) pretrained on ICDAR dataset, if you want to convert the model to TensorRT with 16 bits float point, please try the following script.
|
||||
There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.
|
||||
|
||||
```python
|
||||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
from typing import Sequence
|
||||
**Method I:** Install precompiled package
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
> **TODO**. MMDeploy hasn't released based on dev-1.x branch.
|
||||
|
||||
from mmdeploy.core import FUNCTION_REWRITER
|
||||
from mmdeploy.utils.constants import Backend
|
||||
**Method II:** Build using scripts
|
||||
|
||||
FACTOR = 32
|
||||
ENABLE = False
|
||||
CHANNEL_THRESH = 400
|
||||
If your target platform is **Ubuntu 18.04 or later version**, we encourage you to run
|
||||
[scripts](../01-how-to-build/build_from_script.md). For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
@FUNCTION_REWRITER.register_rewriter(
|
||||
func_name='mmocr.models.textdet.necks.FPEM_FFM.forward',
|
||||
backend=Backend.TENSORRT.value)
|
||||
def fpem_ffm__forward__trt(ctx, self, x: Sequence[torch.Tensor], *args,
|
||||
**kwargs) -> Sequence[torch.Tensor]:
|
||||
"""Rewrite `forward` of FPEM_FFM for tensorrt backend.
|
||||
**Method III:** Build from source
|
||||
|
||||
Rewrite this function avoid overflow for tensorrt-fp16 with the checkpoint
|
||||
`https://download.openmmlab.com/mmocr/textdet/panet/panet_r18_fpem_ffm
|
||||
_sbn_600e_icdar2015_20210219-42dbe46a.pth`
|
||||
If neither **I** nor **II** meets your requirements, [building mmdeploy from source](../01-how-to-build/build_from_source.md) is the last option.
|
||||
|
||||
Args:
|
||||
ctx (ContextCaller): The context with additional information.
|
||||
self: The instance of the class FPEM_FFM.
|
||||
x (List[Tensor]): A list of feature maps of shape (N, C, H, W).
|
||||
## Convert model
|
||||
|
||||
Returns:
|
||||
outs (List[Tensor]): A list of feature maps of shape (N, C, H, W).
|
||||
"""
|
||||
c2, c3, c4, c5 = x
|
||||
# reduce channel
|
||||
c2 = self.reduce_conv_c2(c2)
|
||||
c3 = self.reduce_conv_c3(c3)
|
||||
c4 = self.reduce_conv_c4(c4)
|
||||
You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) to convert mmocr models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
if ENABLE:
|
||||
bn_w = self.reduce_conv_c5[1].weight / torch.sqrt(
|
||||
self.reduce_conv_c5[1].running_var + self.reduce_conv_c5[1].eps)
|
||||
bn_b = self.reduce_conv_c5[
|
||||
1].bias - self.reduce_conv_c5[1].running_mean * bn_w
|
||||
bn_w = bn_w.reshape(1, -1, 1, 1).repeat(1, 1, c5.size(2), c5.size(3))
|
||||
bn_b = bn_b.reshape(1, -1, 1, 1).repeat(1, 1, c5.size(2), c5.size(3))
|
||||
conv_b = self.reduce_conv_c5[0].bias.reshape(1, -1, 1, 1).repeat(
|
||||
1, 1, c5.size(2), c5.size(3))
|
||||
c5 = FACTOR * (self.reduce_conv_c5[:-1](c5)) - (FACTOR - 1) * (
|
||||
bn_w * conv_b + bn_b)
|
||||
c5 = self.reduce_conv_c5[-1](c5)
|
||||
else:
|
||||
c5 = self.reduce_conv_c5(c5)
|
||||
|
||||
# FPEM
|
||||
for i, fpem in enumerate(self.fpems):
|
||||
c2, c3, c4, c5 = fpem(c2, c3, c4, c5)
|
||||
if i == 0:
|
||||
c2_ffm = c2
|
||||
c3_ffm = c3
|
||||
c4_ffm = c4
|
||||
c5_ffm = c5
|
||||
else:
|
||||
c2_ffm += c2
|
||||
c3_ffm += c3
|
||||
c4_ffm += c4
|
||||
c5_ffm += c5
|
||||
|
||||
# FFM
|
||||
c5 = F.interpolate(
|
||||
c5_ffm,
|
||||
c2_ffm.size()[-2:],
|
||||
mode='bilinear',
|
||||
align_corners=self.align_corners)
|
||||
c4 = F.interpolate(
|
||||
c4_ffm,
|
||||
c2_ffm.size()[-2:],
|
||||
mode='bilinear',
|
||||
align_corners=self.align_corners)
|
||||
c3 = F.interpolate(
|
||||
c3_ffm,
|
||||
c2_ffm.size()[-2:],
|
||||
mode='bilinear',
|
||||
align_corners=self.align_corners)
|
||||
outs = [c2_ffm, c3, c4, c5]
|
||||
return tuple(outs)
|
||||
|
||||
|
||||
@FUNCTION_REWRITER.register_rewriter(
|
||||
func_name='mmdet.models.backbones.resnet.BasicBlock.forward',
|
||||
backend=Backend.TENSORRT.value)
|
||||
def basic_block__forward__trt(ctx, self, x: torch.Tensor) -> torch.Tensor:
|
||||
"""Rewrite `forward` of BasicBlock for tensorrt backend.
|
||||
|
||||
Rewrite this function avoid overflow for tensorrt-fp16 with the checkpoint
|
||||
`https://download.openmmlab.com/mmocr/textdet/panet/panet_r18_fpem_ffm
|
||||
_sbn_600e_icdar2015_20210219-42dbe46a.pth`
|
||||
|
||||
Args:
|
||||
ctx (ContextCaller): The context with additional information.
|
||||
self: The instance of the class FPEM_FFM.
|
||||
x (Tensor): The input tensor of shape (N, C, H, W).
|
||||
|
||||
Returns:
|
||||
outs (Tensor): The output tensor of shape (N, C, H, W).
|
||||
"""
|
||||
if self.conv1.in_channels < CHANNEL_THRESH:
|
||||
return ctx.origin_func(self, x)
|
||||
|
||||
identity = x
|
||||
|
||||
out = self.conv1(x)
|
||||
out = self.norm1(out)
|
||||
out = self.relu(out)
|
||||
|
||||
out = self.conv2(out)
|
||||
|
||||
if torch.abs(self.norm2(out)).max() < 65504:
|
||||
out = self.norm2(out)
|
||||
out += identity
|
||||
out = self.relu(out)
|
||||
return out
|
||||
else:
|
||||
global ENABLE
|
||||
ENABLE = True
|
||||
# the output of the last bn layer exceeds the range of fp16
|
||||
w1 = self.norm2.weight / torch.sqrt(self.norm2.running_var +
|
||||
self.norm2.eps)
|
||||
bias = self.norm2.bias - self.norm2.running_mean * w1
|
||||
w1 = w1.reshape(1, -1, 1, 1).repeat(1, 1, out.size(2), out.size(3))
|
||||
bias = bias.reshape(1, -1, 1, 1).repeat(1, 1, out.size(2),
|
||||
out.size(3)) + identity
|
||||
out = self.relu(w1 * (out / FACTOR) + bias / FACTOR)
|
||||
|
||||
return out
|
||||
When using `tools/deploy.py`, it is crucial to specify the correct deployment config. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmocr) of all supported backends for mmocr, under which the config file path follows the pattern:
|
||||
|
||||
```
|
||||
{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
- **{task}:** task in mmocr.
|
||||
|
||||
MMDeploy supports models of two tasks of mmocr, one is `text detection` and the other is `text-recogntion`.
|
||||
|
||||
**DO REMEMBER TO USE** the corresponding deployment config file when trying to convert models of different tasks.
|
||||
|
||||
- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
|
||||
|
||||
- **{precision}:** fp16, int8. When it's empty, it means fp32
|
||||
|
||||
- **{static | dynamic}:** static shape or dynamic shape
|
||||
|
||||
- **{shape}:** input shape or shape range of a model
|
||||
|
||||
In the next two chapters, we will task `dbnet` model from `text detection` task and `crnn` model from `text recognition` task respectively as examples, showing how to convert them to onnx model that can be inferred by ONNX Runtime.
|
||||
|
||||
### Convert text detection model
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download dbnet model from mmocr model zoo
|
||||
mim download mmocr --config dbnet_resnet18_fpnc_1200e_icdar2015 --dest .
|
||||
# convert mmocr model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py \
|
||||
dbnet_resnet18_fpnc_1200e_icdar2015.py \
|
||||
dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth \
|
||||
demo/resources/text_det.jpg \
|
||||
--work-dir mmdeploy_models/mmocr/dbnet/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
### Convert text recognition model
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download crnn model from mmocr model zoo
|
||||
mim download mmocr --config crnn_mini-vgg_5e_mj --dest .
|
||||
# convert mmocr model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmocr/text-recognition/text-recognition_onnxruntime_dynamic.py \
|
||||
crnn_mini-vgg_5e_mj.py \
|
||||
crnn_mini-vgg_5e_mj_20220826_224120-8afbedbb.pth \
|
||||
demo/resources/text_recog.jpg \
|
||||
--work-dir mmdeploy_models/mmocr/crnn/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
You can also convert the above models to other backend models by changing the deployment config file `*_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmocr), e.g., converting `dbnet` to tensorrt-fp32 model by `text-detection/text-detection_tensorrt-_dynamic-320x320-2240x2240.py`.
|
||||
|
||||
```{tip}
|
||||
When converting mmocr models to tensorrt models, --device should be set to "cuda"
|
||||
```
|
||||
|
||||
## Model specification
|
||||
|
||||
Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.
|
||||
|
||||
The converted model locates in the working directory like `mmdeploy_models/mmocr/dbnet/ort` in the previous example. It includes:
|
||||
|
||||
```
|
||||
mmdeploy_models/mmocr/dbnet/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
in which,
|
||||
|
||||
- **end2end.onnx**: backend model which can be inferred by ONNX Runtime
|
||||
- \***.json**: the necessary information for mmdeploy SDK
|
||||
|
||||
The whole package **mmdeploy_models/mmocr/dbnet/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.
|
||||
|
||||
## Model Inference
|
||||
|
||||
### Backend model inference
|
||||
|
||||
Take the previous converted `end2end.onnx` mode of `dbnet` as an example, you can use the following code to inference the model and visualize the results.
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py'
|
||||
model_cfg = 'dbnet_resnet18_fpnc_1200e_icdar2015.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmocr/dbnet/ort/end2end.onnx']
|
||||
image = './demo/resources/text_det.jpg'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_ocr.png')
|
||||
```
|
||||
|
||||
**Tip**:
|
||||
|
||||
Map 'deploy_cfg', 'model_cfg', 'backend_model' and 'image' to corresponding arguments in chapter [convert text recognition model](#convert-text-recognition-model), you will get the ONNX Runtime inference results of `crnn` onnx model.
|
||||
|
||||
### SDK model inference
|
||||
|
||||
Given the above SDK models of `dbnet` and `crnn`, you can also perform SDK model inference like following,
|
||||
|
||||
#### Text detection SDK model inference
|
||||
|
||||
```python
|
||||
import cv2
|
||||
from mmdeploy_python import TextDetector
|
||||
|
||||
img = cv2.imread('demo/resources/text_det.jpg')
|
||||
# create text detector
|
||||
detector = TextDetector(
|
||||
model_path='mmdeploy_models/mmocr/dbnet/ort',
|
||||
device_name='cpu',
|
||||
device_id=0)
|
||||
# do model inference
|
||||
bboxes = detector(img)
|
||||
# draw detected bbox into the input image
|
||||
if len(bboxes) > 0:
|
||||
pts = ((bboxes[:, 0:8] + 0.5).reshape(len(bboxes), -1,
|
||||
2).astype(int))
|
||||
cv2.polylines(img, pts, True, (0, 255, 0), 2)
|
||||
cv2.imwrite('output_ocr.png', img)
|
||||
```
|
||||
|
||||
#### Text Recognition SDK model inference
|
||||
|
||||
```python
|
||||
import cv2
|
||||
from mmdeploy_python import TextRecognizer
|
||||
|
||||
img = cv2.imread('demo/resources/text_recog.jpg')
|
||||
# create text recognizer
|
||||
recognizer = TextRecognizer(
|
||||
model_path='mmdeploy_models/mmocr/crnn/ort',
|
||||
device_name='cpu',
|
||||
device_id=0
|
||||
)
|
||||
# do model inference
|
||||
texts = recognizer(img)
|
||||
# print the result
|
||||
print(texts)
|
||||
```
|
||||
|
||||
Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo).
|
||||
|
||||
## Supported models
|
||||
|
||||
| Model | Task | TorchScript | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :----------------------------------------------------------------------------- | :--------------- | :---------: | :---------: | :------: | :--: | :---: | :------: |
|
||||
| [DBNet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/dbnet) | text-detection | Y | Y | Y | Y | Y | Y |
|
||||
| [PSENet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/psenet) | text-detection | Y | Y | Y | Y | N | Y |
|
||||
| [PANet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/panet) | text-detection | Y | Y | Y | Y | N | Y |
|
||||
| [CRNN](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/crnn) | text-recognition | Y | Y | Y | Y | Y | N |
|
||||
| [SAR](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/sar) | text-recognition | N | Y | N | N | N | N |
|
||||
| [SATRN](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/satrn) | text-recognition | Y | Y | Y | N | N | N |
|
||||
|
|
|
|||
|
|
@ -1,31 +1,158 @@
|
|||
# MMPose Support
|
||||
# MMPose Deployment
|
||||
|
||||
[MMPose](https://github.com/open-mmlab/mmpose) is an open-source toolbox for pose estimation based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
- [Installation](#installation)
|
||||
- [Install mmpose](#install-mmpose)
|
||||
- [Install mmdeploy](#install-mmdeploy)
|
||||
- [Convert model](#convert-model)
|
||||
- [Model specification](#model-specification)
|
||||
- [Model inference](#model-inference)
|
||||
- [Backend model inference](#backend-model-inference)
|
||||
- [SDK model inference](#sdk-model-inference)
|
||||
- [Supported models](#supported-models)
|
||||
|
||||
## MMPose installation tutorial
|
||||
______________________________________________________________________
|
||||
|
||||
Please refer to [official installation guide](https://mmpose.readthedocs.io/en/latest/install.html) to install the codebase.
|
||||
[MMPose](https://github.com/open-mmlab/mmpose/tree/1.x) aka `mmpose` is an open-source toolbox for pose estimation based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
## MMPose models support
|
||||
## Installation
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :-------- | :------------ | :----------: | :------: | :--: | :---: | :------: | :-----------------------------------------------------------------------------------------: |
|
||||
| HRNet | PoseDetection | Y | Y | Y | N | Y | [config](https://mmpose.readthedocs.io/en/latest/papers/backbones.html#hrnet-cvpr-2019) |
|
||||
| MSPN | PoseDetection | Y | Y | Y | N | Y | [config](https://mmpose.readthedocs.io/en/latest/papers/backbones.html#mspn-arxiv-2019) |
|
||||
| LiteHRNet | PoseDetection | Y | Y | Y | N | Y | [config](https://mmpose.readthedocs.io/en/latest/papers/backbones.html#litehrnet-cvpr-2021) |
|
||||
### Install mmpose
|
||||
|
||||
### Example
|
||||
Please follow the [best practice](https://mmpose.readthedocs.io/en/1.x/installation.html#best-practices) to install mmpose.
|
||||
|
||||
```bash
|
||||
python tools/deploy.py \
|
||||
configs/mmpose/posedetection_tensorrt_static-256x192.py \
|
||||
$MMPOSE_DIR/configs/body/2d_kpt_sview_rgb_img/topdown_heatmap/coco/hrnet_w48_coco_256x192.py \
|
||||
$MMPOSE_DIR/checkpoints/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth \
|
||||
$MMDEPLOY_DIR/demo/resources/human-pose.jpg \
|
||||
--work-dir work-dirs/mmpose/topdown/hrnet/trt \
|
||||
--device cuda
|
||||
### Install mmdeploy
|
||||
|
||||
There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.
|
||||
|
||||
**Method I:** Install precompiled package
|
||||
|
||||
> **TODO**. MMDeploy hasn't released based on dev-1.x branch.
|
||||
|
||||
**Method II:** Build using scripts
|
||||
|
||||
If your target platform is **Ubuntu 18.04 or later version**, we encourage you to run
|
||||
[scripts](../01-how-to-build/build_from_script.md). For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
Note
|
||||
**Method III:** Build from source
|
||||
|
||||
- Usually, mmpose models need some extra information for the input image, but we can't get it directly. So, when exporting the model, you can use `$MMDEPLOY_DIR/demo/resources/human-pose.jpg` as input.
|
||||
If neither **I** nor **II** meets your requirements, [building mmdeploy from source](../01-how-to-build/build_from_source.md) is the last option.
|
||||
|
||||
## Convert model
|
||||
|
||||
You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) to convert mmpose models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
The command below shows an example about converting `hrnet` model to onnx model that can be inferred by ONNX Runtime.
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download hrnet model from mmpose model zoo
|
||||
mim download mmpose --config td-hm_hrnet-w32_8xb64-210e_coco-256x192 --dest .
|
||||
# convert mmdet model to onnxruntime model with static shape
|
||||
python tools/deploy.py \
|
||||
configs/mmpose/pose-detection_onnxruntime_static.py \
|
||||
td-hm_hrnet-w32_8xb64-210e_coco-256x192.py \
|
||||
hrnet_w32_coco_256x192-c78dce93_20200708.pth \
|
||||
demo/resources/human-pose.jpg \
|
||||
--work-dir mmdeploy_models/mmpose/ort \
|
||||
--device cpu \
|
||||
--show
|
||||
```
|
||||
|
||||
It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmpose) of all supported backends for mmpose. The config filename pattern is:
|
||||
|
||||
```
|
||||
pose-detection_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
|
||||
- **{precision}:** fp16, int8. When it's empty, it means fp32
|
||||
- **{static | dynamic}:** static shape or dynamic shape
|
||||
- **{shape}:** input shape or shape range of a model
|
||||
|
||||
Therefore, in the above example, you can also convert `hrnet` to other backend models by changing the deployment config file `pose-detection_onnxruntime_static.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmpose), e.g., converting to tensorrt model by `pose-detection_tensorrt_static-256x192.py`.
|
||||
|
||||
```{tip}
|
||||
When converting mmpose models to tensorrt models, --device should be set to "cuda"
|
||||
```
|
||||
|
||||
## Model specification
|
||||
|
||||
Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.
|
||||
|
||||
The converted model locates in the working directory like `mmdeploy_models/mmpose/ort` in the previous example. It includes:
|
||||
|
||||
```
|
||||
mmdeploy_models/mmpose/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
in which,
|
||||
|
||||
- **end2end.onnx**: backend model which can be inferred by ONNX Runtime
|
||||
- \***.json**: the necessary information for mmdeploy SDK
|
||||
|
||||
The whole package **mmdeploy_models/mmpose/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.
|
||||
|
||||
## Model inference
|
||||
|
||||
### Backend model inference
|
||||
|
||||
Take the previous converted `end2end.onnx` model as an example, you can use the following code to inference the model and visualize the results.
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmpose/pose-detection_onnxruntime_static.py'
|
||||
model_cfg = 'td-hm_hrnet-w32_8xb64-210e_coco-256x192.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmpose/ort/end2end.onnx']
|
||||
image = './demo/resources/human-pose.jpg'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_pose.png')
|
||||
```
|
||||
|
||||
### SDK model inference
|
||||
|
||||
TODO
|
||||
|
||||
## Supported models
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :---------------------------------------------------------------------------------------------------- | :------------ | :----------: | :------: | :--: | :---: | :------: |
|
||||
| [HRNet](https://mmpose.readthedocs.io/en/1.x/model_zoo_papers/backbones.html#hrnet-cvpr-2019) | PoseDetection | Y | Y | Y | N | Y |
|
||||
| [MSPN](https://mmpose.readthedocs.io/en/1.x/model_zoo_papers/backbones.html#mspn-arxiv-2019) | PoseDetection | Y | Y | Y | N | Y |
|
||||
| [LiteHRNet](https://mmpose.readthedocs.io/en/1.x/model_zoo_papers/backbones.html#litehrnet-cvpr-2021) | PoseDetection | Y | Y | Y | N | Y |
|
||||
|
|
|
|||
|
|
@ -1,48 +1,33 @@
|
|||
# MMRotate Support
|
||||
# MMRotate Deployment
|
||||
|
||||
- [Installation](#installation)
|
||||
- [Install mmrotate](#install-mmrotate)
|
||||
- [Install mmdeploy](#install-mmdeploy)
|
||||
- [Convert model](#convert-model)
|
||||
- [Model specification](#model-specification)
|
||||
- [Model inference](#model-inference)
|
||||
- [Backend model inference](#backend-model-inference)
|
||||
- [SDK model inference](#sdk-model-inference)
|
||||
- [Supported models](#supported-models)
|
||||
|
||||
______________________________________________________________________
|
||||
|
||||
[MMRotate](https://github.com/open-mmlab/mmrotate) is an open-source toolbox for rotated object detection based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
## MMRotate installation tutorial
|
||||
## Installation
|
||||
|
||||
Please refer to [official installation guide](https://mmrotate.readthedocs.io/en/latest/install.html) to install the codebase.
|
||||
### Install mmrotate
|
||||
|
||||
## MMRotate models support
|
||||
### Install mmdeploy
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | NCNN | PPLNN | OpenVINO | Model config |
|
||||
| :--------------- | :--------------- | :----------: | :------: | :--: | :---: | :------: | :--------------------------------------------------------------------------------------------: |
|
||||
| RotatedRetinaNet | RotatedDetection | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmrotate/blob/main/configs/rotated_retinanet/README.md) |
|
||||
| Oriented RCNN | RotatedDetection | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmrotate/blob/main/configs/oriented_rcnn/README.md) |
|
||||
| Gliding Vertex | RotatedDetection | N | Y | N | N | N | [config](https://github.com/open-mmlab/mmrotate/blob/main/configs/gliding_vertex/README.md) |
|
||||
| RoI Transformer | RotatedDetection | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmrotate/blob/main/configs/roi_trans/README.md) |
|
||||
## Convert model
|
||||
|
||||
### Example
|
||||
## Model specification
|
||||
|
||||
```bash
|
||||
# convert ort
|
||||
python tools/deploy.py \
|
||||
configs/mmrotate/rotated-detection_onnxruntime_dynamic.py \
|
||||
$MMROTATE_DIR/configs/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135.py \
|
||||
$MMROTATE_DIR/checkpoints/rotated_retinanet_obb_r50_fpn_1x_dota_le135-e4131166.pth \
|
||||
$MMROTATE_DIR/demo/demo.jpg \
|
||||
--work-dir work-dirs/mmrotate/rotated_retinanet/ort \
|
||||
--device cpu
|
||||
## Model inference
|
||||
|
||||
# compute metric
|
||||
python tools/test.py \
|
||||
configs/mmrotate/rotated-detection_onnxruntime_dynamic.py \
|
||||
$MMROTATE_DIR/configs/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135.py \
|
||||
--model work-dirs/mmrotate/rotated_retinanet/ort/end2end.onnx \
|
||||
--metrics mAP
|
||||
### Backend model inference
|
||||
|
||||
# generate submit file
|
||||
python tools/test.py \
|
||||
configs/mmrotate/rotated-detection_onnxruntime_dynamic.py \
|
||||
$MMROTATE_DIR/configs/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135.py \
|
||||
--model work-dirs/mmrotate/rotated_retinanet/ort/end2end.onnx \
|
||||
--format-only \
|
||||
--metric-options submission_dir=work-dirs/mmrotate/rotated_retinanet/ort/Task1_results
|
||||
```
|
||||
### SDK model inference
|
||||
|
||||
Note
|
||||
|
||||
- Usually, mmrotate models need some extra information for the input image, but we can't get it directly. So, when exporting the model, you can use `$MMROTATE_DIR/demo/demo.jpg` as input.
|
||||
## Supported models
|
||||
|
|
|
|||
|
|
@ -1,53 +1,232 @@
|
|||
# MMSegmentation Support
|
||||
# MMSegmentation Deployment
|
||||
|
||||
MMSegmentation is an open source object segmentation toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
- [Installation](#installation)
|
||||
- [Install mmseg](#install-mmseg)
|
||||
- [Install mmdeploy](#install-mmdeploy)
|
||||
- [Convert model](#convert-model)
|
||||
- [Model specification](#model-specification)
|
||||
- [Model inference](#model-inference)
|
||||
- [Backend model inference](#backend-model-inference)
|
||||
- [SDK model inference](#sdk-model-inference)
|
||||
- [Supported models](#supported-models)
|
||||
- [Reminder](#reminder)
|
||||
|
||||
## MMSegmentation installation tutorial
|
||||
______________________________________________________________________
|
||||
|
||||
Please refer to [get_started.md](https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/get_started.md#installation) for installation.
|
||||
[MMSegmentation](https://github.com/open-mmlab/mmsegmentation/tree/1.x) aka `mmseg` is an open source semantic segmentation toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
## List of MMSegmentation models supported by MMDeploy
|
||||
## Installation
|
||||
|
||||
| Model | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVino | Model config |
|
||||
| :--------------------------- | :---------: | :------: | :--: | :---: | :------: | :--------------------------------------------------------------------------------------: |
|
||||
| FCN | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fcn) |
|
||||
| PSPNet[\*](#static_shape) | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/pspnet) |
|
||||
| DeepLabV3 | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/deeplabv3) |
|
||||
| DeepLabV3+ | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/deeplabv3plus) |
|
||||
| Fast-SCNN[\*](#static_shape) | Y | Y | N | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fastscnn) |
|
||||
| UNet | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/unet) |
|
||||
| ANN[\*](#static_shape) | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ann) |
|
||||
| APCNet | Y | Y | Y | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/apcnet) |
|
||||
| BiSeNetV1 | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv1) |
|
||||
| BiSeNetV2 | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv2) |
|
||||
| CGNet | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/cgnet) |
|
||||
| DMNet | Y | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dmnet) |
|
||||
| DNLNet | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dnlnet) |
|
||||
| EMANet | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/emanet) |
|
||||
| EncNet | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/encnet) |
|
||||
| ERFNet | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/erfnet) |
|
||||
| FastFCN | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fastfcn) |
|
||||
| GCNet | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/gcnet) |
|
||||
| ICNet[\*](#static_shape) | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/icnet) |
|
||||
| ISANet[\*](#static_shape) | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/isanet) |
|
||||
| NonLocal Net | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/nonlocal_net) |
|
||||
| OCRNet | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ocrnet) |
|
||||
| PointRend[\*](#static_shape) | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/point_rend) |
|
||||
| Semantic FPN | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/sem_fpn) |
|
||||
| STDC | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/stdc) |
|
||||
| UPerNet[\*](#static_shape) | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/upernet) |
|
||||
| DANet | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/danet) |
|
||||
| Segmenter[\*](#static_shape) | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segmenter) |
|
||||
| SegFormer[\*](#static_shape) | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segformer) |
|
||||
| SETR | Y | N | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/setr) |
|
||||
| CCNet | N | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ccnet) |
|
||||
| PSANet | N | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/psanet) |
|
||||
| DPT | N | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dpt) |
|
||||
### Install mmseg
|
||||
|
||||
Please follow the [installation guide](https://mmsegmentation.readthedocs.io/en/1.x/get_started.html) to install mmseg.
|
||||
|
||||
### Install mmdeploy
|
||||
|
||||
There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device.
|
||||
|
||||
**Method I:** Install precompiled package
|
||||
|
||||
> **TODO**. MMDeploy hasn't released based on dev-1.x branch.
|
||||
|
||||
**Method II:** Build using scripts
|
||||
|
||||
If your target platform is **Ubuntu 18.04 or later version**, we encourage you to run
|
||||
[scripts](../01-how-to-build/build_from_script.md). For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
**NOTE**:
|
||||
|
||||
- Adding `$(pwd)/build/lib` to `PYTHONPATH` is for importing mmdeploy SDK python module - `mmdeploy_python`, which will be presented in chapter [SDK model inference](#sdk-model-inference).
|
||||
- When [inference onnx model by ONNX Runtime](#backend-model-inference), it requests ONNX Runtime library be found. Thus, we add it to `LD_LIBRARY_PATH`.
|
||||
|
||||
**Method III:** Build from source
|
||||
|
||||
If neither **I** nor **II** meets your requirements, [building mmdeploy from source](../01-how-to-build/build_from_source.md) is the last option.
|
||||
|
||||
## Convert model
|
||||
|
||||
You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) to convert mmseg models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
The command below shows an example about converting `unet` model to onnx model that can be inferred by ONNX Runtime.
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
|
||||
# download unet model from mmseg model zoo
|
||||
mim download mmsegmentation --config unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024 --dest .
|
||||
|
||||
# convert mmseg model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmseg/segmentation_onnxruntime_dynamic.py \
|
||||
unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py \
|
||||
fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes_20211210_145204-6860854e.pth \
|
||||
demo/resources/cityscapes.png \
|
||||
--work-dir mmdeploy_models/mmseg/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
It is crucial to specify the correct deployment config during model conversion. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmseg) of all supported backends for mmsegmentation. The config filename pattern is:
|
||||
|
||||
```
|
||||
segmentation_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
|
||||
- **{precision}:** fp16, int8. When it's empty, it means fp32
|
||||
- **{static | dynamic}:** static shape or dynamic shape
|
||||
- **{shape}:** input shape or shape range of a model
|
||||
|
||||
Therefore, in the above example, you can also convert `unet` to other backend models by changing the deployment config file `segmentation_onnxruntime_dynamic.py` to [others](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmseg), e.g., converting to tensorrt-fp16 model by `segmentation_tensorrt-fp16_dynamic-512x1024-2048x2048.py`.
|
||||
|
||||
```{tip}
|
||||
When converting mmseg models to tensorrt models, --device should be set to "cuda"
|
||||
```
|
||||
|
||||
## Model specification
|
||||
|
||||
Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.
|
||||
|
||||
The converted model locates in the working directory like `mmdeploy_models/mmseg/ort` in the previous example. It includes:
|
||||
|
||||
```
|
||||
mmdeploy_models/mmseg/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
in which,
|
||||
|
||||
- **end2end.onnx**: backend model which can be inferred by ONNX Runtime
|
||||
- \***.json**: the necessary information for mmdeploy SDK
|
||||
|
||||
The whole package **mmdeploy_models/mmseg/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information.
|
||||
|
||||
## Model inference
|
||||
|
||||
### Backend model inference
|
||||
|
||||
Take the previous converted `end2end.onnx` model as an example, you can use the following code to inference the model and visualize the results.
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmseg/segmentation_onnxruntime_dynamic.py'
|
||||
model_cfg = './unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmseg/ort/end2end.onnx']
|
||||
image = './demo/resources/cityscapes.png'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='./output_segmentation.png')
|
||||
```
|
||||
|
||||
### SDK model inference
|
||||
|
||||
You can also perform SDK model inference like following,
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Segmentor
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
img = cv2.imread('./demo/resources/cityscapes.png')
|
||||
|
||||
# create a classifier
|
||||
segmentor = Segmentor(model_path='./mmdeploy_models/mmseg/ort', device_name='cpu', device_id=0)
|
||||
# perform inference
|
||||
seg = segmentor(img)
|
||||
|
||||
# visualize inference result
|
||||
## random a palette with size 256x3
|
||||
palette = np.random.randint(0, 256, size=(256, 3))
|
||||
color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8)
|
||||
for label, color in enumerate(palette):
|
||||
color_seg[seg == label, :] = color
|
||||
# convert to BGR
|
||||
color_seg = color_seg[..., ::-1]
|
||||
img = img * 0.5 + color_seg * 0.5
|
||||
img = img.astype(np.uint8)
|
||||
cv2.imwrite('output_segmentation.png', img)
|
||||
```
|
||||
|
||||
Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo).
|
||||
|
||||
## Supported models
|
||||
|
||||
| Model | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVino |
|
||||
| :---------------------------------------------------------------------------------------------------------- | :---------: | :------: | :--: | :---: | :------: |
|
||||
| [FCN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fcn) | Y | Y | Y | Y | Y |
|
||||
| [PSPNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/pspnet)[\*](#static_shape) | Y | Y | Y | Y | Y |
|
||||
| [DeepLabV3](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/deeplabv3) | Y | Y | Y | Y | Y |
|
||||
| [DeepLabV3+](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/deeplabv3plus) | Y | Y | Y | Y | Y |
|
||||
| [Fast-SCNN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fastscnn)[\*](#static_shape) | Y | Y | N | Y | Y |
|
||||
| [UNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/unet) | Y | Y | Y | Y | Y |
|
||||
| [ANN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ann)[\*](#static_shape) | Y | Y | N | N | N |
|
||||
| [APCNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/apcnet) | Y | Y | Y | N | N |
|
||||
| [BiSeNetV1](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv1) | Y | Y | Y | N | Y |
|
||||
| [BiSeNetV2](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv2) | Y | Y | Y | N | Y |
|
||||
| [CGNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/cgnet) | Y | Y | Y | N | Y |
|
||||
| [DMNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dmnet) | Y | N | N | N | N |
|
||||
| [DNLNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dnlnet) | Y | Y | Y | N | Y |
|
||||
| [EMANet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/emanet) | Y | Y | N | N | Y |
|
||||
| [EncNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/encnet) | Y | Y | N | N | Y |
|
||||
| [ERFNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/erfnet) | Y | Y | Y | N | Y |
|
||||
| [FastFCN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fastfcn) | Y | Y | Y | N | Y |
|
||||
| [GCNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/gcnet) | Y | Y | N | N | N |
|
||||
| [ICNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/icnet)[\*](#static_shape) | Y | Y | N | N | Y |
|
||||
| [ISANet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/isanet)[\*](#static_shape) | Y | Y | N | N | Y |
|
||||
| [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/nonlocal_net) | Y | Y | Y | N | Y |
|
||||
| [OCRNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ocrnet) | Y | Y | Y | N | Y |
|
||||
| [PointRend](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/point_rend)[\*](#static_shape) | Y | Y | N | N | N |
|
||||
| [Semantic FPN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/sem_fpn) | Y | Y | Y | N | Y |
|
||||
| [STDC](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/stdc) | Y | Y | Y | N | Y |
|
||||
| [UPerNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/upernet)[\*](#static_shape) | Y | Y | N | N | N |
|
||||
| [DANet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/danet) | Y | Y | N | N | Y |
|
||||
| [Segmenter](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segmenter)[\*](#static_shape) | Y | Y | Y | N | Y |
|
||||
| [SegFormer](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segformer)[\*](#static_shape) | Y | Y | N | N | Y |
|
||||
| [SETR](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/setr) | Y | N | N | N | Y |
|
||||
| [CCNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ccnet) | N | N | N | N | N |
|
||||
| [PSANet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/psanet) | N | N | N | N | N |
|
||||
| [DPT](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dpt) | N | N | N | N | N |
|
||||
|
||||
## Reminder
|
||||
|
||||
- Only `whole` inference mode is supported for all mmseg models.
|
||||
|
||||
- <i id="static_shape">PSPNet, Fast-SCNN</i> only support static shape, because [nn.AdaptiveAvgPool2d](https://github.com/open-mmlab/mmsegmentation/blob/97f9670c5a4a2a3b4cfb411bcc26db16b23745f7/mmseg/models/decode_heads/psp_head.py#L38) is not supported in most of backends dynamically.
|
||||
- <i id="static_shape">PSPNet, Fast-SCNN</i> only support static shape, because [nn.AdaptiveAvgPool2d](https://github.com/open-mmlab/mmsegmentation/blob/97f9670c5a4a2a3b4cfb411bcc26db16b23745f7/mmseg/models/decode_heads/psp_head.py#L38) is not supported by most inference backends.
|
||||
|
||||
- For models only supporting static shape, you should use the deployment config file of static shape such as `configs/mmseg/segmentation_tensorrt_static-1024x2048.py`.
|
||||
- For models that only supports static shape, you should use the deployment config file of static shape such as `configs/mmseg/segmentation_tensorrt_static-1024x2048.py`.
|
||||
|
|
|
|||
|
|
@ -1,85 +0,0 @@
|
|||
# How to get partitioned ONNX models
|
||||
|
||||
MMDeploy 支持将PyTorch模型导出到onnx模型并进行拆分得到多个onnx模型文件，用户可以自由的对模型图节点进行标记并根据这些标记的节点定制任意的onnx模型拆分策略。在这个教程中，我们将通过具体例子来展示如何进行onnx模型拆分。在这个例子中，我们的目标是将YOLOV3模型拆分成两个部分，保留不带后处理的onnx模型，丢弃包含Anchor生成，NMS的后处理部分。
|
||||
|
||||
## 步骤 1: 添加模型标记点
|
||||
|
||||
为了进行图拆分，我们定义了`Mark`类型op，标记模型导出的边界。在实现方法上，采用`mark`装饰器对函数的输入、输出`Tensor`打标记。需要注意的是，我们的标记函数需要在某个重写函数中执行才能生效。
|
||||
|
||||
为了对YOLOV3进行拆分，首先我们需要标记模型的输入。这里为了通用性，我们标记检测器父类`BaseDetector`的`forward`方法中的`img` `Tensor`，同时为了支持其他拆分方案，也对`forward`函数的输出进行了标记，分别是`dets`, `labels`和`masks`。下面的代码是截图[mmdeploy/codebase/mmdet/models/detectors/base.py](https://github.com/open-mmlab/mmdeploy/blob/86a50e343a3a45d7bc2ba3256100accc4973e71d/mmdeploy/codebase/mmdet/models/detectors/base.py)中的一部分，可以看出我们使用`mark`装饰器标记了`__forward_impl`函数的输入输出，并在重写函数`base_detector__forward`进行了调用，从而完成了对检测器输入的标记。
|
||||
|
||||
```python
|
||||
from mmdeploy.core import FUNCTION_REWRITER, mark
|
||||
|
||||
@mark(
|
||||
'detector_forward', inputs=['input'], outputs=['dets', 'labels', 'masks'])
|
||||
def __forward_impl(ctx, self, img, img_metas=None, **kwargs):
|
||||
...
|
||||
|
||||
|
||||
@FUNCTION_REWRITER.register_rewriter(
|
||||
'mmdet.models.detectors.base.BaseDetector.forward')
|
||||
def base_detector__forward(ctx, self, img, img_metas=None, **kwargs):
|
||||
...
|
||||
# call the mark function
|
||||
return __forward_impl(...)
|
||||
```
|
||||
|
||||
接下来，我们只需要对`YOLOV3Head`中最后一层输出特征`Tensor`进行标记就可以将整个`YOLOV3`模型拆分成两部分。通过查看`mmdet`源码我们可以知道`YOLOV3Head`的`get_bboxes`方法中输入参数`pred_maps`就是我们想要的拆分点，因此可以在重写函数[`yolov3_head__get_bboxes`](https://github.com/open-mmlab/mmdeploy/blob/86a50e343a3a45d7bc2ba3256100accc4973e71d/mmdeploy/codebase/mmdet/models/dense_heads/yolo_head.py#L14)中添加内部函数对`pred_mapes`进行标记，具体参考如下示例代码。值得注意的是，输入参数`pred_maps`是由三个`Tensor`组成的列表，所以我们在onnx模型中添加了三个`Mark`标记节点。
|
||||
|
||||
```python
|
||||
from mmdeploy.core import FUNCTION_REWRITER, mark
|
||||
|
||||
@FUNCTION_REWRITER.register_rewriter(
|
||||
func_name='mmdet.models.dense_heads.YOLOV3Head.get_bboxes')
|
||||
def yolov3_head__get_bboxes(ctx,
|
||||
self,
|
||||
pred_maps,
|
||||
img_metas,
|
||||
cfg=None,
|
||||
rescale=False,
|
||||
with_nms=True):
|
||||
# mark pred_maps
|
||||
@mark('yolo_head', inputs=['pred_maps'])
|
||||
def __mark_pred_maps(pred_maps):
|
||||
return pred_maps
|
||||
pred_maps = __mark_pred_maps(pred_maps)
|
||||
...
|
||||
```
|
||||
|
||||
## 步骤 2: 添加部署配置文件
|
||||
|
||||
在完成模型中节点标记之后，我们需要创建部署配置文件，我们假设部署后端是`onnxruntime`,并模型输入是固定尺寸`608x608`,因此添加文件`configs/mmdet/detection/yolov3_partition_onnxruntime_static.py`. 我们需要在配置文件中添加基本的配置信息如`onnx_config`，如何你还不熟悉如何添加配置文件，可以参考[write_config.md](../02-how-to-run/write_config.md).
|
||||
|
||||
在这个部署配置文件中, 我们需要添加一个特殊的模型分段配置字段`partition_config`. 在模型分段配置中，我们可以可以给分段策略添加一个类型名称如`yolov3_partition`，设定`apply_marks=True`。在分段方式`partition_cfg`,我们需要指定每段模型的分割起始点`start`, 终止点`end`以及保存分段onnx的文件名。需要提醒的是，各段模型起始点`start`和终止点`end`是由多个标记节点`Mark`组成，例如`'detector_forward:input'`代表`detector_forward`标记处输入所产生的标记节点。配置文件具体内容参考如下代码：
|
||||
|
||||
```python
|
||||
_base_ = ['./detection_onnxruntime_static.py']
|
||||
|
||||
onnx_config = dict(input_shape=[608, 608])
|
||||
partition_config = dict(
|
||||
type='yolov3_partition', # the partition policy name
|
||||
apply_marks=True, # should always be set to True
|
||||
partition_cfg=[
|
||||
dict(
|
||||
save_file='yolov3.onnx', # filename to save the partitioned onnx model
|
||||
start=['detector_forward:input'], # [mark_name:input/output, ...]
|
||||
end=['yolo_head:input']) # [mark_name:input/output, ...]
|
||||
])
|
||||
|
||||
```
|
||||
|
||||
## 步骤 3: 拆分onnx模型
|
||||
|
||||
添加好节点标记和部署配置文件，我们可以使用`tools/torch2onnx.py`工具导出带有`Mark`标记的完成onnx模型并根据分段策略提取分段的onnx模型文件。我们可以执行如下脚本，得到不带后处理的`YOLOV3`onnx模型文件`yolov3.onnx`，同时输出文件中也包含了添加`Mark`标记的完整模型文件`end2end.onnx`。此外，用户可以使用网页版模型可视化工具[netron](https://netron.app/)来查看和验证输出onnx模型的结构是否正确。
|
||||
|
||||
```shell
|
||||
python tools/torch2onnx.py \
|
||||
configs/mmdet/detection/yolov3_partition_onnxruntime_static.py \
|
||||
../mmdetection/configs/yolo/yolov3_d53_mstrain-608_273e_coco.py \
|
||||
https://download.openmmlab.com/mmdetection/v2.0/yolo/yolov3_d53_mstrain-608_273e_coco/yolov3_d53_mstrain-608_273e_coco_20210518_115020-a2c3acb8.pth \
|
||||
../mmdetection/demo/demo.jpg \
|
||||
--work-dir ./work-dirs/mmdet/yolov3/ort/partition
|
||||
```
|
||||
|
||||
当得到分段onnx模型之后，我们可以使用mmdeploy提供的其他工具如`mmdeploy_onnx2ncnn`, `onnx2tensorrt`来进行后续的模型部署工作。
|
||||
|
|
@ -1,20 +1,188 @@
|
|||
# mmcls 模型支持列表
|
||||
# MMClassification 模型部署
|
||||
|
||||
[MMClassification](https://github.com/open-mmlab/mmclassification) 是基于 Python 的的图像分类工具，属于 [OpenMMLab](https://openmmlab.com)。
|
||||
- [安装](#安装)
|
||||
- [安装 mmcls](#安装-mmcls)
|
||||
- [安装 mmdeploy](#安装-mmdeploy)
|
||||
- [模型转换](#模型转换)
|
||||
- [模型规范](#模型规范)
|
||||
- [模型推理](#模型推理)
|
||||
- [后端模型推理](#后端模型推理)
|
||||
- [SDK 模型推理](#sdk-模型推理)
|
||||
- [模型支持列表](#模型支持列表)
|
||||
|
||||
## 安装 mmcls
|
||||
______________________________________________________________________
|
||||
|
||||
请参考 [install.md](https://github.com/open-mmlab/mmclassification/blob/master/docs/en/install.md) 进行安装。
|
||||
[MMClassification](https://github.com/open-mmlab/mmclassification)，又称 `mmcls` ，是基于 Python 的的图像分类工具，属于 [OpenMMLab](https://openmmlab.com)。
|
||||
|
||||
## 支持列表
|
||||
## 安装
|
||||
|
||||
| Model | TorchScript | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :---------------- | :---------: | :----------: | :------: | :--: | :---: | :------: | :---------------------------------------------------------------------------------------------: |
|
||||
| ResNet | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnet) |
|
||||
| ResNeXt | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnext) |
|
||||
| SE-ResNet | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/seresnet) |
|
||||
| MobileNetV2 | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/mobilenet_v2) |
|
||||
| ShuffleNetV1 | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v1) |
|
||||
| ShuffleNetV2 | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v2) |
|
||||
| VisionTransformer | Y | Y | Y | Y | ? | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/vision_transformer) |
|
||||
| SwinTransformer | Y | Y | Y | N | ? | N | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/swin_transformer) |
|
||||
### 安装 mmcls
|
||||
|
||||
请参考[官网安装指南](https://github.com/open-mmlab/mmclassification/tree/1.x#installation).
|
||||
|
||||
### 安装 mmdeploy
|
||||
|
||||
mmdeploy 有以下几种安装方式:
|
||||
|
||||
**方式一：** 安装预编译包
|
||||
|
||||
> 待 mmdeploy 正式发布 dev-1.x，再补充
|
||||
|
||||
**方式二：** 一键式脚本安装
|
||||
|
||||
如果部署平台是 **Ubuntu 18.04 及以上版本**， 请参考[脚本安装说明](../01-how-to-build/build_from_script.md)，完成安装过程。
|
||||
比如，以下命令可以安装 mmdeploy 以及配套的推理引擎——`ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
**方式三：** 源码安装
|
||||
|
||||
在方式一、二都满足不了的情况下，请参考[源码安装说明](../01-how-to-build/build_from_source.md) 安装 mmdeploy 以及所需推理引擎。
|
||||
|
||||
## 模型转换
|
||||
|
||||
你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) 把 mmcls 模型一键式转换为推理后端模型。
|
||||
该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
以下，我们将演示如何把 `resnet18` 转换为 onnx 模型。
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
|
||||
# download resnet18 model from mmcls model zoo
|
||||
mim download mmcls --config resnet18_8xb32_in1k --dest .
|
||||
|
||||
# convert mmcls model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmcls/classification_onnxruntime_dynamic.py \
|
||||
resnet18_8xb32_in1k.py \
|
||||
resnet18_8xb32_in1k_20210831-fbbb1da6.pth \
|
||||
tests/data/tiger.jpeg \
|
||||
--work-dir mmdeploy_models/mmcls/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmcls)。
|
||||
文件的命名模式是：
|
||||
|
||||
```
|
||||
classification_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
其中：
|
||||
|
||||
- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等
|
||||
- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32
|
||||
- **{static | dynamic}:** 动态、静态 shape
|
||||
- **{shape}:** 模型输入的 shape 或者 shape 范围
|
||||
|
||||
在上例中，你也可以把 `resnet18` 转为其他后端模型。比如使用`classification_tensorrt-fp16_dynamic-224x224-224x224.py`，把模型转为 tensorrt-fp16 模型。
|
||||
|
||||
```{tip}
|
||||
当转 tensorrt 模型时, --device 需要被设置为 "cuda"
|
||||
```
|
||||
|
||||
## 模型规范
|
||||
|
||||
在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。
|
||||
|
||||
上例中的`mmdeploy_models/mmcls/ort`，结构如下：
|
||||
|
||||
```
|
||||
mmdeploy_models/mmcls/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
重要的是：
|
||||
|
||||
- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理
|
||||
- \***.json**: mmdeploy SDK 推理所需的 meta 信息
|
||||
|
||||
整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。
|
||||
|
||||
## 模型推理
|
||||
|
||||
### 后端模型推理
|
||||
|
||||
以上述模型转换后的 `end2end.onnx` 为例，你可以使用如下代码进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmcls/classification_onnxruntime_dynamic.py'
|
||||
model_cfg = './resnet18_8xb32_in1k.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmcls/ort/end2end.onnx']
|
||||
image = 'tests/data/tiger.jpeg'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_classification.png')
|
||||
```
|
||||
|
||||
### SDK 模型推理
|
||||
|
||||
你也可以参考如下代码，对 SDK model 进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Classifier
|
||||
import cv2
|
||||
|
||||
img = cv2.imread('tests/data/tiger.jpeg')
|
||||
|
||||
# create a classifier
|
||||
classifier = Classifier(model_path='./mmdeploy_models/mmcls/ort', device_name='cpu', device_id=0)
|
||||
# perform inference
|
||||
result = classifier(img)
|
||||
# show inference result
|
||||
for label_id, score in result:
|
||||
print(label_id, score)
|
||||
```
|
||||
|
||||
除了python API，mmdeploy SDK 还提供了诸如 C、C++、C#、Java等多语言接口。
|
||||
你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo)学习其他语言接口的使用方法。
|
||||
|
||||
## 模型支持列表
|
||||
|
||||
| Model | TorchScript | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :--------------------------------------------------------------------------------------------------------- | :---------: | :----------: | :------: | :--: | :---: | :------: |
|
||||
| [ResNet](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnet) | Y | Y | Y | Y | Y | Y |
|
||||
| [ResNeXt](https://github.com/open-mmlab/mmclassification/tree/master/configs/resnext) | Y | Y | Y | Y | Y | Y |
|
||||
| [SE-ResNet](https://github.com/open-mmlab/mmclassification/tree/master/configs/seresnet) | Y | Y | Y | Y | Y | Y |
|
||||
| [MobileNetV2](https://github.com/open-mmlab/mmclassification/tree/master/configs/mobilenet_v2) | Y | Y | Y | Y | Y | Y |
|
||||
| [ShuffleNetV1](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v1) | Y | Y | Y | Y | Y | Y |
|
||||
| [ShuffleNetV2](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v2) | Y | Y | Y | Y | Y | Y |
|
||||
| [VisionTransformer](https://github.com/open-mmlab/mmclassification/tree/master/configs/vision_transformer) | Y | Y | Y | Y | ? | Y |
|
||||
| [SwinTransformer](https://github.com/open-mmlab/mmclassification/tree/master/configs/swin_transformer) | Y | Y | Y | N | ? | N |
|
||||
|
|
|
|||
|
|
@ -1,30 +1,213 @@
|
|||
# mmdet 模型支持列表
|
||||
# MMDetection 模型部署
|
||||
|
||||
mmdet 是基于 pytorch 的检测工具箱，属于 [OpenMMLab](https://openmmlab.com/)。
|
||||
- [安装](#安装)
|
||||
- [安装 mmdet](#安装-mmdet)
|
||||
- [安装 mmdeploy](#安装-mmdeploy)
|
||||
- [模型转换](#模型转换)
|
||||
- [模型规范](#模型规范)
|
||||
- [模型推理](#模型推理)
|
||||
- [后端模型推理](#后端模型推理)
|
||||
- [SDK 模型推理](#sdk-模型推理)
|
||||
- [模型支持列表](#模型支持列表)
|
||||
|
||||
## 安装 mmdet
|
||||
______________________________________________________________________
|
||||
|
||||
请参照 [get_started.md](https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md) 。
|
||||
[MMDetection](https://github.com/open-mmlab/mmdetection) ，又称 `mmdet`， 是一个基于 PyTorch 的目标检测开源工具箱。它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。
|
||||
|
||||
## 支持列表
|
||||
## 安装
|
||||
|
||||
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----------------: | :------------------: | :---------: | :------: | :--: | :---: | :------: | :----------------------------------------------------------------------------------: |
|
||||
| ATSS | ObjectDetection | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/atss) |
|
||||
| FCOS | ObjectDetection | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/fcos) |
|
||||
| FoveaBox | ObjectDetection | Y | N | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/foveabox) |
|
||||
| FSAF | ObjectDetection | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/fsaf) |
|
||||
| RetinaNet | ObjectDetection | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/retinanet) |
|
||||
| SSD | ObjectDetection | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/ssd) |
|
||||
| VFNet | ObjectDetection | N | N | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/vfnet) |
|
||||
| YOLOv3 | ObjectDetection | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/yolo) |
|
||||
| YOLOX | ObjectDetection | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/yolox) |
|
||||
| Cascade R-CNN | ObjectDetection | Y | Y | N | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) |
|
||||
| Faster R-CNN | ObjectDetection | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) |
|
||||
| Faster R-CNN + DCN | ObjectDetection | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) |
|
||||
| GFL | ObjectDetection | Y | Y | N | ? | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/gfl) |
|
||||
| RepPoints | ObjectDetection | N | Y | N | ? | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/reppoints) |
|
||||
| DETR | ObjectDetection | Y | Y | N | ? | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/detr) |
|
||||
| Cascade Mask R-CNN | InstanceSegmentation | Y | N | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) |
|
||||
| Mask R-CNN | InstanceSegmentation | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn) |
|
||||
| Swin Transformer | InstanceSegmentation | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/swin) |
|
||||
### 安装 mmdet
|
||||
|
||||
请参考[官网安装指南](https://mmdetection.readthedocs.io/en/3.x/get_started.html)。
|
||||
|
||||
### 安装 mmdeploy
|
||||
|
||||
mmdeploy 有以下几种安装方式:
|
||||
|
||||
**方式一：** 安装预编译包
|
||||
|
||||
> 待 mmdeploy 正式发布 dev-1.x，再补充
|
||||
|
||||
**方式二：** 一键式脚本安装
|
||||
|
||||
如果部署平台是 **Ubuntu 18.04 及以上版本**， 请参考[脚本安装说明](../01-how-to-build/build_from_script.md)，完成安装过程。
|
||||
比如，以下命令可以安装 mmdeploy 以及配套的推理引擎——`ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
**方式三：** 源码安装
|
||||
|
||||
在方式一、二都满足不了的情况下，请参考[源码安装说明](../01-how-to-build/build_from_source.md) 安装 mmdeploy 以及所需推理引擎。
|
||||
|
||||
## 模型转换
|
||||
|
||||
你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) 把 mmdet 模型一键式转换为推理后端模型。
|
||||
该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
以下，我们将演示如何把 `Faster R-CNN` 转换为 onnx 模型。
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download faster r-cnn model from mmdet model zoo
|
||||
mim download mmdet --config faster-rcnn_r50_fpn_1x_coco --dest .
|
||||
# convert mmdet model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmdet/detection/detection_onnxruntime_dynamic.py \
|
||||
faster-rcnn_r50_fpn_1x_coco.py \
|
||||
faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
|
||||
demo/resources/det.jpg \
|
||||
--work-dir mmdeploy_models/mmdet/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmdet)。
|
||||
文件的命名模式是：
|
||||
|
||||
```
|
||||
{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
其中：
|
||||
|
||||
- **{task}:** mmdet 中的任务
|
||||
|
||||
mmdet 任务有2种：物体检测（detection）、实例分割（instance-seg）。例如，`RetinaNet`、`Faster R-CNN`、`DETR`等属于前者。`Mask R-CNN`、`SOLO`等属于后者。更多`模型-任务`的划分，请参考章节[模型支持列表](#模型支持列表)。
|
||||
|
||||
**请务必**使用 `detection/detection_*.py` 转换检测模型，使用 `instance-seg/instance-seg_*.py` 转换实例分割模型。
|
||||
|
||||
- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等
|
||||
|
||||
- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32
|
||||
|
||||
- **{static | dynamic}:** 动态、静态 shape
|
||||
|
||||
- **{shape}:** 模型输入的 shape 或者 shape 范围
|
||||
|
||||
在上例中，你也可以把 `Faster R-CNN` 转为其他后端模型。比如使用`detection_tensorrt-fp16_dynamic-320x320-1344x1344.py`，把模型转为 tensorrt-fp16 模型。
|
||||
|
||||
```{tip}
|
||||
当转 tensorrt 模型时, --device 需要被设置为 "cuda"
|
||||
```
|
||||
|
||||
## 模型规范
|
||||
|
||||
在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。
|
||||
|
||||
上例中的`mmdeploy_models/mmdet/ort`，结构如下：
|
||||
|
||||
```
|
||||
mmdeploy_models/mmdet/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
重要的是：
|
||||
|
||||
- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理
|
||||
- \***.json**: mmdeploy SDK 推理所需的 meta 信息
|
||||
|
||||
整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。
|
||||
|
||||
## 模型推理
|
||||
|
||||
## 后端模型推理
|
||||
|
||||
以上述模型转换后的 `end2end.onnx` 为例，你可以使用如下代码进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmdet/detection/detection_onnxruntime_dynamic.py'
|
||||
model_cfg = './faster-rcnn_r50_fpn_1x_coco.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmdet/ort/end2end.onnx']
|
||||
image = './demo/resources/det.jpg'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_detection.png')
|
||||
```
|
||||
|
||||
## SDK 模型推理
|
||||
|
||||
你也可以参考如下代码，对 SDK model 进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Detector
|
||||
import cv2
|
||||
|
||||
img = cv2.imread('./demo/resources/det.jpg')
|
||||
|
||||
# create a detector
|
||||
detector = Detector(model_path='./mmdeploy_models/mmdet/ort', device_name='cpu', device_id=0)
|
||||
# perform inference
|
||||
bboxes, labels, masks = detector(img)
|
||||
|
||||
# visualize inference result
|
||||
indices = [i for i in range(len(bboxes))]
|
||||
for index, bbox, label_id in zip(indices, bboxes, labels):
|
||||
[left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
|
||||
if score < 0.3:
|
||||
continue
|
||||
|
||||
cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))
|
||||
|
||||
cv2.imwrite('output_detection.png', img)
|
||||
```
|
||||
|
||||
除了python API，mmdeploy SDK 还提供了诸如 C、C++、C#、Java等多语言接口。
|
||||
你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo)学习其他语言接口的使用方法。
|
||||
|
||||
## 模型支持列表
|
||||
|
||||
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :----------------------------------------------------------------------------------------------: | :------------------: | :---------: | :------: | :--: | :---: | :------: |
|
||||
| [ATSS](https://github.com/open-mmlab/mmdetection/tree/master/configs/atss) | ObjectDetection | Y | Y | N | N | Y |
|
||||
| [FCOS](https://github.com/open-mmlab/mmdetection/tree/master/configs/fcos) | ObjectDetection | Y | Y | Y | N | Y |
|
||||
| [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/master/configs/foveabox) | ObjectDetection | Y | N | N | N | Y |
|
||||
| [FSAF](https://github.com/open-mmlab/mmdetection/tree/master/configs/fsaf) | ObjectDetection | Y | Y | Y | Y | Y |
|
||||
| [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/master/configs/retinanet) | ObjectDetection | Y | Y | Y | Y | Y |
|
||||
| [SSD](https://github.com/open-mmlab/mmdetection/tree/master/configs/ssd) | ObjectDetection | Y | Y | Y | N | Y |
|
||||
| [VFNet](https://github.com/open-mmlab/mmdetection/tree/master/configs/vfnet) | ObjectDetection | N | N | N | N | Y |
|
||||
| [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/master/configs/yolo) | ObjectDetection | Y | Y | Y | N | Y |
|
||||
| [YOLOX](https://github.com/open-mmlab/mmdetection/tree/master/configs/yolox) | ObjectDetection | Y | Y | Y | N | Y |
|
||||
| [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) | ObjectDetection | Y | Y | N | Y | Y |
|
||||
| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) | ObjectDetection | Y | Y | Y | Y | Y |
|
||||
| [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) | ObjectDetection | Y | Y | Y | Y | Y |
|
||||
| [GFL](https://github.com/open-mmlab/mmdetection/tree/master/configs/gfl) | ObjectDetection | Y | Y | N | ? | Y |
|
||||
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/master/configs/reppoints) | ObjectDetection | N | Y | N | ? | Y |
|
||||
| [DETR](https://github.com/open-mmlab/mmdetection/tree/master/configs/detr) | ObjectDetection | Y | Y | N | ? | Y |
|
||||
| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) | InstanceSegmentation | Y | N | N | N | Y |
|
||||
| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn) | InstanceSegmentation | Y | Y | N | N | Y |
|
||||
| [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/master/configs/swin) | InstanceSegmentation | Y | Y | N | N | N |
|
||||
|
|
|
|||
|
|
@ -1,42 +1,33 @@
|
|||
# mmdet3d 模型支持列表
|
||||
# MMDetection3d 模型部署
|
||||
|
||||
MMDetection3d是用于通用 3D 物体检测平台。属于 [OpenMMLab](https://openmmlab.com/)。
|
||||
- [安装](#安装)
|
||||
- [安装 mmcls](#安装-mmdet3d)
|
||||
- [安装 mmdeploy](#安装-mmdeploy)
|
||||
- [模型转换](#模型转换)
|
||||
- [模型规范](#模型规范)
|
||||
- [模型推理](#模型推理)
|
||||
- [后端模型推理](#后端模型推理)
|
||||
- [SDK 模型推理](#sdk-模型推理)
|
||||
- [模型支持列表](#模型支持列表)
|
||||
|
||||
## 安装 mmdet3d
|
||||
______________________________________________________________________
|
||||
|
||||
参照 [getting_started.md](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/getting_started.md)。
|
||||
[MMDetection3d](https://github.com/open-mmlab/mmdetection3d)，又称 `mmdet3d`， 是一个基于 PyTorch 的目标检测开源工具箱, 下一代面向3D检测的平台。它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。
|
||||
|
||||
## 示例
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
export MODEL_PATH=https://download.openmmlab.com/mmdetection3d/v1.0.0_models/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car/hv_pointpillars_secfpn_6x8_160e_kitti-3d-car_20220331_134606-d42d15ed.pth
|
||||
### 安装 mmdet3d
|
||||
|
||||
python tools/deploy.py \
|
||||
configs/mmdet3d/voxel-detection/voxel-detection_tensorrt_dynamic.py \
|
||||
${MMDET3D_DIR}/configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py \
|
||||
${MODEL_PATH} \
|
||||
${MMDET3D_DIR}/demo/data/kitti/kitti_000008.bin \
|
||||
--work-dir \
|
||||
work_dir \
|
||||
--show \
|
||||
--device \
|
||||
cuda:0
|
||||
```
|
||||
### 安装 mmdeploy
|
||||
|
||||
## 支持列表
|
||||
## 模型转换
|
||||
|
||||
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----------: | :------------: | :---------: | :------: | :--: | :---: | :------: | :------------------------------------------------------------------------------------: |
|
||||
| PointPillars | VoxelDetection | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/pointpillars) |
|
||||
## 模型规范
|
||||
|
||||
## 注意事项
|
||||
## 模型推理
|
||||
|
||||
体素检测 onnx 模型不包含 model.voxelize 层和模型后处理，可用 python api 来调这些函数。
|
||||
### 后端模型推理
|
||||
|
||||
示例：
|
||||
### SDK 模型推理
|
||||
|
||||
```python
|
||||
from mmdeploy.codebase.mmdet3d.deploy import VoxelDetectionModel
|
||||
VoxelDetectionModel.voxelize(...)
|
||||
VoxelDetectionModel.post_process(...)
|
||||
```
|
||||
## 模型支持列表
|
||||
|
|
|
|||
|
|
@ -1,20 +1,201 @@
|
|||
# mmedit 模型支持列表
|
||||
# MMEditing 模型推理
|
||||
|
||||
[mmedit](https://github.com/open-mmlab/mmediting) 是基于 PyTorch 的开源图像和视频编辑工具箱，属于 [OpenMMLab](https://openmmlab.com/)。
|
||||
- [MMEditing 模型推理](#mmediting-模型推理)
|
||||
- [安装](#安装)
|
||||
- [安装 mmedit](#安装-mmedit)
|
||||
- [安装 mmdeploy](#安装-mmdeploy)
|
||||
- [模型转换](#模型转换)
|
||||
- [超分任务模型转换](#超分任务模型转换)
|
||||
- [模型规范](#模型规范)
|
||||
- [模型推理](#模型推理)
|
||||
- [后端模型推理](#后端模型推理)
|
||||
- [SDK 模型推理](#sdk-模型推理)
|
||||
- [模型支持列表](#模型支持列表)
|
||||
|
||||
## 安装 mmedit
|
||||
______________________________________________________________________
|
||||
|
||||
参照 [official installation guide](https://mmediting.readthedocs.io/en/latest/install.html#installation)。
|
||||
[MMEditing](https://github.com/open-mmlab/mmediting)，又称 `mmedit`，是基于 PyTorch 的开源图像和视频编辑工具箱。它是 [OpenMMLab](https://openmmlab.com/) 项目成员之一。
|
||||
|
||||
## 支持列表
|
||||
## 安装
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :---------- | :--------------- | :----------: | :------: | :--: | :---: | :------: | :--------------------------------------------------------------------------------------------: |
|
||||
| SRCNN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srcnn) |
|
||||
| ESRGAN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/esrgan) |
|
||||
| ESRGAN-PSNR | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/esrgan) |
|
||||
| SRGAN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srresnet_srgan) |
|
||||
| SRResNet | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srresnet_srgan) |
|
||||
| Real-ESRGAN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/real_esrgan) |
|
||||
| EDSR | super-resolution | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/edsr) |
|
||||
| RDN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/rdn) |
|
||||
### 安装 mmedit
|
||||
|
||||
请参考[官网安装指南](https://github.com/open-mmlab/mmediting/tree/1.x#installation)。
|
||||
|
||||
### 安装 mmdeploy
|
||||
|
||||
mmdeploy 有以下几种安装方式:
|
||||
|
||||
**方式一：** 安装预编译包
|
||||
|
||||
> 待 mmdeploy 正式发布 dev-1.x，再补充
|
||||
|
||||
**方式二：** 一键式脚本安装
|
||||
|
||||
如果部署平台是 **Ubuntu 18.04 及以上版本**， 请参考[脚本安装说明](../01-how-to-build/build_from_script.md)，完成安装过程。
|
||||
比如，以下命令可以安装 mmdeploy 以及配套的推理引擎——`ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
**方式三：** 源码安装
|
||||
|
||||
在方式一、二都满足不了的情况下，请参考[源码安装说明](../01-how-to-build/build_from_source.md) 安装 mmdeploy 以及所需推理引擎。
|
||||
|
||||
## 模型转换
|
||||
|
||||
你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) 把 mmedit 模型一键式转换为推理后端模型。
|
||||
该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmedit)。
|
||||
文件的命名模式是：
|
||||
|
||||
```
|
||||
{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
其中：
|
||||
|
||||
- **{task}:** mmedit 中的任务
|
||||
|
||||
mmedit 中任务有多种。目前，mmdeploy 支持其中的超分（super resolution）任务。关于`模型-任务`的划分，请参考章节[模型支持列表](#模型支持列表)。
|
||||
|
||||
**请务必**使用对应的部署文件转换相关的模型。
|
||||
|
||||
- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等
|
||||
|
||||
- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32
|
||||
|
||||
- **{static | dynamic}:** 动态、静态 shape
|
||||
|
||||
- **{shape}:** 模型输入的 shape 或者 shape 范围
|
||||
|
||||
### 超分任务模型转换
|
||||
|
||||
以下，我们将演示如何把 `ESRGAN` 转换为 onnx 模型。
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download esrgan model from mmedit model zoo
|
||||
mim download mmedit --config esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k --dest .
|
||||
# convert esrgan model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py \
|
||||
esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py \
|
||||
esrgan_psnr_x4c64b23g32_1x16_1000k_div2k_20200420-bf5c993c.pth \
|
||||
demo/resources/face.png \
|
||||
--work-dir mmdeploy_models/mmedit/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
你也可以把 `ESRGAN` 转为其他后端模型。比如使用`super-resolution/super-resolution_tensorrt-_dynamic-32x32-512x512.py`，把模型转为 tensorrt 模型。
|
||||
|
||||
```{tip}
|
||||
当转 tensorrt 模型时, --device 需要被设置为 "cuda"
|
||||
```
|
||||
|
||||
## 模型规范
|
||||
|
||||
在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。
|
||||
|
||||
上例中的`mmdeploy_models/mmedit/ort`，结构如下：
|
||||
|
||||
```
|
||||
mmdeploy_models/mmedit/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
重要的是：
|
||||
|
||||
- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理
|
||||
- \***.json**: mmdeploy SDK 推理所需的 meta 信息
|
||||
|
||||
整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。
|
||||
|
||||
## 模型推理
|
||||
|
||||
### 后端模型推理
|
||||
|
||||
以上述模型转换后的 `end2end.onnx` 为例，你可以使用如下代码进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmedit/super-resolution/super-resolution_onnxruntime_dynamic.py'
|
||||
model_cfg = 'esrgan_psnr-x4c64b23g32_1xb16-1000k_div2k.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmedit/ort/end2end.onnx']
|
||||
image = './demo/resources/face.png'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_restorer.bmp')
|
||||
```
|
||||
|
||||
### SDK 模型推理
|
||||
|
||||
你也可以参考如下代码，对 SDK model 进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Restorer
|
||||
import cv2
|
||||
|
||||
img = cv2.imread('./demo/resources/face.png')
|
||||
|
||||
# create a classifier
|
||||
restorer = Restorer(model_path='./mmdeploy_models/mmedit/ort', device_name='cpu', device_id=0)
|
||||
# perform inference
|
||||
result = restorer(img)
|
||||
|
||||
# visualize inference result
|
||||
# convert to BGR
|
||||
result = result[..., ::-1]
|
||||
cv2.imwrite('output_restorer.bmp', result)
|
||||
```
|
||||
|
||||
除了python API，mmdeploy SDK 还提供了诸如 C、C++、C#、Java等多语言接口。
|
||||
你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo)学习其他语言接口的使用方法。
|
||||
|
||||
## 模型支持列表
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :----------------------------------------------------------------------------------------------- | :--------------- | :----------: | :------: | :--: | :---: | :------: |
|
||||
| [SRCNN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srcnn) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [ESRGAN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/esrgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [ESRGAN-PSNR](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/esrgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [SRGAN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srresnet_srgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [SRResNet](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/srresnet_srgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [Real-ESRGAN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/real_esrgan) | super-resolution | Y | Y | Y | Y | Y |
|
||||
| [EDSR](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/edsr) | super-resolution | Y | Y | Y | N | Y |
|
||||
| [RDN](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/rdn) | super-resolution | Y | Y | Y | Y | Y |
|
||||
|
|
|
|||
|
|
@ -1,163 +1,246 @@
|
|||
# mmocr 模型支持列表
|
||||
# MMOCR 模型部署
|
||||
|
||||
mmocr 是一个基于 PyTorch 和 mmdetection 的开源工具箱，用于文本检测，文本识别以及相应的下游任务，例如关键信息提取，是 [OpenMMLab](https://openmmlab.com/)项目的一部分。
|
||||
- [安装](#安装)
|
||||
- [安装 mmocr](#安装-mmocr)
|
||||
- [安装 mmdeploy](#安装-mmdeploy)
|
||||
- [模型转换](#模型转换)
|
||||
- [文字检测任务模型转换](#文字检测任务模型转换)
|
||||
- [文字识别任务模型转换](#文字识别任务模型转换)
|
||||
- [模型规范](#模型规范)
|
||||
- [模型推理](#模型推理)
|
||||
- [后端模型推理](#后端模型推理)
|
||||
- [SDK 模型推理](#sdk-模型推理)
|
||||
- [文字检测 SDK 模型推理](#文字检测-sdk-模型推理)
|
||||
- [文字识别 SDK 模型推理](#文字识别-sdk-模型推理)
|
||||
- [模型支持列表](#模型支持列表)
|
||||
|
||||
______________________________________________________________________
|
||||
|
||||
[MMOCR](https://github.com/open-mmlab/mmocr/tree/1.x)，又称 `mmocr`，是基于 PyTorch 和 mmdetection 的开源工具箱，专注于文本检测，文本识别以及相应的下游任务，如关键信息提取。 它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。
|
||||
|
||||
## 安装
|
||||
|
||||
参照 [install.md](https://mmocr.readthedocs.io/en/latest/install.html)。
|
||||
### 安装 mmocr
|
||||
|
||||
## 支持列表
|
||||
请参考[官网安装指南](https://mmocr.readthedocs.io/en/dev-1.x/get_started/install.html).
|
||||
|
||||
| Model | Task | TorchScript | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----- | :--------------- | :---------: | :---------: | :------: | :--: | :---: | :------: | :-----------------------------------------------------------------------------: |
|
||||
| DBNet | text-detection | Y | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/dbnet) |
|
||||
| PSENet | text-detection | Y | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/psenet) |
|
||||
| PANet | text-detection | Y | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/panet) |
|
||||
| CRNN | text-recognition | Y | Y | Y | Y | Y | N | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/crnn) |
|
||||
| SAR | text-recognition | N | Y | N | N | N | N | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/sar) |
|
||||
| SATRN | text-recognition | Y | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/satrn) |
|
||||
### 安装 mmdeploy
|
||||
|
||||
## 注意事项
|
||||
mmdeploy 有以下几种安装方式:
|
||||
|
||||
请注意，ncnn、pplnn 和 OpenVINO 仅支持 DBNet 的 DBNet18 配置。
|
||||
**方式一：** 安装预编译包
|
||||
|
||||
对于在 ICDAR 数据集上预训 [checkpoint](https://download.openmmlab.com/mmocr/textdet/panet/panet_r18_fpem_ffm_sbn_600e_icdar2015_20210219-42dbe46a.pth) 的 PANet，如果要将模型转为具有 fp16 TensorRT，请尝试以下脚本。
|
||||
> 待 mmdeploy 正式发布 dev-1.x，再补充
|
||||
|
||||
```python
|
||||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
from typing import Sequence
|
||||
**方式二：** 一键式脚本安装
|
||||
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
如果部署平台是 **Ubuntu 18.04 及以上版本**， 请参考[脚本安装说明](../01-how-to-build/build_from_script.md)，完成安装过程。
|
||||
比如，以下命令可以安装 mmdeploy 以及配套的推理引擎——`ONNX Runtime`.
|
||||
|
||||
from mmdeploy.core import FUNCTION_REWRITER
|
||||
from mmdeploy.utils.constants import Backend
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
FACTOR = 32
|
||||
ENABLE = False
|
||||
CHANNEL_THRESH = 400
|
||||
**方式三：** 源码安装
|
||||
|
||||
在方式一、二都满足不了的情况下，请参考[源码安装说明](../01-how-to-build/build_from_source.md) 安装 mmdeploy 以及所需推理引擎。
|
||||
|
||||
@FUNCTION_REWRITER.register_rewriter(
|
||||
func_name='mmocr.models.textdet.necks.FPEM_FFM.forward',
|
||||
backend=Backend.TENSORRT.value)
|
||||
def fpem_ffm__forward__trt(ctx, self, x: Sequence[torch.Tensor], *args,
|
||||
**kwargs) -> Sequence[torch.Tensor]:
|
||||
"""Rewrite `forward` of FPEM_FFM for tensorrt backend.
|
||||
## 模型转换
|
||||
|
||||
Rewrite this function avoid overflow for tensorrt-fp16 with the checkpoint
|
||||
`https://download.openmmlab.com/mmocr/textdet/panet/panet_r18_fpem_ffm
|
||||
_sbn_600e_icdar2015_20210219-42dbe46a.pth`
|
||||
你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) 把 mmocr 模型一键式转换为推理后端模型。
|
||||
该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
Args:
|
||||
ctx (ContextCaller): The context with additional information.
|
||||
self: The instance of the class FPEM_FFM.
|
||||
x (List[Tensor]): A list of feature maps of shape (N, C, H, W).
|
||||
|
||||
Returns:
|
||||
outs (List[Tensor]): A list of feature maps of shape (N, C, H, W).
|
||||
"""
|
||||
c2, c3, c4, c5 = x
|
||||
# reduce channel
|
||||
c2 = self.reduce_conv_c2(c2)
|
||||
c3 = self.reduce_conv_c3(c3)
|
||||
c4 = self.reduce_conv_c4(c4)
|
||||
|
||||
if ENABLE:
|
||||
bn_w = self.reduce_conv_c5[1].weight / torch.sqrt(
|
||||
self.reduce_conv_c5[1].running_var + self.reduce_conv_c5[1].eps)
|
||||
bn_b = self.reduce_conv_c5[
|
||||
1].bias - self.reduce_conv_c5[1].running_mean * bn_w
|
||||
bn_w = bn_w.reshape(1, -1, 1, 1).repeat(1, 1, c5.size(2), c5.size(3))
|
||||
bn_b = bn_b.reshape(1, -1, 1, 1).repeat(1, 1, c5.size(2), c5.size(3))
|
||||
conv_b = self.reduce_conv_c5[0].bias.reshape(1, -1, 1, 1).repeat(
|
||||
1, 1, c5.size(2), c5.size(3))
|
||||
c5 = FACTOR * (self.reduce_conv_c5[:-1](c5)) - (FACTOR - 1) * (
|
||||
bn_w * conv_b + bn_b)
|
||||
c5 = self.reduce_conv_c5[-1](c5)
|
||||
else:
|
||||
c5 = self.reduce_conv_c5(c5)
|
||||
|
||||
# FPEM
|
||||
for i, fpem in enumerate(self.fpems):
|
||||
c2, c3, c4, c5 = fpem(c2, c3, c4, c5)
|
||||
if i == 0:
|
||||
c2_ffm = c2
|
||||
c3_ffm = c3
|
||||
c4_ffm = c4
|
||||
c5_ffm = c5
|
||||
else:
|
||||
c2_ffm += c2
|
||||
c3_ffm += c3
|
||||
c4_ffm += c4
|
||||
c5_ffm += c5
|
||||
|
||||
# FFM
|
||||
c5 = F.interpolate(
|
||||
c5_ffm,
|
||||
c2_ffm.size()[-2:],
|
||||
mode='bilinear',
|
||||
align_corners=self.align_corners)
|
||||
c4 = F.interpolate(
|
||||
c4_ffm,
|
||||
c2_ffm.size()[-2:],
|
||||
mode='bilinear',
|
||||
align_corners=self.align_corners)
|
||||
c3 = F.interpolate(
|
||||
c3_ffm,
|
||||
c2_ffm.size()[-2:],
|
||||
mode='bilinear',
|
||||
align_corners=self.align_corners)
|
||||
outs = [c2_ffm, c3, c4, c5]
|
||||
return tuple(outs)
|
||||
|
||||
|
||||
@FUNCTION_REWRITER.register_rewriter(
|
||||
func_name='mmdet.models.backbones.resnet.BasicBlock.forward',
|
||||
backend=Backend.TENSORRT.value)
|
||||
def basic_block__forward__trt(ctx, self, x: torch.Tensor) -> torch.Tensor:
|
||||
"""Rewrite `forward` of BasicBlock for tensorrt backend.
|
||||
|
||||
Rewrite this function avoid overflow for tensorrt-fp16 with the checkpoint
|
||||
`https://download.openmmlab.com/mmocr/textdet/panet/panet_r18_fpem_ffm
|
||||
_sbn_600e_icdar2015_20210219-42dbe46a.pth`
|
||||
|
||||
Args:
|
||||
ctx (ContextCaller): The context with additional information.
|
||||
self: The instance of the class FPEM_FFM.
|
||||
x (Tensor): The input tensor of shape (N, C, H, W).
|
||||
|
||||
Returns:
|
||||
outs (Tensor): The output tensor of shape (N, C, H, W).
|
||||
"""
|
||||
if self.conv1.in_channels < CHANNEL_THRESH:
|
||||
return ctx.origin_func(self, x)
|
||||
|
||||
identity = x
|
||||
|
||||
out = self.conv1(x)
|
||||
out = self.norm1(out)
|
||||
out = self.relu(out)
|
||||
|
||||
out = self.conv2(out)
|
||||
|
||||
if torch.abs(self.norm2(out)).max() < 65504:
|
||||
out = self.norm2(out)
|
||||
out += identity
|
||||
out = self.relu(out)
|
||||
return out
|
||||
else:
|
||||
global ENABLE
|
||||
ENABLE = True
|
||||
# the output of the last bn layer exceeds the range of fp16
|
||||
w1 = self.norm2.weight / torch.sqrt(self.norm2.running_var +
|
||||
self.norm2.eps)
|
||||
bias = self.norm2.bias - self.norm2.running_mean * w1
|
||||
w1 = w1.reshape(1, -1, 1, 1).repeat(1, 1, out.size(2), out.size(3))
|
||||
bias = bias.reshape(1, -1, 1, 1).repeat(1, 1, out.size(2),
|
||||
out.size(3)) + identity
|
||||
out = self.relu(w1 * (out / FACTOR) + bias / FACTOR)
|
||||
|
||||
return out
|
||||
转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmocr)。
|
||||
文件的命名模式是：
|
||||
|
||||
```
|
||||
{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
其中：
|
||||
|
||||
- **{task}:** mmocr 中的任务
|
||||
|
||||
mmdeploy 支持 mmocr 中的文字检测（text detection）、文字识别（text recognition）任务中的模型。关于`模型-任务`的划分，请参考章节[模型支持列表](#模型支持列表)。
|
||||
|
||||
**请务必**使用对应的部署文件转换相关的模型。
|
||||
|
||||
- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等
|
||||
|
||||
- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32
|
||||
|
||||
- **{static | dynamic}:** 动态、静态 shape
|
||||
|
||||
- **{shape}:** 模型输入的 shape 或者 shape 范围
|
||||
|
||||
在接下来来的两个章节，我们将分别演示文字检测任务中的`dbnet`模型，和文字识别任务中的`crnn`模型转换 onnx 模型的方法。
|
||||
|
||||
### 文字检测任务模型转换
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download dbnet model from mmocr model zoo
|
||||
mim download mmocr --config dbnet_resnet18_fpnc_1200e_icdar2015 --dest .
|
||||
# convert mmocr model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py \
|
||||
dbnet_resnet18_fpnc_1200e_icdar2015.py \
|
||||
dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth \
|
||||
demo/resources/text_det.jpg \
|
||||
--work-dir mmdeploy_models/mmocr/dbnet/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
### 文字识别任务模型转换
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download crnn model from mmocr model zoo
|
||||
mim download mmocr --config crnn_mini-vgg_5e_mj --dest .
|
||||
# convert mmocr model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmocr/text-recognition/text-recognition_onnxruntime_dynamic.py \
|
||||
crnn_mini-vgg_5e_mj.py \
|
||||
crnn_mini-vgg_5e_mj_20220826_224120-8afbedbb.pth \
|
||||
demo/resources/text_recog.jpg \
|
||||
--work-dir mmdeploy_models/mmocr/crnn/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
你也可以把它们转为其他后端模型。比如使用`text-detection/text-detection_tensorrt-_dynamic-320x320-2240x2240.py`，把 `dbnet` 模型转为 tensorrt 模型。
|
||||
|
||||
```{tip}
|
||||
当转 tensorrt 模型时, --device 需要被设置为 "cuda"
|
||||
```
|
||||
|
||||
## 模型规范
|
||||
|
||||
在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。
|
||||
|
||||
[文字检测任务模型转换](#文字检测任务模型转换)例子中的`mmdeploy_models/mmocr/dbnet/ort`，结构如下：
|
||||
|
||||
```
|
||||
mmdeploy_models/mmocr/dbnet/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
重要的是：
|
||||
|
||||
- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理
|
||||
- \***.json**: mmdeploy SDK 推理所需的 meta 信息
|
||||
|
||||
整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。
|
||||
|
||||
## 模型推理
|
||||
|
||||
### 后端模型推理
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py'
|
||||
model_cfg = 'dbnet_resnet18_fpnc_1200e_icdar2015.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmocr/dbnet/ort/end2end.onnx']
|
||||
image = './demo/resources/text_det.jpg'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_ocr.png')
|
||||
```
|
||||
|
||||
**提示：**
|
||||
|
||||
在这个脚本中，把'deploy_cfg', 'model_cfg', 'backend_model' and 'image' 替换为[文字识别任务模型转换](#文字识别任务模型转换)中对应的参数，
|
||||
就可以推理 `crnn` onnx 模型了。
|
||||
|
||||
### SDK 模型推理
|
||||
|
||||
#### 文字检测 SDK 模型推理
|
||||
|
||||
你也可以参考如下代码，对 `dbnet` SDK model 进行推理：
|
||||
|
||||
```python
|
||||
import cv2
|
||||
from mmdeploy_python import TextDetector
|
||||
|
||||
img = cv2.imread('demo/resources/text_det.jpg')
|
||||
# create text detector
|
||||
detector = TextDetector(
|
||||
model_path='mmdeploy_models/mmocr/dbnet/ort',
|
||||
device_name='cpu',
|
||||
device_id=0)
|
||||
# do model inference
|
||||
bboxes = detector(img)
|
||||
# draw detected bbox into the input image
|
||||
if len(bboxes) > 0:
|
||||
pts = ((bboxes[:, 0:8] + 0.5).reshape(len(bboxes), -1,
|
||||
2).astype(int))
|
||||
cv2.polylines(img, pts, True, (0, 255, 0), 2)
|
||||
cv2.imwrite('output_ocr.png', img)
|
||||
```
|
||||
|
||||
#### 文字识别 SDK 模型推理
|
||||
|
||||
```python
|
||||
import cv2
|
||||
from mmdeploy_python import TextRecognizer
|
||||
|
||||
img = cv2.imread('demo/resources/text_recog.jpg')
|
||||
# create text recognizer
|
||||
recognizer = TextRecognizer(
|
||||
model_path='mmdeploy_models/mmocr/crnn/ort',
|
||||
device_name='cpu',
|
||||
device_id=0
|
||||
)
|
||||
# do model inference
|
||||
texts = recognizer(img)
|
||||
# print the result
|
||||
print(texts)
|
||||
```
|
||||
|
||||
除了python API，mmdeploy SDK 还提供了诸如 C、C++、C#、Java等多语言接口。
|
||||
你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo)学习其他语言接口的使用方法。
|
||||
|
||||
## 模型支持列表
|
||||
|
||||
| Model | Task | TorchScript | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :----------------------------------------------------------------------------- | :--------------- | :---------: | :---------: | :------: | :--: | :---: | :------: |
|
||||
| [DBNet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/dbnet) | text-detection | Y | Y | Y | Y | Y | Y |
|
||||
| [PSENet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/psenet) | text-detection | Y | Y | Y | Y | N | Y |
|
||||
| [PANet](https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/panet) | text-detection | Y | Y | Y | Y | N | Y |
|
||||
| [CRNN](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/crnn) | text-recognition | Y | Y | Y | Y | Y | N |
|
||||
| [SAR](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/sar) | text-recognition | N | Y | N | N | N | N |
|
||||
| [SATRN](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/satrn) | text-recognition | Y | Y | Y | N | N | N |
|
||||
|
|
|
|||
|
|
@ -1,31 +1,162 @@
|
|||
# mmpose 模型支持列表
|
||||
# MMPose 模型部署
|
||||
|
||||
[mmpose](https://github.com/open-mmlab/mmpose) 是一个基于 PyTorch 的姿态估计的开源工具箱，也是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。
|
||||
- [安装](#安装)
|
||||
- [安装 mmcls](#安装-mmpose)
|
||||
- [安装 mmdeploy](#安装-mmdeploy)
|
||||
- [模型转换](#模型转换)
|
||||
- [模型规范](#模型规范)
|
||||
- [模型推理](#模型推理)
|
||||
- [后端模型推理](#后端模型推理)
|
||||
- [SDK 模型推理](#sdk-模型推理)
|
||||
- [模型支持列表](#模型支持列表)
|
||||
|
||||
## 安装 mmpose
|
||||
______________________________________________________________________
|
||||
|
||||
参照 [official installation guide](https://mmpose.readthedocs.io/en/latest/install.html)。
|
||||
[MMPose](https://github.com/open-mmlab/mmpose/tree/1.x)，又称 `mmpose`，是一个基于 PyTorch 的姿态估计的开源工具箱，是 [OpenMMLab](https://openmmlab.com/) 项目的成员之一。
|
||||
|
||||
## 支持列表
|
||||
## 安装
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :-------- | :------------ | :----------: | :------: | :--: | :---: | :------: | :-----------------------------------------------------------------------------------------: |
|
||||
| HRNet | PoseDetection | Y | Y | Y | N | Y | [config](https://mmpose.readthedocs.io/en/latest/papers/backbones.html#hrnet-cvpr-2019) |
|
||||
| MSPN | PoseDetection | Y | Y | Y | N | Y | [config](https://mmpose.readthedocs.io/en/latest/papers/backbones.html#mspn-arxiv-2019) |
|
||||
| LiteHRNet | PoseDetection | Y | Y | Y | N | Y | [config](https://mmpose.readthedocs.io/en/latest/papers/backbones.html#litehrnet-cvpr-2021) |
|
||||
### 安装 mmpose
|
||||
|
||||
### 使用方法
|
||||
请参考[官网安装指南](https://mmpose.readthedocs.io/en/1.x/installation.html#best-practices)。
|
||||
|
||||
```bash
|
||||
python tools/deploy.py \
|
||||
configs/mmpose/posedetection_tensorrt_static-256x192.py \
|
||||
$MMPOSE_DIR/configs/body/2d_kpt_sview_rgb_img/topdown_heatmap/coco/hrnet_w48_coco_256x192.py \
|
||||
$MMPOSE_DIR/checkpoints/hrnet_w48_coco_256x192-b9e0b3ab_20200708.pth \
|
||||
$MMDEPLOY_DIR/demo/resources/human-pose.jpg \
|
||||
--work-dir work-dirs/mmpose/topdown/hrnet/trt \
|
||||
--device cuda
|
||||
### 安装 mmdeploy
|
||||
|
||||
mmdeploy 有以下几种安装方式:
|
||||
|
||||
**方式一：** 安装预编译包
|
||||
|
||||
> 待 mmdeploy 正式发布 dev-1.x，再补充
|
||||
|
||||
**方式二：** 一键式脚本安装
|
||||
|
||||
如果部署平台是 **Ubuntu 18.04 及以上版本**， 请参考[脚本安装说明](../01-how-to-build/build_from_script.md)，完成安装过程。
|
||||
比如，以下命令可以安装 mmdeploy 以及配套的推理引擎——`ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
注意事项
|
||||
**方式三：** 源码安装
|
||||
|
||||
- mmpose 模型需要额外的输入，但我们无法直接获取它。在导出模型时，可以使用 `$MMDEPLOY_DIR/demo/resources/human-pose.jpg` 作为输入。
|
||||
在方式一、二都满足不了的情况下，请参考[源码安装说明](../01-how-to-build/build_from_source.md) 安装 mmdeploy 以及所需推理引擎。
|
||||
|
||||
## 模型转换
|
||||
|
||||
你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) 把 mmpose 模型一键式转换为推理后端模型。
|
||||
该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
以下，我们将演示如何把 `hrnet` 转换为 onnx 模型。
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
# download hrnet model from mmpose model zoo
|
||||
mim download mmpose --config td-hm_hrnet-w32_8xb64-210e_coco-256x192 --dest .
|
||||
# convert mmdet model to onnxruntime model with static shape
|
||||
python tools/deploy.py \
|
||||
configs/mmpose/pose-detection_onnxruntime_static.py \
|
||||
td-hm_hrnet-w32_8xb64-210e_coco-256x192.py \
|
||||
hrnet_w32_coco_256x192-c78dce93_20200708.pth \
|
||||
demo/resources/human-pose.jpg \
|
||||
--work-dir mmdeploy_models/mmpose/ort \
|
||||
--device cpu \
|
||||
--show
|
||||
```
|
||||
|
||||
转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmpose)。
|
||||
文件的命名模式是：
|
||||
|
||||
```
|
||||
pose-detection_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
其中：
|
||||
|
||||
- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等
|
||||
- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32
|
||||
- **{static | dynamic}:** 动态、静态 shape
|
||||
- **{shape}:** 模型输入的 shape 或者 shape 范围
|
||||
|
||||
在上例中，你也可以把 `hrnet` 转为其他后端模型。比如使用`pose-detection_tensorrt_static-256x192.py`，把模型转为 tensorrt 模型。
|
||||
|
||||
```{tip}
|
||||
当转 tensorrt 模型时, --device 需要被设置为 "cuda"
|
||||
```
|
||||
|
||||
## 模型规范
|
||||
|
||||
在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。
|
||||
|
||||
上例中的`mmdeploy_models/mmpose/ort`，结构如下：
|
||||
|
||||
```
|
||||
mmdeploy_models/mmpose/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
重要的是：
|
||||
|
||||
- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理
|
||||
- \***.json**: mmdeploy SDK 推理所需的 meta 信息
|
||||
|
||||
整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。
|
||||
|
||||
## 模型推理
|
||||
|
||||
### 后端模型推理
|
||||
|
||||
以上述模型转换后的 `end2end.onnx` 为例，你可以使用如下代码进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmpose/pose-detection_onnxruntime_static.py'
|
||||
model_cfg = 'td-hm_hrnet-w32_8xb64-210e_coco-256x192.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmpose/ort/end2end.onnx']
|
||||
image = './demo/resources/human-pose.jpg'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='output_pose.png')
|
||||
```
|
||||
|
||||
### SDK 模型推理
|
||||
|
||||
> TODO
|
||||
|
||||
## 模型支持列表
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO |
|
||||
| :---------------------------------------------------------------------------------------------------- | :------------ | :----------: | :------: | :--: | :---: | :------: |
|
||||
| [HRNet](https://mmpose.readthedocs.io/en/1.x/model_zoo_papers/backbones.html#hrnet-cvpr-2019) | PoseDetection | Y | Y | Y | N | Y |
|
||||
| [MSPN](https://mmpose.readthedocs.io/en/1.x/model_zoo_papers/backbones.html#mspn-arxiv-2019) | PoseDetection | Y | Y | Y | N | Y |
|
||||
| [LiteHRNet](https://mmpose.readthedocs.io/en/1.x/model_zoo_papers/backbones.html#litehrnet-cvpr-2021) | PoseDetection | Y | Y | Y | N | Y |
|
||||
|
|
|
|||
|
|
@ -1,48 +1,33 @@
|
|||
# mmrotate 模型支持列表
|
||||
# MMRotate 模型部署
|
||||
|
||||
[mmrotate](https://github.com/open-mmlab/mmrotate) 是一个基于 PyTorch 的旋转物体检测的开源工具箱，也是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。
|
||||
- [安装](#安装)
|
||||
- [安装 mmcls](#安装-mmrotate)
|
||||
- [安装 mmdeploy](#安装-mmdeploy)
|
||||
- [模型转换](#模型转换)
|
||||
- [模型规范](#模型规范)
|
||||
- [模型推理](#模型推理)
|
||||
- [后端模型推理](#后端模型推理)
|
||||
- [SDK 模型推理](#sdk-模型推理)
|
||||
- [模型支持列表](#模型支持列表)
|
||||
|
||||
## 安装 mmrotate
|
||||
______________________________________________________________________
|
||||
|
||||
参照 [official installation guide](https://mmrotate.readthedocs.io/en/latest/install.html)。
|
||||
[MMRotate](https://github.com/open-mmlab/mmrotate) 是一个基于 PyTorch 的旋转物体检测的开源工具箱，也是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。
|
||||
|
||||
## 支持列表
|
||||
## 安装
|
||||
|
||||
| Model | Task | ONNX Runtime | TensorRT | NCNN | PPLNN | OpenVINO | Model config |
|
||||
| :--------------- | :--------------- | :----------: | :------: | :--: | :---: | :------: | :--------------------------------------------------------------------------------------------: |
|
||||
| RotatedRetinaNet | RotatedDetection | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmrotate/blob/main/configs/rotated_retinanet/README.md) |
|
||||
| Oriented RCNN | RotatedDetection | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmrotate/blob/main/configs/oriented_rcnn/README.md) |
|
||||
| Gliding Vertex | RotatedDetection | N | Y | N | N | N | [config](https://github.com/open-mmlab/mmrotate/blob/main/configs/gliding_vertex/README.md) |
|
||||
| RoI Transformer | RotatedDetection | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmrotate/blob/main/configs/roi_trans/README.md) |
|
||||
### 安装 mmrotate
|
||||
|
||||
### 使用举例
|
||||
### 安装 mmdeploy
|
||||
|
||||
```bash
|
||||
# convert ort
|
||||
python tools/deploy.py \
|
||||
configs/mmrotate/rotated-detection_onnxruntime_dynamic.py \
|
||||
$MMROTATE_DIR/configs/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135.py \
|
||||
$MMROTATE_DIR/checkpoints/rotated_retinanet_obb_r50_fpn_1x_dota_le135-e4131166.pth \
|
||||
$MMROTATE_DIR/demo/demo.jpg \
|
||||
--work-dir work-dirs/mmrotate/rotated_retinanet/ort \
|
||||
--device cpu
|
||||
## 模型转换
|
||||
|
||||
# compute metric
|
||||
python tools/test.py \
|
||||
configs/mmrotate/rotated-detection_onnxruntime_dynamic.py \
|
||||
$MMROTATE_DIR/configs/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135.py \
|
||||
--model work-dirs/mmrotate/rotated_retinanet/ort/end2end.onnx \
|
||||
--metrics mAP
|
||||
## 模型规范
|
||||
|
||||
# generate submit file
|
||||
python tools/test.py \
|
||||
configs/mmrotate/rotated-detection_onnxruntime_dynamic.py \
|
||||
$MMROTATE_DIR/configs/rotated_retinanet/rotated_retinanet_obb_r50_fpn_1x_dota_le135.py \
|
||||
--model work-dirs/mmrotate/rotated_retinanet/ort/end2end.onnx \
|
||||
--format-only \
|
||||
--metric-options submission_dir=work-dirs/mmrotate/rotated_retinanet/ort/Task1_results
|
||||
```
|
||||
## 模型推理
|
||||
|
||||
注意：
|
||||
### 后端模型推理
|
||||
|
||||
- mmrotate 模型需要额外输入，但我们无法直接获取它。在导出模型时，可以使用 `$MMROTATE_DIR/demo/demo.jpg` 作为输入。
|
||||
### SDK 模型推理
|
||||
|
||||
## 模型支持列表
|
||||
|
|
|
|||
|
|
@ -1,48 +1,230 @@
|
|||
# mmseg 模型支持列表
|
||||
# MMSegmentation 模型部署
|
||||
|
||||
mmseg 是一个基于 PyTorch 的开源对象分割工具箱，也是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。
|
||||
- [安装](#安装)
|
||||
- [安装 mmcls](#安装-mmseg)
|
||||
- [安装 mmdeploy](#安装-mmdeploy)
|
||||
- [模型转换](#模型转换)
|
||||
- [模型规范](#模型规范)
|
||||
- [模型推理](#模型推理)
|
||||
- [后端模型推理](#后端模型推理)
|
||||
- [SDK 模型推理](#sdk-模型推理)
|
||||
- [模型支持列表](#模型支持列表)
|
||||
|
||||
## 安装 mmseg
|
||||
______________________________________________________________________
|
||||
|
||||
参照 [get_started.md](https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/get_started.md#installation)。
|
||||
[MMSegmentation](https://github.com/open-mmlab/mmsegmentation/tree/1.x) 又称`mmseg`，是一个基于 PyTorch 的开源对象分割工具箱。它是 [OpenMMLab](https://openmmlab.com/) 项目的一部分。
|
||||
|
||||
## 支持列表
|
||||
## 安装
|
||||
|
||||
| Model | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVino | Model config |
|
||||
| :--------------------------- | :---------: | :------: | :--: | :---: | :------: | :--------------------------------------------------------------------------------------: |
|
||||
| FCN | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fcn) |
|
||||
| PSPNet[\*](#static_shape) | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/pspnet) |
|
||||
| DeepLabV3 | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/deeplabv3) |
|
||||
| DeepLabV3+ | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/deeplabv3plus) |
|
||||
| Fast-SCNN[\*](#static_shape) | Y | Y | N | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fastscnn) |
|
||||
| UNet | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/unet) |
|
||||
| ANN[\*](#static_shape) | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ann) |
|
||||
| APCNet | Y | Y | Y | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/apcnet) |
|
||||
| BiSeNetV1 | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv1) |
|
||||
| BiSeNetV2 | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv2) |
|
||||
| CGNet | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/cgnet) |
|
||||
| DMNet | Y | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dmnet) |
|
||||
| DNLNet | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dnlnet) |
|
||||
| EMANet | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/emanet) |
|
||||
| EncNet | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/encnet) |
|
||||
| ERFNet | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/erfnet) |
|
||||
| FastFCN | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fastfcn) |
|
||||
| GCNet | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/gcnet) |
|
||||
| ICNet[\*](#static_shape) | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/icnet) |
|
||||
| ISANet[\*](#static_shape) | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/isanet) |
|
||||
| NonLocal Net | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/nonlocal_net) |
|
||||
| OCRNet | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ocrnet) |
|
||||
| PointRend[\*](#static_shape) | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/point_rend) |
|
||||
| Semantic FPN | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/sem_fpn) |
|
||||
| STDC | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/stdc) |
|
||||
| UPerNet[\*](#static_shape) | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/upernet) |
|
||||
| DANet | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/danet) |
|
||||
| Segmenter[\*](#static_shape) | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segmenter) |
|
||||
| SegFormer[\*](#static_shape) | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segformer) |
|
||||
| SETR | Y | N | N | N | Y | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/setr) |
|
||||
| CCNet | N | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ccnet) |
|
||||
| PSANet | N | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/psanet) |
|
||||
| DPT | N | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dpt) |
|
||||
### 安装 mmseg
|
||||
|
||||
请参考[官网安装指南](https://mmsegmentation.readthedocs.io/en/1.x/get_started.html)。
|
||||
|
||||
### 安装 mmdeploy
|
||||
|
||||
mmdeploy 有以下几种安装方式:
|
||||
|
||||
**方式一：** 安装预编译包
|
||||
|
||||
> 待 mmdeploy 正式发布 dev-1.x，再补充
|
||||
|
||||
**方式二：** 一键式脚本安装
|
||||
|
||||
如果部署平台是 **Ubuntu 18.04 及以上版本**， 请参考[脚本安装说明](../01-how-to-build/build_from_script.md)，完成安装过程。
|
||||
比如，以下命令可以安装 mmdeploy 以及配套的推理引擎——`ONNX Runtime`.
|
||||
|
||||
```shell
|
||||
git clone --recursive -b dev-1.x https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
|
||||
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
|
||||
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
**说明**:
|
||||
|
||||
- 把 `$(pwd)/build/lib` 添加到 `PYTHONPATH`，目的是为了加载 mmdeploy SDK python 包 `mmdeploy_python`，在章节 [SDK模型推理](#sdk模型推理)中讲述其用法。
|
||||
- 在[使用 ONNX Runtime推理后端模型](#后端模型推理)时，需要加载自定义算子库，需要把 ONNX Runtime 库的路径加入环境变量 `LD_LIBRARY_PATH`中。
|
||||
**方式三：** 源码安装
|
||||
|
||||
在方式一、二都满足不了的情况下，请参考[源码安装说明](../01-how-to-build/build_from_source.md) 安装 mmdeploy 以及所需推理引擎。
|
||||
|
||||
## 模型转换
|
||||
|
||||
你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/dev-1.x/tools/deploy.py) 把 mmseg 模型一键式转换为推理后端模型。
|
||||
该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage).
|
||||
|
||||
以下，我们将演示如何把 `unet` 转换为 onnx 模型。
|
||||
|
||||
```shell
|
||||
cd mmdeploy
|
||||
|
||||
# download unet model from mmseg model zoo
|
||||
mim download mmsegmentation --config unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024 --dest .
|
||||
|
||||
# convert mmseg model to onnxruntime model with dynamic shape
|
||||
python tools/deploy.py \
|
||||
configs/mmseg/segmentation_onnxruntime_dynamic.py \
|
||||
unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py \
|
||||
fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes_20211210_145204-6860854e.pth \
|
||||
demo/resources/cityscapes.png \
|
||||
--work-dir mmdeploy_models/mmseg/ort \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/configs/mmseg)。
|
||||
文件的命名模式是：
|
||||
|
||||
```
|
||||
segmentation_{backend}-{precision}_{static | dynamic}_{shape}.py
|
||||
```
|
||||
|
||||
其中：
|
||||
|
||||
- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等
|
||||
- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32
|
||||
- **{static | dynamic}:** 动态、静态 shape
|
||||
- **{shape}:** 模型输入的 shape 或者 shape 范围
|
||||
|
||||
在上例中，你也可以把 `unet` 转为其他后端模型。比如使用`segmentation_tensorrt-fp16_dynamic-512x1024-2048x2048.py`，把模型转为 tensorrt-fp16 模型。
|
||||
|
||||
```{tip}
|
||||
当转 tensorrt 模型时, --device 需要被设置为 "cuda"
|
||||
```
|
||||
|
||||
## 模型规范
|
||||
|
||||
在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。
|
||||
|
||||
上例中的`mmdeploy_models/mmseg/ort`，结构如下：
|
||||
|
||||
```
|
||||
mmdeploy_models/mmseg/ort
|
||||
├── deploy.json
|
||||
├── detail.json
|
||||
├── end2end.onnx
|
||||
└── pipeline.json
|
||||
```
|
||||
|
||||
重要的是：
|
||||
|
||||
- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理
|
||||
- \***.json**: mmdeploy SDK 推理所需的 meta 信息
|
||||
|
||||
整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。
|
||||
|
||||
## 模型推理
|
||||
|
||||
### 后端模型推理
|
||||
|
||||
以上述模型转换后的 `end2end.onnx` 为例，你可以使用如下代码进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy.apis.utils import build_task_processor
|
||||
from mmdeploy.utils import get_input_shape, load_config
|
||||
import torch
|
||||
|
||||
deploy_cfg = 'configs/mmseg/segmentation_onnxruntime_dynamic.py'
|
||||
model_cfg = './unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py'
|
||||
device = 'cpu'
|
||||
backend_model = ['./mmdeploy_models/mmseg/ort/end2end.onnx']
|
||||
image = './demo/resources/cityscapes.png'
|
||||
|
||||
# read deploy_cfg and model_cfg
|
||||
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)
|
||||
|
||||
# build task and backend model
|
||||
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
|
||||
model = task_processor.build_backend_model(backend_model)
|
||||
|
||||
# process input image
|
||||
input_shape = get_input_shape(deploy_cfg)
|
||||
model_inputs, _ = task_processor.create_input(image, input_shape)
|
||||
|
||||
# do model inference
|
||||
with torch.no_grad():
|
||||
result = model.test_step(model_inputs)
|
||||
|
||||
# visualize results
|
||||
task_processor.visualize(
|
||||
image=image,
|
||||
model=model,
|
||||
result=result[0],
|
||||
window_name='visualize',
|
||||
output_file='./output_segmentation.png')
|
||||
```
|
||||
|
||||
### SDK 模型推理
|
||||
|
||||
你也可以参考如下代码，对 SDK model 进行推理：
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Segmentor
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
img = cv2.imread('./demo/resources/cityscapes.png')
|
||||
|
||||
# create a classifier
|
||||
segmentor = Segmentor(model_path='./mmdeploy_models/mmseg/ort', device_name='cpu', device_id=0)
|
||||
# perform inference
|
||||
seg = segmentor(img)
|
||||
|
||||
# visualize inference result
|
||||
## random a palette with size 256x3
|
||||
palette = np.random.randint(0, 256, size=(256, 3))
|
||||
color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8)
|
||||
for label, color in enumerate(palette):
|
||||
color_seg[seg == label, :] = color
|
||||
# convert to BGR
|
||||
color_seg = color_seg[..., ::-1]
|
||||
img = img * 0.5 + color_seg * 0.5
|
||||
img = img.astype(np.uint8)
|
||||
cv2.imwrite('output_segmentation.png', img)
|
||||
```
|
||||
|
||||
除了python API，mmdeploy SDK 还提供了诸如 C、C++、C#、Java等多语言接口。
|
||||
你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/dev-1.x/demo)学习其他语言接口的使用方法。
|
||||
|
||||
## 模型支持列表
|
||||
|
||||
| Model | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVino |
|
||||
| :---------------------------------------------------------------------------------------------------------- | :---------: | :------: | :--: | :---: | :------: |
|
||||
| [FCN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fcn) | Y | Y | Y | Y | Y |
|
||||
| [PSPNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/pspnet)[\*](#static_shape) | Y | Y | Y | Y | Y |
|
||||
| [DeepLabV3](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/deeplabv3) | Y | Y | Y | Y | Y |
|
||||
| [DeepLabV3+](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/deeplabv3plus) | Y | Y | Y | Y | Y |
|
||||
| [Fast-SCNN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fastscnn)[\*](#static_shape) | Y | Y | N | Y | Y |
|
||||
| [UNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/unet) | Y | Y | Y | Y | Y |
|
||||
| [ANN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ann)[\*](#static_shape) | Y | Y | N | N | N |
|
||||
| [APCNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/apcnet) | Y | Y | Y | N | N |
|
||||
| [BiSeNetV1](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv1) | Y | Y | Y | N | Y |
|
||||
| [BiSeNetV2](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/bisenetv2) | Y | Y | Y | N | Y |
|
||||
| [CGNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/cgnet) | Y | Y | Y | N | Y |
|
||||
| [DMNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dmnet) | Y | N | N | N | N |
|
||||
| [DNLNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dnlnet) | Y | Y | Y | N | Y |
|
||||
| [EMANet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/emanet) | Y | Y | N | N | Y |
|
||||
| [EncNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/encnet) | Y | Y | N | N | Y |
|
||||
| [ERFNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/erfnet) | Y | Y | Y | N | Y |
|
||||
| [FastFCN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/fastfcn) | Y | Y | Y | N | Y |
|
||||
| [GCNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/gcnet) | Y | Y | N | N | N |
|
||||
| [ICNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/icnet)[\*](#static_shape) | Y | Y | N | N | Y |
|
||||
| [ISANet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/isanet)[\*](#static_shape) | Y | Y | N | N | Y |
|
||||
| [NonLocal Net](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/nonlocal_net) | Y | Y | Y | N | Y |
|
||||
| [OCRNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ocrnet) | Y | Y | Y | N | Y |
|
||||
| [PointRend](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/point_rend)[\*](#static_shape) | Y | Y | N | N | N |
|
||||
| [Semantic FPN](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/sem_fpn) | Y | Y | Y | N | Y |
|
||||
| [STDC](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/stdc) | Y | Y | Y | N | Y |
|
||||
| [UPerNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/upernet)[\*](#static_shape) | Y | Y | N | N | N |
|
||||
| [DANet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/danet) | Y | Y | N | N | Y |
|
||||
| [Segmenter](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segmenter)[\*](#static_shape) | Y | Y | Y | N | Y |
|
||||
| [SegFormer](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/segformer)[\*](#static_shape) | Y | Y | N | N | Y |
|
||||
| [SETR](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/setr) | Y | N | N | N | Y |
|
||||
| [CCNet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/ccnet) | N | N | N | N | N |
|
||||
| [PSANet](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/psanet) | N | N | N | N | N |
|
||||
| [DPT](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dpt) | N | N | N | N | N |
|
||||
|
||||
## 注意事项
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue