mmsegmentation/docs/en/user_guides/5_deployment.md

Tutorial 5: Model Deployment

MMSegmentation Model Deployment

• Tutorial 5: Model Deployment
• MMSegmentation Model Deployment
  • Installation
    • Install mmseg
    • Install mmdeploy
  • Convert model
  • Model specification
  • Model inference
    • Backend model inference
    • SDK model inference
  • Supported models
  • Note

---

MMSegmentation, also known as mmseg, is an open source semantic segmentation toolbox based on Pytorch. It's a part of the OpenMMLab object.

Installation

Install mmseg

Please follow the Installation Guide.

Install mmdeploy

mmdeploy can be installed as follows:

Option 1: Install precompiled package

Please follow the Installation overview

Option 2: Automatic Installation script

If the deployment platform is Ubuntu 18.04 +, please follow the scription installation to install. For example, the following commands describe how to install mmdeploy and inference engine-ONNX Runtime.

git clone --recursive -b main https://github.com/open-mmlab/mmdeploy.git
cd mmdeploy
python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc)
export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH
export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH

NOTE:

• Add $(pwd)/build/lib to PYTHONPATH, can loading mmdeploy SDK python package mmdeploy_runtime. See SDK model inference for more information.
• With ONNX Runtime model inference, need to load custom operator library and add ONNX Runtime Library's PATH to LD_LIBRARY_PATH.

Option 3: Install with mim

1. Use mim to install mmcv

pip install -U openmim
mim install "mmcv>=2.0.0rc2"

2. Install mmdeploy

git clone https://github.com/open-mmlab/mmdeploy.git
cd mmdeploy
mim install -e .

Option 4: Build MMDeploy from source

If the first three methods aren't suitable, please Build MMDeploy from source

Convert model

tools/deploy.py can convert mmseg Model to backend model conveniently. See this for detailed information.

Then convert unet to onnx model as follows:

cd mmdeploy

# download unet model from mmseg model zoo
mim download mmsegmentation --config unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024 --dest .

# convert mmseg model to onnxruntime model with dynamic shape
python tools/deploy.py \
    configs/mmseg/segmentation_onnxruntime_dynamic.py \
    unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py \
    fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes_20211210_145204-6860854e.pth \
    demo/resources/cityscapes.png \
    --work-dir mmdeploy_models/mmseg/ort \
    --device cpu \
    --show \
    --dump-info

It is crucial to specify the correct deployment config during model conversion. MMDeploy has already provided builtin deployment config files of all supported backends for mmsegmentation, under which the config file path follows the pattern:

segmentation_{backend}-{precision}_{static | dynamic}_{shape}.py

• {backend}: inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc.
• {precision}: fp16, int8. When it's empty, it means fp32
• {static | dynamic}: static shape or dynamic shape
• {shape}: input shape or shape range of a model

Therefore, in the above example, you can also convert unet to tensorrt-fp16 model by segmentation_tensorrt-fp16_dynamic-512x1024-2048x2048.py.

When converting mmsegmentation models to tensorrt models, --device should be set to "cuda"

Model specification

Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference.

The converted model locates in the working directory like mmdeploy_models/mmseg/ort in the previous example. It includes:

mmdeploy_models/mmseg/ort
├── deploy.json
├── detail.json
├── end2end.onnx
└── pipeline.json

in which,

• end2end.onnx: backend model which can be inferred by ONNX Runtime
• xxx.json: the necessary information for mmdeploy SDK

The whole package mmdeploy_models/mmseg/ort is defined as mmdeploy SDK model, i.e., mmdeploy SDK model includes both backend model and inference meta information.

Model inference

Backend model inference

Take the previous converted end2end.onnx model as an example, you can use the following code to inference the model and visualize the results:

from mmdeploy.apis.utils import build_task_processor
from mmdeploy.utils import get_input_shape, load_config
import torch

deploy_cfg = 'configs/mmseg/segmentation_onnxruntime_dynamic.py'
model_cfg = './unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py'
device = 'cpu'
backend_model = ['./mmdeploy_models/mmseg/ort/end2end.onnx']
image = './demo/resources/cityscapes.png'

# read deploy_cfg and model_cfg
deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg)

# build task and backend model
task_processor = build_task_processor(model_cfg, deploy_cfg, device)
model = task_processor.build_backend_model(backend_model)

# process input image
input_shape = get_input_shape(deploy_cfg)
model_inputs, _ = task_processor.create_input(image, input_shape)

# do model inference
with torch.no_grad():
    result = model.test_step(model_inputs)

# visualize results
task_processor.visualize(
    image=image,
    model=model,
    result=result[0],
    window_name='visualize',
    output_file='./output_segmentation.png')

SDK model inference

You can also perform SDK model inference like following:

from mmdeploy_runtime import Segmentor
import cv2
import numpy as np

img = cv2.imread('./demo/resources/cityscapes.png')

# create a classifier
segmentor = Segmentor(model_path='./mmdeploy_models/mmseg/ort', device_name='cpu', device_id=0)
# perform inference
seg = segmentor(img)

# visualize inference result
## random a palette with size 256x3
palette = np.random.randint(0, 256, size=(256, 3))
color_seg = np.zeros((seg.shape[0], seg.shape[1], 3), dtype=np.uint8)
for label, color in enumerate(palette):
    color_seg[seg == label, :] = color
# convert to BGR
color_seg = color_seg[..., ::-1]
img = img * 0.5 + color_seg * 0.5
img = img.astype(np.uint8)
cv2.imwrite('output_segmentation.png', img)

Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from demo

Supported models

Model	TorchScript	OnnxRuntime	TensorRT	ncnn	PPLNN	OpenVino
FCN	Y	Y	Y	Y	Y	Y
PSPNet*	Y	Y	Y	Y	Y	Y
DeepLabV3	Y	Y	Y	Y	Y	Y
DeepLabV3+	Y	Y	Y	Y	Y	Y
Fast-SCNN*	Y	Y	Y	N	Y	Y
UNet	Y	Y	Y	Y	Y	Y
ANN*	Y	Y	Y	N	N	N
APCNet	Y	Y	Y	Y	N	N
BiSeNetV1	Y	Y	Y	Y	N	Y
BiSeNetV2	Y	Y	Y	Y	N	Y
CGNet	Y	Y	Y	Y	N	Y
DMNet	?	Y	N	N	N	N
DNLNet	?	Y	Y	Y	N	Y
EMANet	Y	Y	Y	N	N	Y
EncNet	Y	Y	Y	N	N	Y
ERFNet	Y	Y	Y	Y	N	Y
FastFCN	Y	Y	Y	Y	N	Y
GCNet	Y	Y	Y	N	N	N
ICNet*	Y	Y	Y	N	N	Y
ISANet*	N	Y	Y	N	N	Y
NonLocal Net	?	Y	Y	Y	N	Y
OCRNet	Y	Y	Y	Y	N	Y
PointRend*	Y	Y	Y	N	N	N
Semantic FPN	Y	Y	Y	Y	N	Y
STDC	Y	Y	Y	Y	N	Y
UPerNet*	N	Y	Y	N	N	N
DANet	?	Y	Y	N	N	Y
Segmenter*	N	Y	Y	Y	N	Y
SegFormer*	?	Y	Y	N	N	Y
SETR	?	Y	N	N	N	Y
CCNet	?	N	N	N	N	N
PSANet	?	N	N	N	N	N
DPT	?	N	N	N	N	N

Note

• All mmseg models only support the 'whole' inference mode.

• PSPNet，Fast-SCNN only supports static input, because most inference framework's nn.AdaptiveAvgPool2d don't support dynamic input。

• For models that only support static shapes, should use the static shape deployment config file, such as configs/mmseg/segmentation_tensorrt_static-1024x2048.py

• To deploy models to generate probabilistic feature maps, please add codebase_config = dict(with_argmax=False) to deployment config file.