[Docs] reorganize the documents in English and update the contents based on v0.5.0 status (#531)
* checkout qq group qrcode * update the cover image * update build doc * reorganize chapters * update readme * remove index of build on different platforms in readthedocs * update benchmark * update get started document in Chinese based on the prebuild package * update get_started * re-style benchmark * update get_started in zh_cn * update get_started in english * update get_started in english * update get_started in english * update get_started doc * update according to reviewer comments * update linker ci * fix(.github/scripts/check_doc_linker.py): skip code block * specify PYTHONPATH * update get_started * update diagram * rename some documents * fix according to reviewer comments Co-authored-by: tpoisonooo <khj.application@aliyun.com>pull/560/head
lvhan028
GitHub
parent
908a69b7d0
commit
e929201827
|
|
@ -24,9 +24,18 @@ pattern = re.compile(r'\[.*?\]\(.*?\)')
|
|||
def analyze_doc(home, path):
|
||||
print('analyze {}'.format(path))
|
||||
problem_list = []
|
||||
code_block = False
|
||||
with open(path) as f:
|
||||
lines = f.readlines()
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
if line.startswith('```'):
|
||||
code_block = not code_block
|
||||
continue
|
||||
|
||||
if code_block is True:
|
||||
continue
|
||||
|
||||
if '[' in line and ']' in line and '(' in line and ')' in line:
|
||||
all = pattern.findall(line)
|
||||
for item in all:
|
||||
|
|
|
|||
|
|
@ -16,3 +16,5 @@ jobs:
|
|||
python-version: '3.7'
|
||||
- run: python3 .github/scripts/doc_link_checker.py --target docs/zh_cn
|
||||
- run: python3 .github/scripts/doc_link_checker.py --target README_zh-CN.md
|
||||
- run: python3 .github/scripts/doc_link_checker.py --target docs/en
|
||||
- run: python3 .github/scripts/doc_link_checker.py --target README.md
|
||||
|
|
|
|||
92
README.md
92
README.md
|
|
@ -23,82 +23,88 @@
|
|||
[](https://github.com/open-mmlab/mmdeploy/actions)
|
||||
[](https://codecov.io/gh/open-mmlab/mmdeploy)
|
||||
[](https://github.com/open-mmlab/mmdeploy/blob/master/LICENSE)
|
||||
[](https://github.com/open-mmlab/mmdeploy/issues)
|
||||
[](https://github.com/open-mmlab/mmdeploy/issues)
|
||||
[](https://github.com/open-mmlab/mmdeploy/issues)
|
||||
[](https://github.com/open-mmlab/mmdeploy/issues)
|
||||
|
||||
## Introduction
|
||||
|
||||
English | [简体中文](README_zh-CN.md)
|
||||
|
||||
MMDeploy is an open-source deep learning model deployment toolset. It is
|
||||
a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
## Introduction
|
||||
|
||||
MMDeploy is an open-source deep learning model deployment toolset. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
<div align="center">
|
||||
<img src="resources/introduction.png" width="800"/>
|
||||
<img src="resources/introduction.png">
|
||||
</div>
|
||||
|
||||
|
||||
### Major features
|
||||
## Main features
|
||||
|
||||
- **Fully support OpenMMLab models**
|
||||
### Fully support OpenMMLab models
|
||||
|
||||
We provide a unified model deployment toolbox for the codebases in OpenMMLab. The supported codebases are listed as below, and more will be added in the future
|
||||
- [x] MMClassification
|
||||
- [x] MMDetection
|
||||
- [x] MMSegmentation
|
||||
- [x] MMEditing
|
||||
- [x] MMOCR
|
||||
- [x] MMPose
|
||||
The currently supported codebases and models are as follows, and more will be included in the future
|
||||
- [mmcls](docs/en/04-supported-codebases/mmcls.md)
|
||||
- [mmdet](docs/en/04-supported-codebases/mmdet.md)
|
||||
- [mmseg](docs/en/04-supported-codebases/mmseg.md)
|
||||
- [mmedit](docs/en/04-supported-codebases/mmedit.md)
|
||||
- [mmocr](docs/en/04-supported-codebases/mmocr.md)
|
||||
- [mmpose](docs/en/04-supported-codebases/mmpose.md)
|
||||
- [mmdet3d](docs/en/04-supported-codebases/mmdet3d.md)
|
||||
- [mmrotate](docs/en/04-supported-codebases/mmrotate.md)
|
||||
|
||||
- **Multiple inference backends are available**
|
||||
### Multiple inference backends are available
|
||||
|
||||
Models can be exported and run in different backends. The following ones are supported, and more will be taken into consideration
|
||||
- [x] ONNX Runtime
|
||||
- [x] TensorRT
|
||||
- [x] PPLNN
|
||||
- [x] ncnn
|
||||
- [x] OpenVINO
|
||||
Models can be exported and run in the following backends, and more will be compatible
|
||||
|
||||
- **Efficient and highly scalable SDK Framework by C/C++**
|
||||
| ONNX Runtime | TensorRT | ppl.nn | ncnn | OpenVINO | LibTorch | more |
|
||||
| ------------ | -------- | ------ | ---- | -------- | -------- | ---------------------------------------------- |
|
||||
| ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | [benchmark](docs/en/03-benchmark/benchmark.md) |
|
||||
|
||||
All kinds of modules in SDK can be extensible, such as `Transform` for image processing, `Net` for Neural Network inference, `Module` for postprocessing and so on
|
||||
|
||||
## License
|
||||
|
||||
This project is released under the [Apache 2.0 license](LICENSE).
|
||||
### Efficient and scalable C/C++ SDK Framework
|
||||
|
||||
## Installation
|
||||
All kinds of modules in the SDK can be extended, such as `Transform` for image processing, `Net` for Neural Network inference, `Module` for postprocessing and so on
|
||||
|
||||
Please refer to [build.md](https://mmdeploy.readthedocs.io/en/latest/build.html) for installation.
|
||||
## Get Started
|
||||
|
||||
## Getting Started
|
||||
Please read [getting_started.md](docs/en/get_started.md) for the basic usage of MMDeploy. We also provide tutoials about:
|
||||
|
||||
Please see [getting_started.md](https://mmdeploy.readthedocs.io/en/latest/get_started.html) for the basic usage of MMDeploy. We also provide other tutorials for:
|
||||
* [Build](docs/en/01-how-to-build/build_from_source.md)
|
||||
* [Build from Docker](docs/en/01-how-to-build/build_from_docker.md)
|
||||
* [Build for Linux](docs/en/01-how-to-build/linux-x86_64.md)
|
||||
* [Build for Win10](docs/en/01-how-to-build/windows.md)
|
||||
* [Build for Android](docs/en/01-how-to-build/android.md)
|
||||
* [Build for Jetson](docs/en/01-how-to-build/jetsons.md)
|
||||
* User Guide
|
||||
* [How to convert model](docs/en/02-how-to-run/convert_model.md)
|
||||
* [How to write config](docs/en/02-how-to-run/write_config.md)
|
||||
* [How to evaluate deployed models](docs/en/02-how-to-run/how_to_evaluate_a_model.md)
|
||||
* [How to measure performance of deployed models](docs/en/02-how-to-run/how_to_measure_performance_of_models.md)
|
||||
* Developer Guide
|
||||
* [How to support new models](docs/en/06-developer-guide/support_new_model.md)
|
||||
* [How to support new backends](docs/en/06-developer-guide/support_new_backend.md)
|
||||
* [FAQ](docs/en/faq.md)
|
||||
* [Contributing](.github/CONTRIBUTING.md)
|
||||
|
||||
- [how to convert model](https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_convert_model.html)
|
||||
- [how to write config](https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_write_config.html)
|
||||
- [how to support new models](https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_support_new_models.html)
|
||||
- [how to measure performance of models](https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_measure_performance_of_models.html)
|
||||
## Benchmark and Model zoo
|
||||
|
||||
Please refer to [FAQ](https://mmdeploy.readthedocs.io/en/latest/faq.html) for frequently asked questions.
|
||||
|
||||
## Benchmark and model zoo
|
||||
|
||||
Results and supported model list are available in the [benchmark](https://mmdeploy.readthedocs.io/en/latest/benchmark.html) and [model list](https://mmdeploy.readthedocs.io/en/latest/supported_models.html).
|
||||
You can find the supported models from [here](docs/en/03-benchmark/supported_models.md) and their performance in the [benchmark](docs/en/03-benchmark/benchmark.md).
|
||||
|
||||
## Contributing
|
||||
|
||||
We appreciate all contributions to improve MMDeploy. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.
|
||||
We appreciate all contributions to MMDeploy. Please refer to [CONTRIBUTING.md](.github/CONTRIBUTING.md) for the contributing guideline.
|
||||
|
||||
## Acknowledgement
|
||||
|
||||
We would like to sincerely thank the following teams for their contributions to [MMDeploy](https://github.com/open-mmlab/mmdeploy):
|
||||
- [OpenPPL](https://github.com/openppl-public)
|
||||
- [OpenVINO](https://github.com/openvinotoolkit/openvino)
|
||||
- [ncnn](https://github.com/Tencent/ncnn)
|
||||
|
||||
## Citation
|
||||
|
||||
If you find this project useful in your research, please consider cite:
|
||||
If you find this project useful in your research, please consider citing:
|
||||
|
||||
```BibTeX
|
||||
@misc{=mmdeploy,
|
||||
|
|
@ -109,6 +115,10 @@ If you find this project useful in your research, please consider cite:
|
|||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
This project is released under the [Apache 2.0 license](LICENSE).
|
||||
|
||||
## Projects in OpenMMLab
|
||||
|
||||
- [MMCV](https://github.com/open-mmlab/mmcv): OpenMMLab foundational library for computer vision.
|
||||
|
|
|
|||
|
|
@ -23,13 +23,13 @@
|
|||
[](https://github.com/open-mmlab/mmdeploy/actions)
|
||||
[](https://codecov.io/gh/open-mmlab/mmdeploy)
|
||||
[](https://github.com/open-mmlab/mmdeploy/blob/master/LICENSE)
|
||||
[](https://github.com/open-mmlab/mmdeploy/issues)
|
||||
[](https://github.com/open-mmlab/mmdeploy/issues)
|
||||
|
||||
## 介绍
|
||||
[](https://github.com/open-mmlab/mmdeploy/issues)
|
||||
[](https://github.com/open-mmlab/mmdeploy/issues)
|
||||
|
||||
[English](README.md) | 简体中文
|
||||
|
||||
## 介绍
|
||||
|
||||
MMDeploy 是 [OpenMMLab](https://openmmlab.com/) 模型部署工具箱，**为各算法库提供统一的部署体验**。基于 MMDeploy，开发者可以轻松从训练 repo 生成指定硬件所需 SDK，省去大量适配时间。
|
||||
|
||||
## 架构简析
|
||||
|
|
@ -41,37 +41,33 @@ MMDeploy 是 [OpenMMLab](https://openmmlab.com/) 模型部署工具箱，**为
|
|||
## 特性简介
|
||||
|
||||
### 支持超多 OpenMMLab 算法库
|
||||
- mmcls
|
||||
- mmdet
|
||||
- mmdet3d
|
||||
- mmedit
|
||||
- mmocr
|
||||
- mmpose
|
||||
- mmseg
|
||||
- [支持列表](docs/zh_cn/03-benchmark/supported_models.md)
|
||||
- [mmcls](docs/en/04-supported-codebases/mmcls.md)
|
||||
- [mmdet](docs/en/04-supported-codebases/mmdet.md)
|
||||
- [mmseg](docs/en/04-supported-codebases/mmseg.md)
|
||||
- [mmedit](docs/en/04-supported-codebases/mmedit.md)
|
||||
- [mmocr](docs/en/04-supported-codebases/mmocr.md)
|
||||
- [mmpose](docs/en/04-supported-codebases/mmpose.md)
|
||||
- [mmdet3d](docs/en/04-supported-codebases/mmdet3d.md)
|
||||
- [mmrotate](docs/en/04-supported-codebases/mmrotate.md)
|
||||
|
||||
### 支持多种推理后端
|
||||
|ONNX Runtime|TensorRT|ppl.nn|ncnn|OpenVINO|more|
|
||||
|------------|--------|------|----|--------|----|
|
||||
|✔️ |✔️ |✔️ |✔️ |✔️ |[benchmark](docs/zh_cn/03-benchmark/benchmark.md)|
|
||||
| ONNX Runtime | TensorRT | ppl.nn | ncnn | OpenVINO | more |
|
||||
| ------------ | -------- | ------ | ---- | -------- | ------------------------------------------------- |
|
||||
| ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | [benchmark](docs/zh_cn/03-benchmark/benchmark.md) |
|
||||
|
||||
### SDK 可高度定制化
|
||||
- Transform 数据预处理
|
||||
- Net 推理
|
||||
- Module 后处理
|
||||
|
||||
## 开源许可证
|
||||
|
||||
该项目采用 [Apache 2.0 开源许可证](LICENSE)。
|
||||
|
||||
## [快速上手](docs/zh_cn/get_started.md)
|
||||
|
||||
* [编译](docs/zh_cn/01-how-to-build/build_from_source.md)
|
||||
* [Build for Android](docs/zh_cn/01-how-to-build/android.md)
|
||||
* [Build from Docker](docs/zh_cn/01-how-to-build/build_from_docker.md)
|
||||
* [Build for Linux](docs/zh_cn/01-how-to-build/linux-x86_64.md)
|
||||
* [Build for Win10](docs/zh_cn/01-how-to-build/windows.md)
|
||||
* [Build for Jetson](docs/en/tutorials/how_to_install_mmdeploy_on_jetsons.md)
|
||||
* [Build from Docker](docs/zh_cn/01-how-to-build/build_from_docker.md)
|
||||
* [Build for Android](docs/zh_cn/01-how-to-build/android.md)
|
||||
* [Build for Jetson](docs/en/01-how-to-build/jetsons.md)
|
||||
* 使用
|
||||
* [把模型转换到推理 Backend](docs/zh_cn/02-how-to-run/convert_model.md)
|
||||
* [配置转换参数](docs/zh_cn/02-how-to-run/write_config.md)
|
||||
|
|
@ -114,6 +110,10 @@ MMDeploy 是 [OpenMMLab](https://openmmlab.com/) 模型部署工具箱，**为
|
|||
}
|
||||
```
|
||||
|
||||
## 开源许可证
|
||||
|
||||
该项目采用 [Apache 2.0 开源许可证](LICENSE)。
|
||||
|
||||
## OpenMMLab 的其他项目
|
||||
|
||||
- [MMCV](https://github.com/open-mmlab/mmcv): OpenMMLab 计算机视觉基础库
|
||||
|
|
@ -138,11 +138,12 @@ MMDeploy 是 [OpenMMLab](https://openmmlab.com/) 模型部署工具箱，**为
|
|||
|
||||
## 欢迎加入 OpenMMLab 社区
|
||||
|
||||
扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的 [官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=aCvMxdr3)
|
||||
扫描下方的二维码可关注 OpenMMLab 团队的 [知乎官方账号](https://www.zhihu.com/people/openmmlab)，加入 OpenMMLab 团队的 [官方交流 QQ 群](https://jq.qq.com/?_wv=1027&k=MSMAfWOe)，或添加微信小助手”OpenMMLabwx“加入官方交流微信群。
|
||||
|
||||
<div align="center">
|
||||
<img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/zhihu_qrcode.jpg" height="400" />
|
||||
<img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/qq_group_qrcode.jpg" height="400" />
|
||||
<img src="resources/qq_group_qrcode.jpg" height="400" />
|
||||
<img src="https://raw.githubusercontent.com/open-mmlab/mmcv/master/docs/en/_static/wechat_qrcode.jpg" height="400" />
|
||||
</div>
|
||||
|
||||
我们会在 OpenMMLab 社区为大家
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ Model converter is executed on linux platform, and SDK is executed on android pl
|
|||
|
||||
Here are two steps for android build.
|
||||
|
||||
1. Build model converter on linux, please refer to [How to build linux](./linux.md)
|
||||
1. Build model converter on linux, please refer to [How to build linux](./linux-x86_64.md)
|
||||
|
||||
2. Build SDK using android toolchain on linux.
|
||||
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
## Docker usage
|
||||
## Use Docker Image
|
||||
|
||||
We provide two dockerfiles for CPU and GPU respectively. For CPU users, we install MMDeploy with ONNXRuntime, ncnn and OpenVINO backends. For GPU users, we install MMDeploy with TensorRT backend. Besides, users can install mmdeploy with different versions when building the docker image.
|
||||
|
||||
|
|
@ -1,19 +1,19 @@
|
|||
# Build MMDeploy
|
||||
|
||||
## Download MMDeploy
|
||||
# Build From Source
|
||||
|
||||
|
||||
```bash
|
||||
git clone -b master git@github.com:open-mmlab/mmdeploy.git MMDeploy
|
||||
cd MMDeploy
|
||||
git submodule update --init --recursive
|
||||
## Download
|
||||
|
||||
|
||||
```shell
|
||||
git clone -b master git@github.com:open-mmlab/mmdeploy.git --recursive
|
||||
```
|
||||
|
||||
Note:
|
||||
|
||||
- If fetching submodule fails, you could get submodule manually by following instructions:
|
||||
|
||||
```bash
|
||||
```shell
|
||||
cd mmdeploy
|
||||
git clone git@github.com:NVIDIA/cub.git third_party/cub
|
||||
cd third_party/cub
|
||||
git checkout c3cceac115
|
||||
|
|
@ -26,15 +26,13 @@ Note:
|
|||
```
|
||||
- If it fails when `git clone` via `SSH`, you can try the `HTTPS` protocol like this:
|
||||
|
||||
```bash
|
||||
git clone -b master https://github.com/open-mmlab/mmdeploy.git MMDeploy
|
||||
cd MMDeploy
|
||||
git submodule update --init --recursive
|
||||
```shell
|
||||
git clone -b master https://github.com/open-mmlab/mmdeploy.git --recursive
|
||||
```
|
||||
|
||||
## Build MMDeploy
|
||||
## Build
|
||||
Please visit the following links to find out how to build MMDeploy according to the target platform.
|
||||
- [Linux-x86_64](build/linux.md)
|
||||
- [Windows](build/windows.md)
|
||||
- [Android-aarch64](build/android.md)
|
||||
- [NVIDIA Jetson](tutorials/how_to_install_mmdeploy_on_jetsons.md)
|
||||
- [Linux-x86_64](linux-x86_64.md)
|
||||
- [Windows](windows.md)
|
||||
- [Android-aarch64](android.md)
|
||||
- [NVIDIA Jetson](jetsons.md)
|
||||
|
|
@ -1,4 +1,4 @@
|
|||
## How to install mmdeploy on Jetsons
|
||||
## Build for Jetson
|
||||
|
||||
This tutorial introduces how to install mmdeploy on Nvidia Jetson systems. It mainly introduces the installation of mmdeploy on three Jetson series boards:
|
||||
- Jetson Nano
|
||||
|
|
@ -102,7 +102,7 @@ cmake --version
|
|||
```
|
||||
|
||||
### Install mmdeploy
|
||||
Just follow the instruction [here](../build.md). If it throws `failed building wheel for numpy...ERROR: Failed to build one or more wheels` when installing `h5py`, try install `h5py` manually.
|
||||
Just follow the instruction [here](../01-how-to-build/build_from_source.md). If it throws `failed building wheel for numpy...ERROR: Failed to build one or more wheels` when installing `h5py`, try install `h5py` manually.
|
||||
```
|
||||
sudo apt-get install pkg-config libhdf5-100 libhdf5-dev
|
||||
pip install versioned-hdf5 --no-cache-dir
|
||||
|
|
@ -1,30 +1,22 @@
|
|||
# Build for Linux-x86_64
|
||||
|
||||
- [Build for Linux-x86_64](#build-for-linux-x86_64)
|
||||
- [Dockerfile (RECOMMENDED)](#dockerfile-recommended)
|
||||
- [Build From Source](#build-from-source)
|
||||
- [Install Toolchains](#install-toolchains)
|
||||
- [Install Dependencies](#install-dependencies)
|
||||
- [Install Dependencies for Model Converter](#install-dependencies-for-model-converter)
|
||||
- [Install Dependencies for SDK](#install-dependencies-for-sdk)
|
||||
- [Install Inference Engines for MMDeploy](#install-inference-engines-for-mmdeploy)
|
||||
- [Build MMDeploy](#build-mmdeploy)
|
||||
- [Build Options Spec](#build-options-spec)
|
||||
- [Build Model Converter](#build-model-converter)
|
||||
- [Build Custom Ops](#build-custom-ops)
|
||||
- [Install Model Converter](#install-model-converter)
|
||||
- [Build SDK](#build-sdk)
|
||||
- [Build Demo](#build-demo)
|
||||
- [Install Toolchains](#install-toolchains)
|
||||
- [Install Dependencies](#install-dependencies)
|
||||
- [Install Dependencies for Model Converter](#install-dependencies-for-model-converter)
|
||||
- [Install Dependencies for SDK](#install-dependencies-for-sdk)
|
||||
- [Install Inference Engines for MMDeploy](#install-inference-engines-for-mmdeploy)
|
||||
- [Build MMDeploy](#build-mmdeploy)
|
||||
- [Build Options Spec](#build-options-spec)
|
||||
- [Build Model Converter](#build-model-converter)
|
||||
- [Build Custom Ops](#build-custom-ops)
|
||||
- [Install Model Converter](#install-model-converter)
|
||||
- [Build SDK](#build-sdk)
|
||||
- [Build Demo](#build-demo)
|
||||
|
||||
---
|
||||
MMDeploy provides two build ways for linux-x86_64 platform, including dockerfile and build from source.
|
||||
|
||||
## Dockerfile (RECOMMENDED)
|
||||
please refer to [how to use docker](../tutorials/how_to_use_docker.md).
|
||||
|
||||
## Build From Source
|
||||
|
||||
### Install Toolchains
|
||||
## Install Toolchains
|
||||
|
||||
- cmake
|
||||
|
||||
|
|
@ -47,8 +39,8 @@ please refer to [how to use docker](../tutorials/how_to_use_docker.md).
|
|||
sudo apt-get install g++-7
|
||||
```
|
||||
|
||||
### Install Dependencies
|
||||
#### Install Dependencies for Model Converter
|
||||
## Install Dependencies
|
||||
### Install Dependencies for Model Converter
|
||||
|
||||
<table class="docutils">
|
||||
<thead>
|
||||
|
|
@ -90,7 +82,7 @@ pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/${cu_ve
|
|||
</tbody>
|
||||
</table>
|
||||
|
||||
#### Install Dependencies for SDK
|
||||
### Install Dependencies for SDK
|
||||
|
||||
You can skip this chapter if you are only interested in the model converter.
|
||||
<table class="docutils">
|
||||
|
|
@ -129,7 +121,7 @@ git checkout tags/v0.6.2 -b v0.6.2
|
|||
</tbody>
|
||||
</table>
|
||||
|
||||
#### Install Inference Engines for MMDeploy
|
||||
### Install Inference Engines for MMDeploy
|
||||
|
||||
Both MMDeploy's model converter and SDK share the same inference engines.
|
||||
|
||||
|
|
@ -258,13 +250,13 @@ echo 'export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH' >> ~/.bashrc
|
|||
source ~/.bashrc
|
||||
```
|
||||
|
||||
### Build MMDeploy
|
||||
## Build MMDeploy
|
||||
```bash
|
||||
cd /the/root/path/of/MMDeploy
|
||||
export MMDEPLOY_DIR=$(pwd)
|
||||
```
|
||||
|
||||
#### Build Options Spec
|
||||
### Build Options Spec
|
||||
<table class="docutils">
|
||||
<thead>
|
||||
<tr>
|
||||
|
|
@ -339,9 +331,9 @@ Currently, <b>The Model Converter supports torchscript, but SDK doesn't</b>.
|
|||
</tbody>
|
||||
</table>
|
||||
|
||||
#### Build Model Converter
|
||||
### Build Model Converter
|
||||
|
||||
##### Build Custom Ops
|
||||
#### Build Custom Ops
|
||||
If one of inference engines among ONNXRuntime, TensorRT, ncnn and libtorch is selected, you have to build the corresponding custom ops.
|
||||
|
||||
- **ONNXRuntime** Custom Ops
|
||||
|
|
@ -380,7 +372,7 @@ If one of inference engines among ONNXRuntime, TensorRT, ncnn and libtorch is se
|
|||
make -j$(nproc)
|
||||
```
|
||||
|
||||
##### Install Model Converter
|
||||
#### Install Model Converter
|
||||
|
||||
```bash
|
||||
cd ${MMDEPLOY_DIR}
|
||||
|
|
@ -391,7 +383,7 @@ pip install -e .
|
|||
- Some dependencies are optional. Simply running `pip install -e .` will only install the minimum runtime requirements.
|
||||
To use optional dependencies, install them manually with `pip install -r requirements/optional.txt` or specify desired extras when calling `pip` (e.g. `pip install -e .[optional]`).
|
||||
Valid keys for the extras field are: `all`, `tests`, `build`, `optional`.
|
||||
#### Build SDK
|
||||
### Build SDK
|
||||
|
||||
MMDeploy provides two recipes as shown below for building SDK with ONNXRuntime and TensorRT as inference engines respectively.
|
||||
You can also activate other engines after the model.
|
||||
|
|
@ -430,7 +422,7 @@ You can also activate other engines after the model.
|
|||
make -j$(nproc) && make install
|
||||
```
|
||||
|
||||
#### Build Demo
|
||||
### Build Demo
|
||||
|
||||
```Bash
|
||||
cd ${MMDEPLOY_DIR}/build/install/example
|
||||
|
|
@ -1,35 +1,19 @@
|
|||
## How to convert model
|
||||
|
||||
<!-- TOC -->
|
||||
|
||||
- [How to convert model](#how-to-convert-model)
|
||||
- [How to convert models from Pytorch to other backends](#how-to-convert-models-from-pytorch-to-other-backends)
|
||||
- [Prerequisite](#prerequisite)
|
||||
- [Usage](#usage)
|
||||
- [Description of all arguments](#description-of-all-arguments)
|
||||
- [How to find the corresponding deployment config of a PyTorch model](#how-to-find-the-corresponding-deployment-config-of-a-pytorch-model)
|
||||
- [Example](#example)
|
||||
- [How to evaluate the exported models](#how-to-evaluate-the-exported-models)
|
||||
- [List of supported models exportable to other backends](#list-of-supported-models-exportable-to-other-backends)
|
||||
- [Reminders](#reminders)
|
||||
- [FAQs](#faqs)
|
||||
|
||||
<!-- TOC -->
|
||||
# How to convert model
|
||||
|
||||
This tutorial briefly introduces how to export an OpenMMlab model to a specific backend using MMDeploy tools.
|
||||
Notes:
|
||||
|
||||
- Supported backends are [ONNXRuntime](../backends/onnxruntime.md), [TensorRT](../backends/tensorrt.md), [ncnn](../backends/ncnn.md), [PPLNN](../backends/pplnn.md), [OpenVINO](../backends/openvino.md).
|
||||
- Supported codebases are [MMClassification](../codebases/mmcls.md), [MMDetection](../codebases/mmdet.md), [MMSegmentation](../codebases/mmseg.md), [MMOCR](../codebases/mmocr.md), [MMEditing](../codebases/mmedit.md).
|
||||
- Supported backends are [ONNXRuntime](../05-supported-backends/onnxruntime.md), [TensorRT](../05-supported-backends/tensorrt.md), [ncnn](../05-supported-backends/ncnn.md), [PPLNN](../05-supported-backends/pplnn.md), [OpenVINO](../05-supported-backends/openvino.md).
|
||||
- Supported codebases are [MMClassification](../04-supported-codebases/mmcls.md), [MMDetection](../04-supported-codebases/mmdet.md), [MMSegmentation](../04-supported-codebases/mmseg.md), [MMOCR](../04-supported-codebases/mmocr.md), [MMEditing](../04-supported-codebases/mmedit.md).
|
||||
|
||||
### How to convert models from Pytorch to other backends
|
||||
## How to convert models from Pytorch to other backends
|
||||
|
||||
#### Prerequisite
|
||||
### Prerequisite
|
||||
|
||||
1. Install and build your target backend. You could refer to [ONNXRuntime-install](../backends/onnxruntime.md), [TensorRT-install](../backends/tensorrt.md), [ncnn-install](../backends/ncnn.md), [PPLNN-install](../backends/pplnn.md), [OpenVINO-install](../backends/openvino.md) for more information.
|
||||
1. Install and build your target backend. You could refer to [ONNXRuntime-install](../05-supported-backends/onnxruntime.md), [TensorRT-install](../05-supported-backends/tensorrt.md), [ncnn-install](../05-supported-backends/ncnn.md), [PPLNN-install](../05-supported-backends/pplnn.md), [OpenVINO-install](../05-supported-backends/openvino.md) for more information.
|
||||
2. Install and build your target codebase. You could refer to [MMClassification-install](https://github.com/open-mmlab/mmclassification/blob/master/docs/en/install.md), [MMDetection-install](https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md), [MMSegmentation-install](https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/get_started.md#installation), [MMOCR-install](https://mmocr.readthedocs.io/en/latest/install.html), [MMEditing-install](https://github.com/open-mmlab/mmediting/blob/master/docs/en/install.md).
|
||||
|
||||
#### Usage
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
python ./tools/deploy.py \
|
||||
|
|
@ -46,7 +30,7 @@ python ./tools/deploy.py \
|
|||
--dump-info
|
||||
```
|
||||
|
||||
#### Description of all arguments
|
||||
### Description of all arguments
|
||||
|
||||
- `deploy_cfg` : The path of deploy config file in MMDeploy codebase.
|
||||
- `model_cfg` : The path of model config file in OpenMMLab codebase.
|
||||
|
|
@ -60,13 +44,13 @@ python ./tools/deploy.py \
|
|||
- `--show` : Whether to show detection outputs.
|
||||
- `--dump-info` : Whether to output information for SDK.
|
||||
|
||||
#### How to find the corresponding deployment config of a PyTorch model
|
||||
### How to find the corresponding deployment config of a PyTorch model
|
||||
|
||||
1. Find model's codebase folder in `configs/`. Example, convert a yolov3 model you need to find `configs/mmdet` folder.
|
||||
2. Find model's task folder in `configs/codebase_folder/`. Just like yolov3 model, you need to find `configs/mmdet/detection` folder.
|
||||
3. Find deployment config file in `configs/codebase_folder/task_folder/`. Just like deploy yolov3 model you can use `configs/mmdet/detection/detection_onnxruntime_dynamic.py`.
|
||||
|
||||
#### Example
|
||||
### Example
|
||||
|
||||
```bash
|
||||
python ./tools/deploy.py \
|
||||
|
|
@ -79,18 +63,10 @@ python ./tools/deploy.py \
|
|||
--device cuda:0
|
||||
```
|
||||
|
||||
### How to evaluate the exported models
|
||||
## How to evaluate the exported models
|
||||
|
||||
You can try to evaluate model, referring to [how_to_evaluate_a_model](./how_to_evaluate_a_model.md).
|
||||
|
||||
### List of supported models exportable to other backends
|
||||
## List of supported models exportable to other backends
|
||||
|
||||
Refer to [Support model list](../supported_models.md)
|
||||
|
||||
### Reminders
|
||||
|
||||
- None
|
||||
|
||||
### FAQs
|
||||
|
||||
- None
|
||||
Refer to [Support model list](../03-benchmark/supported_models.md)
|
||||
|
|
@ -1,10 +1,11 @@
|
|||
# How to evaluate a model
|
||||
# How to evaluate model
|
||||
|
||||
After we convert a PyTorch model to a backend model, we may need to evaluate the performance of the model before using it. In MMDeploy, we provide a tool to evaluate backend models in `tools/test.py`
|
||||
After converting a PyTorch model to a backend model, you may evaluate backend models with `tools/test.py`
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Before evaluating a model of a specific backend, you should [install the plugins](../build.md) of the backend and convert the model to the backend with our [deploy tools](how_to_convert_model.md).
|
||||
Install MMDeploy according to [get-started](../get_started.md) instructions.
|
||||
And convert the PyTorch model or ONNX model to the backend model by following the [guide](convert_model.md).
|
||||
|
||||
## Usage
|
||||
|
||||
|
|
@ -52,7 +53,7 @@ python tools/test.py \
|
|||
{MMCLS_DIR}/configs/resnet/resnet50_b32x8_imagenet.py \
|
||||
--model model.onnx \
|
||||
--out out.pkl \
|
||||
--device cuda:0 \
|
||||
--device cuda:0
|
||||
```
|
||||
|
||||
## Note
|
||||
|
|
@ -1,12 +1,13 @@
|
|||
# How to measure the performance of a model
|
||||
# How to profile model
|
||||
|
||||
After we convert a PyTorch model to a backend model, we may need to test the speed of the model before using it. In MMDeploy, we provide a tool to test the speed of backend models in `tools/test.py`
|
||||
After converting a PyTorch model to a backend model, you can profile inference speed using `tools/test.py`.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Before test the speed of a model of a specific backend, you should [install the plugins](../build.md) of the backend and convert the model to the backend with our [deploy tools](how_to_convert_model.md).
|
||||
Install MMDeploy according to [get-started](../get_started.md) instructions.
|
||||
And convert the PyTorch model or ONNX model to the backend model by following the [guide](convert_model.md).
|
||||
|
||||
## Usage
|
||||
## Profile
|
||||
|
||||
```shell
|
||||
python tools/test.py \
|
||||
|
|
@ -16,11 +17,13 @@ ${MODEL_CFG} \
|
|||
[--speed-test] \
|
||||
[--warmup ${WARM_UP}] \
|
||||
[--log-interval ${LOG_INTERVERL}] \
|
||||
[--log2file ${LOG_RESULT_TO_FILE}] \
|
||||
[--log2file ${LOG_RESULT_TO_FILE}]
|
||||
```
|
||||
|
||||
## Description of all arguments
|
||||
|
||||
|
||||
|
||||
* `deploy_cfg`: The config for deployment.
|
||||
* `model_cfg`: The config of the model in OpenMMLab codebases.
|
||||
* `--model`: The backend model files. For example, if we convert a model to ncnn, we need to pass a ".param" file and a ".bin" file. If we convert a model to TensorRT, we need to pass the model file with ".engine" suffix.
|
||||
|
|
@ -39,5 +42,5 @@ python tools/test.py \
|
|||
{MMCLS_DIR}/configs/resnet/resnet50_b32x8_imagenet.py \
|
||||
--model model.onnx \
|
||||
--speed-test \
|
||||
--device cpu \
|
||||
--device cpu
|
||||
```
|
||||
|
|
@ -96,7 +96,7 @@ codebase_config = dict(type='mmcls', task='Classification')
|
|||
|
||||
### 3. How to write backend config
|
||||
|
||||
The backend config is mainly used to specify the backend on which model runs and provide the information needed when the model runs on the backend , referring to [ONNX Runtime](../backends/onnxruntime.md), [TensorRT](../backends/tensorrt.md), [ncnn](../backends/ncnn.md), [PPLNN](../backends/pplnn.md).
|
||||
The backend config is mainly used to specify the backend on which model runs and provide the information needed when the model runs on the backend , referring to [ONNX Runtime](../05-supported-backends/onnxruntime.md), [TensorRT](../05-supported-backends/tensorrt.md), [ncnn](../05-supported-backends/ncnn.md), [PPLNN](../05-supported-backends/pplnn.md).
|
||||
|
||||
- `type`: Model's backend, including `onnxruntime`, `ncnn`, `pplnn`, `tensorrt`, `openvino`.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
|
|
@ -1,4 +1,4 @@
|
|||
## List of supported models exportable to other backends
|
||||
## Supported Models
|
||||
|
||||
The table below lists the models that are guaranteed to be exportable to other backends.
|
||||
|
||||
|
|
@ -1,12 +1,12 @@
|
|||
## MMClassification Support
|
||||
# MMClassification Support
|
||||
|
||||
[MMClassification](https://github.com/open-mmlab/mmclassification) is an open-source image classification toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com) project.
|
||||
|
||||
### MMClassification installation tutorial
|
||||
## MMClassification installation tutorial
|
||||
|
||||
Please refer to [install.md](https://github.com/open-mmlab/mmclassification/blob/master/docs/en/install.md) for installation.
|
||||
|
||||
### List of MMClassification models supported by MMDeploy
|
||||
## List of MMClassification models supported by MMDeploy
|
||||
|
||||
| Model | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----------- | :----------: | :------: | :---: | :---: | :------: | :----------------------------------------------------------------------------------------: |
|
||||
|
|
@ -16,11 +16,3 @@ Please refer to [install.md](https://github.com/open-mmlab/mmclassification/blob
|
|||
| MobileNetV2 | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/mobilenet_v2) |
|
||||
| ShuffleNetV1 | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v1) |
|
||||
| ShuffleNetV2 | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmclassification/tree/master/configs/shufflenet_v2) |
|
||||
|
||||
### Reminder
|
||||
|
||||
None
|
||||
|
||||
### FAQs
|
||||
|
||||
None
|
||||
|
|
@ -1,12 +1,12 @@
|
|||
## MMDetection Support
|
||||
# MMDetection Support
|
||||
|
||||
MMDetection is an open source object detection toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
### MMDetection installation tutorial
|
||||
## MMDetection installation tutorial
|
||||
|
||||
Please refer to [get_started.md](https://github.com/open-mmlab/mmdetection/blob/master/docs/en/get_started.md) for installation.
|
||||
|
||||
### List of MMDetection models supported by MMDeploy
|
||||
## List of MMDetection models supported by MMDeploy
|
||||
|
||||
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----------------: | :------------------: | :---------: | :------: | :---: | :---: | :------: | :----------------------------------------------------------------------------------: |
|
||||
|
|
@ -26,11 +26,3 @@ Please refer to [get_started.md](https://github.com/open-mmlab/mmdetection/blob/
|
|||
| RepPoints | ObjectDetection | N | Y | N | ? | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/reppoints) |
|
||||
| Cascade Mask R-CNN | InstanceSegmentation | Y | N | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) |
|
||||
| Mask R-CNN | InstanceSegmentation | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn) |
|
||||
|
||||
### Reminder
|
||||
|
||||
None
|
||||
|
||||
### FAQs
|
||||
|
||||
None
|
||||
|
|
@ -1,12 +1,12 @@
|
|||
## MMDetection3d Support
|
||||
# MMDetection3d Support
|
||||
|
||||
MMDetection3d is a next-generation platform for general 3D object detection. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
### MMDetection3d installation tutorial
|
||||
## MMDetection3d installation tutorial
|
||||
|
||||
Please refer to [getting_started.md](https://github.com/open-mmlab/mmdetection3d/blob/master/docs/en/getting_started.md) for installation.
|
||||
|
||||
### Example
|
||||
## Example
|
||||
|
||||
```bash
|
||||
python tools/deploy.py \
|
||||
|
|
@ -20,13 +20,13 @@ python tools/deploy.py \
|
|||
--device \
|
||||
cuda:0
|
||||
```
|
||||
### List of MMDetection3d models supported by MMDeploy
|
||||
## List of MMDetection3d models supported by MMDeploy
|
||||
|
||||
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :----------------: | :------------------: | :---------: | :------: | :---: | :---: | :------: | :------------------------------------------------------------------------------------------------------: |
|
||||
| PointPillars | VoxelDetection | Y | Y | N | N | Y | [config](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/pointpillars) |
|
||||
|
||||
### Reminder
|
||||
## Reminder
|
||||
|
||||
Voxel detection onnx model excludes model.voxelize layer and model post process, and you can use python api to call these func.
|
||||
|
||||
|
|
@ -37,7 +37,3 @@ from mmdeploy.codebase.mmdet3d.deploy import VoxelDetectionModel
|
|||
VoxelDetectionModel.voxelize(...)
|
||||
VoxelDetectionModel.post_process(...)
|
||||
```
|
||||
|
||||
### FAQs
|
||||
|
||||
None
|
||||
|
|
@ -18,11 +18,3 @@ Please refer to [official installation guide](https://mmediting.readthedocs.io/e
|
|||
| Real-ESRGAN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/real_esrgan) |
|
||||
| EDSR | super-resolution | Y | Y | Y | N | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/edsr) |
|
||||
| RDN | super-resolution | Y | Y | Y | Y | Y | [config](https://github.com/open-mmlab/mmediting/tree/master/configs/restorers/rdn) |
|
||||
|
||||
## Reminder
|
||||
|
||||
None
|
||||
|
||||
## FAQs
|
||||
|
||||
None
|
||||
|
|
@ -1,12 +1,12 @@
|
|||
## MMOCR Support
|
||||
# MMOCR Support
|
||||
|
||||
MMOCR is an open-source toolbox based on PyTorch and mmdetection for text detection, text recognition, and the corresponding downstream tasks including key information extraction. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
### MMOCR installation tutorial
|
||||
## MMOCR installation tutorial
|
||||
|
||||
Please refer to [install.md](https://mmocr.readthedocs.io/en/latest/install.html) for installation.
|
||||
|
||||
### List of MMOCR models supported by MMDeploy
|
||||
## List of MMOCR models supported by MMDeploy
|
||||
|
||||
| Model | Task | TorchScript | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO | Model config |
|
||||
| :---- | :--------------- | :---------: | :---------: | :------: | :---: | :---: | :------: | :----------------------------------------------------------------------------: |
|
||||
|
|
@ -18,7 +18,7 @@ Please refer to [install.md](https://mmocr.readthedocs.io/en/latest/install.html
|
|||
| SATRN | text-recognition | Y | Y | Y | N | N | N | [config](https://github.com/open-mmlab/mmocr/tree/main/configs/textrecog/satrn)|
|
||||
|
||||
|
||||
### Reminder
|
||||
## Reminder
|
||||
|
||||
Note that ncnn, pplnn, and OpenVINO only support the configs of DBNet18 for DBNet.
|
||||
|
||||
|
|
@ -162,7 +162,3 @@ def basic_block__forward__trt(ctx, self, x: torch.Tensor) -> torch.Tensor:
|
|||
return out
|
||||
|
||||
```
|
||||
|
||||
### FAQs
|
||||
|
||||
None
|
||||
|
|
@ -29,11 +29,3 @@ $MMDEPLOY_DIR/demo/resources/human-pose.jpg \
|
|||
Note
|
||||
|
||||
- Usually, mmpose models need some extra information for the input image, but we can't get it directly. So, when exporting the model, you can use `$MMDEPLOY_DIR/demo/resources/human-pose.jpg` as input.
|
||||
|
||||
## Reminder
|
||||
|
||||
None
|
||||
|
||||
## FAQs
|
||||
|
||||
None
|
||||
|
|
@ -44,11 +44,3 @@ python tools/test.py \
|
|||
Note
|
||||
|
||||
- Usually, mmrotate models need some extra information for the input image, but we can't get it directly. So, when exporting the model, you can use `$MMROTATE_DIR/demo/demo.jpg` as input.
|
||||
|
||||
## Reminder
|
||||
|
||||
None
|
||||
|
||||
## FAQs
|
||||
|
||||
None
|
||||
|
|
@ -1,12 +1,12 @@
|
|||
## MMSegmentation Support
|
||||
# MMSegmentation Support
|
||||
|
||||
MMSegmentation is an open source object segmentation toolbox based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com/) project.
|
||||
|
||||
### MMSegmentation installation tutorial
|
||||
## MMSegmentation installation tutorial
|
||||
|
||||
Please refer to [get_started.md](https://github.com/open-mmlab/mmsegmentation/blob/master/docs/en/get_started.md#installation) for installation.
|
||||
|
||||
### List of MMSegmentation models supported by MMDeploy
|
||||
## List of MMSegmentation models supported by MMDeploy
|
||||
|
||||
| Model | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVino | Model config |
|
||||
|:----------------------------|:-----------:|:--------:|:----:|:-----:|:--------:|:----------------------------------------------------------------------------------------:|
|
||||
|
|
@ -44,14 +44,10 @@ Please refer to [get_started.md](https://github.com/open-mmlab/mmsegmentation/bl
|
|||
| PSANet | N | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/psanet) |
|
||||
| DPT | N | N | N | N | N | [config](https://github.com/open-mmlab/mmsegmentation/tree/master/configs/dpt) |
|
||||
|
||||
### Reminder
|
||||
## Reminder
|
||||
|
||||
- Only `whole` inference mode is supported for all mmseg models.
|
||||
|
||||
- <i id="static_shape">PSPNet, Fast-SCNN</i> only support static shape, because [nn.AdaptiveAvgPool2d](https://github.com/open-mmlab/mmsegmentation/blob/97f9670c5a4a2a3b4cfb411bcc26db16b23745f7/mmseg/models/decode_heads/psp_head.py#L38) is not supported in most of backends dynamically.
|
||||
|
||||
- For models only supporting static shape, you should use the deployment config file of static shape such as `configs/mmseg/segmentation_tensorrt_static-1024x2048.py`.
|
||||
|
||||
### FAQs
|
||||
|
||||
None
|
||||
|
|
@ -1,10 +1,10 @@
|
|||
## ncnn Support
|
||||
# ncnn Support
|
||||
|
||||
MMDeploy now supports ncnn version == 1.0.20211208
|
||||
|
||||
### Installation
|
||||
## Installation
|
||||
|
||||
#### Install ncnn
|
||||
### Install ncnn
|
||||
|
||||
- Download VulkanTools for the compilation of ncnn.
|
||||
|
||||
|
|
@ -47,7 +47,7 @@ You should ensure your gcc satisfies `gcc >= 6`.
|
|||
pip install -e .
|
||||
```
|
||||
|
||||
#### Build custom ops
|
||||
### Build custom ops
|
||||
|
||||
Some custom ops are created to support models in OpenMMLab, the custom ops can be built as follows:
|
||||
|
||||
|
|
@ -65,26 +65,16 @@ If you haven't installed ncnn in the default path, please add `-Dncnn_DIR` flag
|
|||
make -j$(nproc)
|
||||
```
|
||||
|
||||
### Convert model
|
||||
## Convert model
|
||||
|
||||
- This follows the tutorial on [How to convert model](../tutorials/how_to_convert_model.md).
|
||||
- This follows the tutorial on [How to convert model](../02-how-to-run/convert_model.md).
|
||||
- The converted model has two files: `.param` and `.bin`, as model structure file and weight file respectively.
|
||||
|
||||
|
||||
### List of supported custom ops
|
||||
|
||||
| Operator | CPU | MMDeploy Releases |
|
||||
|:--------------------------------|:---:|:------------------|
|
||||
| [Expand](../ops/ncnn.md#expand) | Y | master |
|
||||
| [Gather](../ops/ncnn.md#gather) | Y | master |
|
||||
| [Shape](../ops/ncnn.md#shape) | Y | master |
|
||||
| [TopK](../ops/ncnn.md#topk) | Y | master |
|
||||
|
||||
#### Reminder
|
||||
## Reminder
|
||||
|
||||
- In ncnn version >= 1.0.20201208, the dimension of ncnn.Mat should be no more than 4.
|
||||
|
||||
### FAQs
|
||||
## FAQs
|
||||
|
||||
1. When running ncnn models for inference with custom ops, it fails and shows the error message like:
|
||||
|
||||
|
|
@ -1,10 +1,10 @@
|
|||
## ONNX Runtime Support
|
||||
# ONNX Runtime Support
|
||||
|
||||
### Introduction of ONNX Runtime
|
||||
## Introduction of ONNX Runtime
|
||||
|
||||
**ONNX Runtime** is a cross-platform inferencing and training accelerator compatible with many popular ML/DNN frameworks. Check its [github](https://github.com/microsoft/onnxruntime) for more information.
|
||||
**ONNX Runtime** is a cross-platform inference and training accelerator compatible with many popular ML/DNN frameworks. Check its [github](https://github.com/microsoft/onnxruntime) for more information.
|
||||
|
||||
### Installation
|
||||
## Installation
|
||||
|
||||
*Please note that only **onnxruntime>=1.8.1** of CPU version on Linux platform is supported by now.*
|
||||
|
||||
|
|
@ -14,9 +14,9 @@
|
|||
pip install onnxruntime==1.8.1
|
||||
```
|
||||
|
||||
### Build custom ops
|
||||
## Build custom ops
|
||||
|
||||
#### Prerequisite
|
||||
### Prerequisite
|
||||
|
||||
- Download `onnxruntime-linux` from ONNX Runtime [releases](https://github.com/microsoft/onnxruntime/releases/tag/v1.8.1), extract it, expose `ONNXRUNTIME_DIR` and finally add the lib path to `LD_LIBRARY_PATH` as below:
|
||||
|
||||
|
|
@ -40,7 +40,7 @@ Note:
|
|||
source ~/.bashrc
|
||||
```
|
||||
|
||||
#### Build on Linux
|
||||
### Build on Linux
|
||||
|
||||
```bash
|
||||
cd ${MMDEPLOY_DIR} # To MMDeploy root directory
|
||||
|
|
@ -49,22 +49,13 @@ cmake -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
|
|||
make -j$(nproc)
|
||||
```
|
||||
|
||||
### How to convert a model
|
||||
## How to convert a model
|
||||
|
||||
- You could follow the instructions of tutorial [How to convert model](../tutorials/how_to_convert_model.md)
|
||||
- You could follow the instructions of tutorial [How to convert model](../02-how-to-run/convert_model.md)
|
||||
|
||||
### List of supported custom ops
|
||||
## How to add a new custom op
|
||||
|
||||
| Operator | CPU | GPU | MMDeploy Releases |
|
||||
| :--------------------------------------------------------------------------- | :---: | :---: | :---------------- |
|
||||
| [grid_sampler](../ops/onnxruntime.md#grid_sampler) | Y | N | master |
|
||||
| [MMCVModulatedDeformConv2d](../ops/onnxruntime.md#mmcvmodulateddeformconv2d) | Y | N | master |
|
||||
| [NMSRotated](../ops/onnxruntime.md#nmsrotated) | Y | N | master |
|
||||
| [RoIAlignRotated](../ops/onnxruntime.md#roialignrotated) | Y | N | master |
|
||||
|
||||
### How to add a new custom op
|
||||
|
||||
#### Reminder
|
||||
## Reminder
|
||||
|
||||
- The custom operator is not included in [supported operator list](https://github.com/microsoft/onnxruntime/blob/master/docs/OperatorKernels.md) in ONNX Runtime.
|
||||
- The custom operator should be able to be exported to ONNX.
|
||||
|
|
@ -80,11 +71,7 @@ Take custom operator `roi_align` for example.
|
|||
|
||||
**Finally, welcome to send us PR of adding custom operators for ONNX Runtime in MMDeploy.** :nerd_face:
|
||||
|
||||
### FAQs
|
||||
|
||||
- None
|
||||
|
||||
### References
|
||||
## References
|
||||
|
||||
- [How to export Pytorch model with custom op to ONNX and run it in ONNX Runtime](https://github.com/onnx/tutorials/blob/master/PyTorchCustomOperator/README.md)
|
||||
- [How to add a custom operator/kernel in ONNX Runtime](https://github.com/microsoft/onnxruntime/blob/master/docs/AddingCustomOp.md)
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
## OpenVINO Support
|
||||
# OpenVINO Support
|
||||
|
||||
This tutorial is based on Linux systems like Ubuntu-18.04.
|
||||
### Installation
|
||||
## Installation
|
||||
It is recommended to create a virtual environment for the project.
|
||||
1. Install [OpenVINO](https://docs.openvino.ai/2021.4/get_started.html). It is recommended to use the installer or install using pip.
|
||||
Installation example using [pip](https://pypi.org/project/openvino-dev/):
|
||||
|
|
@ -10,19 +10,11 @@ pip install openvino-dev
|
|||
```
|
||||
2. *`Optional` If you want to use OpenVINO in SDK, you need install OpenVINO with [install_guides](https://docs.openvino.ai/2021.4/openvino_docs_install_guides_installing_openvino_linux.html#install-openvino).
|
||||
|
||||
3. Install MMDeploy following the [instructions](../build.md).
|
||||
3. Install MMDeploy following the [instructions](../01-how-to-build/build_from_source.md).
|
||||
|
||||
To work with models from [MMDetection](https://github.com/open-mmlab/mmdetection/blob/master/docs/get_started.md), you may need to install it additionally.
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
#### ImportError: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
|
||||
To resolve missing external dependency on Ubuntu*, execute the following command:
|
||||
```bash
|
||||
sudo apt-get install libpython3.7
|
||||
```
|
||||
|
||||
### Usage
|
||||
## Usage
|
||||
|
||||
Example:
|
||||
```bash
|
||||
|
|
@ -36,7 +28,7 @@ python tools/deploy.py \
|
|||
--log-level INFO
|
||||
```
|
||||
|
||||
### List of supported models exportable to OpenVINO from MMDetection
|
||||
## List of supported models exportable to OpenVINO from MMDetection
|
||||
|
||||
The table below lists the models that are guaranteed to be exportable to OpenVINO from MMDetection.
|
||||
| Model name | Config | Dynamic Shape |
|
||||
|
|
@ -63,7 +55,7 @@ Notes:
|
|||
the RoiAlign operation is replaced with the [ExperimentalDetectronROIFeatureExtractor](https://docs.openvinotoolkit.org/latest/openvino_docs_ops_detection_ExperimentalDetectronROIFeatureExtractor_6.html) operation in the ONNX graph.
|
||||
- Models "VFNet" and "Faster R-CNN + DCN" use the custom "DeformableConv2D" operation.
|
||||
|
||||
### Deployment config
|
||||
## Deployment config
|
||||
|
||||
With the deployment config, you can specify additional options for the Model Optimizer.
|
||||
To do this, add the necessary parameters to the `backend_config.mo_options` in the fields `args` (for parameters with values) and `flags` (for flags).
|
||||
|
|
@ -84,7 +76,12 @@ backend_config = dict(
|
|||
|
||||
Information about the possible parameters for the Model Optimizer can be found in the [documentation](https://docs.openvino.ai/latest/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model.html).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### FAQs
|
||||
- ImportError: libpython3.7m.so.1.0: cannot open shared object file: No such file or directory
|
||||
|
||||
- None
|
||||
To resolve missing external dependency on Ubuntu*, execute the following command:
|
||||
|
||||
```bash
|
||||
sudo apt-get install libpython3.7
|
||||
```
|
||||
|
|
@ -1,14 +1,14 @@
|
|||
## PPLNN Support
|
||||
# PPLNN Support
|
||||
|
||||
This tutorial is based on Linux systems like Ubuntu-18.04.
|
||||
|
||||
### Installation
|
||||
## Installation
|
||||
|
||||
1. Please install [pyppl](https://github.com/openppl-public/ppl.nn) following [install-guide](https://github.com/openppl-public/ppl.nn/blob/master/docs/en/building-from-source.md).
|
||||
|
||||
2. Install MMdeploy following the [instructions](../build.md).
|
||||
2. Install MMDeploy following the [instructions](../01-how-to-build/build_from_source.md).
|
||||
|
||||
### Usage
|
||||
## Usage
|
||||
|
||||
Example:
|
||||
```bash
|
||||
|
|
@ -19,5 +19,5 @@ python tools/deploy.py \
|
|||
tests/data/tiger.jpeg \
|
||||
--work-dir ../deploy_result \
|
||||
--device cuda \
|
||||
--log-level INFO \
|
||||
--log-level INFO
|
||||
```
|
||||
|
|
@ -1,8 +1,8 @@
|
|||
## TensorRT Support
|
||||
# TensorRT Support
|
||||
|
||||
### Installation
|
||||
## Installation
|
||||
|
||||
#### Install TensorRT
|
||||
### Install TensorRT
|
||||
|
||||
Please install TensorRT 8 follow [install-guide](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing).
|
||||
|
||||
|
|
@ -20,7 +20,7 @@ Please install TensorRT 8 follow [install-guide](https://docs.nvidia.com/deeplea
|
|||
source ~/.bashrc
|
||||
```
|
||||
|
||||
#### Build custom ops
|
||||
### Build custom ops
|
||||
|
||||
Some custom ops are created to support models in OpenMMLab, and the custom ops can be built as follow:
|
||||
|
||||
|
|
@ -38,11 +38,11 @@ If you haven't installed TensorRT in the default path, Please add `-DTENSORRT_DI
|
|||
make -j$(nproc)
|
||||
```
|
||||
|
||||
### Convert model
|
||||
## Convert model
|
||||
|
||||
Please follow the tutorial in [How to convert model](../tutorials/how_to_convert_model.md). **Note** that the device must be `cuda` device.
|
||||
Please follow the tutorial in [How to convert model](../02-how-to-run/convert_model.md). **Note** that the device must be `cuda` device.
|
||||
|
||||
#### Int8 Support
|
||||
### Int8 Support
|
||||
|
||||
Since TensorRT supports INT8 mode, a custom dataset config can be given to calibrate the model. Following is an example for MMDetection:
|
||||
|
||||
|
|
@ -93,7 +93,7 @@ python tools/deploy.py \
|
|||
|
||||
If the calibration dataset is not given, the data will be calibrated with the dataset in model config.
|
||||
|
||||
### FAQs
|
||||
## FAQs
|
||||
|
||||
- Error `Cannot found TensorRT headers` or `Cannot found TensorRT libs`
|
||||
|
||||
|
|
@ -132,6 +132,6 @@ If the calibration dataset is not given, the data will be calibrated with the da
|
|||
|
||||
Read [this](https://forums.developer.nvidia.com/t/matrixmultiply-failed-on-tensorrt-7-2-1/158187/4) for detail.
|
||||
|
||||
- Install mmdeploy on Jetsons
|
||||
- Install mmdeploy on Jetson
|
||||
|
||||
We provide a tutorial to get start on Jetsons [here](../tutorials/how_to_install_mmdeploy_on_jetsons.md).
|
||||
We provide a tutorial to get start on Jetsons [here](../01-how-to-build/jetsons.md).
|
||||
|
|
@ -1,12 +1,12 @@
|
|||
## TorchScript support
|
||||
# TorchScript support
|
||||
|
||||
### Introduction of TorchScript
|
||||
## Introduction of TorchScript
|
||||
|
||||
**TorchScript** a way to create serializable and optimizable models from PyTorch code. Any TorchScript program can be saved from a Python process and loaded in a process where there is no Python dependency. Check the [Introduction to TorchScript](https://pytorch.org/tutorials/beginner/Intro_to_TorchScript_tutorial.html) for more details.
|
||||
|
||||
### Build custom ops
|
||||
## Build custom ops
|
||||
|
||||
#### Prerequisite
|
||||
### Prerequisite
|
||||
|
||||
- Download libtorch from the official website [here](https://pytorch.org/get-started/locally/).
|
||||
|
||||
|
|
@ -34,7 +34,7 @@ Note:
|
|||
source ~/.bashrc
|
||||
```
|
||||
|
||||
#### Build on Linux
|
||||
### Build on Linux
|
||||
|
||||
```bash
|
||||
cd ${MMDEPLOY_DIR} # To MMDeploy root directory
|
||||
|
|
@ -43,11 +43,11 @@ cmake -DMMDEPLOY_TARGET_BACKENDS=torchscript -DTorch_DIR=${Torch_DIR} ..
|
|||
make -j$(nproc)
|
||||
```
|
||||
|
||||
### How to convert a model
|
||||
## How to convert a model
|
||||
|
||||
- You could follow the instructions of tutorial [How to convert model](../tutorials/how_to_convert_model.md)
|
||||
- You could follow the instructions of tutorial [How to convert model](../02-how-to-run/convert_model.md)
|
||||
|
||||
### FAQs
|
||||
## FAQs
|
||||
|
||||
- Error: `projects/thirdparty/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN libraries. Please set the proper cuDNN prefixes and / or install cuDNN.`
|
||||
|
||||
|
|
@ -4,7 +4,7 @@ This tutorial introduces how to add unit test for backend ops. When you add a cu
|
|||
|
||||
### Prerequisite
|
||||
|
||||
- `Compile new ops`: After adding a new custom op, needs to recompile the relevant backend, referring to [build.md](../build.md).
|
||||
- `Compile new ops`: After adding a new custom op, needs to recompile the relevant backend, referring to [build.md](../01-how-to-build/build_from_source.md).
|
||||
|
||||
### 1. Add the test program test_XXXX()
|
||||
|
||||
|
|
@ -106,11 +106,3 @@ Use pytest to call the test function to test ops.
|
|||
```bash
|
||||
pytest tests/test_ops/test_ops.py::test_XXXX
|
||||
```
|
||||
|
||||
### 3. Reminder
|
||||
|
||||
None
|
||||
|
||||
### 4. FAQs
|
||||
|
||||
None
|
||||
|
|
@ -66,7 +66,7 @@ The backends in MMDeploy must support the ONNX. The backend loads the ".onnx" fi
|
|||
fp16_mode=False, max_workspace_size=0))
|
||||
```
|
||||
|
||||
After possessing a base backend config file, you can easily construct a complete deploy config through inheritance. Please refer to our [config tutorial](how_to_write_config.md) for more details. Here is an example:
|
||||
After possessing a base backend config file, you can easily construct a complete deploy config through inheritance. Please refer to our [config tutorial](../02-how-to-run/write_config.md) for more details. Here is an example:
|
||||
|
||||
```Python
|
||||
_base_ = ['../_base_/backends/onnxruntime.py']
|
||||
|
|
@ -151,7 +151,7 @@ The backends in MMDeploy must support the ONNX. The backend loads the ".onnx" fi
|
|||
# ...
|
||||
```
|
||||
|
||||
6. Convert the models of OpenMMLab to backends (if necessary) and inference on backend engine. If you find some incompatible operators when testing, you can try to rewrite the original model for the backend following the [rewriter tutorial](how_to_support_new_models.md) or add custom operators.
|
||||
6. Convert the models of OpenMMLab to backends (if necessary) and inference on backend engine. If you find some incompatible operators when testing, you can try to rewrite the original model for the backend following the [rewriter tutorial](support_new_model.md) or add custom operators.
|
||||
|
||||
7. Add docstring and unit tests for new code :).
|
||||
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
# How to test rewritten models
|
||||
|
||||
After you create a rewritten model using our [rewriter](how_to_support_new_models.md), it's better to write a unit test for the model to validate if the model rewrite would come into effect. Generally, we need to get outputs of the original model and rewritten model, then compare them. The outputs of the original model can be acquired directly by calling the forward function of the model, whereas the way to generate the outputs of the rewritten model depends on the complexity of the rewritten model.
|
||||
After you create a rewritten model using our [rewriter](support_new_model.md), it's better to write a unit test for the model to validate if the model rewrite would come into effect. Generally, we need to get outputs of the original model and rewritten model, then compare them. The outputs of the original model can be acquired directly by calling the forward function of the model, whereas the way to generate the outputs of the rewritten model depends on the complexity of the rewritten model.
|
||||
|
||||
## Test rewritten model with small changes
|
||||
|
||||
|
|
@ -1,64 +1,338 @@
|
|||
## Get Started
|
||||
# Get Started
|
||||
|
||||
MMDeploy provides some useful tools. It is easy to deploy models in OpenMMLab to various platforms. You can convert models in our pre-defined pipeline or build a custom conversion pipeline by yourself. This guide will show you how to convert a model with MMDeploy and integrate MMDeploy's SDK to your application!
|
||||
MMDeploy provides useful tools for deploying OpenMMLab models to various platforms and devices.
|
||||
|
||||
### Prerequisites
|
||||
With the help of them, you can not only do model deployment using our pre-defined pipelines but also customize your own deployment pipeline.
|
||||
|
||||
First we should install MMDeploy following [build.md](./build.md). Note that the build steps are slightly different among the supported backends. Here are some brief introductions to these backends:
|
||||
In the following chapters, we will describe the general routine and demonstrate a "hello-world" example - deploying Faster R-CNN model from [MMDetection](https://github.com/open-mmlab/mmdetection) to NVIDIA TensorRT.
|
||||
|
||||
- [ONNXRuntime](./backends/onnxruntime.md): ONNX Runtime is a cross-platform inference and training machine-learning accelerator. It has best support for <span style="color:red">ONNX IR</span>.
|
||||
- [TensorRT](./backends/tensorrt.md): NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for deep learning inference applications. It is a good choice if you want to deploy your model on <span style="color:red">NVIDIA devices</span>.
|
||||
- [ncnn](./backends/ncnn.md): ncnn is a high-performance neural network inference computing framework optimized for <span style="color:red">mobile platforms</span>. ncnn is deeply considerate about deployment and uses on <span style="color:red">mobile phones</span> from the beginning of design.
|
||||
- [PPLNN](./backends/pplnn.md): PPLNN, which is short for "PPLNN is a Primitive Library for Neural Network", is a high-performance deep-learning inference engine for efficient AI inferencing. It can run various ONNX models and has <span style="color:red">better support for OpenMMLab</span>.
|
||||
- [OpenVINO](./backends/openvino.md): OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference. The open-source toolkit allows to seamlessly integrate with <span style="color:red">Intel AI hardware</span>, the latest neural network accelerator chips, the Intel AI stick, and embedded computers or edge devices.
|
||||
## Introduction
|
||||
|
||||
Choose the backend which can meet your demand and install it following the link provided above.
|
||||
In MMDeploy, the deployment pipeline can be illustrated by a sequential modules, i.e., Model Converter, MMDeploy Model and Inference SDK.
|
||||
|
||||
### Convert Model
|
||||
|
||||
|
||||
Once you have installed MMDeploy, you can convert the PyTorch model in the OpenMMLab model zoo to the backend model with one magic spell! For example, if you want to convert the Faster-RCNN in [MMDetection](https://github.com/open-mmlab/mmdetection) to TensorRT:
|
||||
### Model Converter
|
||||
|
||||
```bash
|
||||
# Assume you have installed MMDeploy in ${MMDEPLOY_DIR} and MMDetection in ${MMDET_DIR}
|
||||
# If you do not know where to find the path. Just type `pip show mmdeploy` and `pip show mmdet` in your console.
|
||||
Model Converter aims at converting training models from OpenMMLab into backend models that can be run on target devices.
|
||||
It is able to transform PyTorch model into IR model, i.e., ONNX, TorchScript, as well as convert IR model to backend model. By combining them together, we can achieve one-click **end-to-end** model deployment.
|
||||
|
||||
### MMDeploy Model
|
||||
|
||||
MMDeploy Model is the result package exported by Model Converter.
|
||||
Beside the backend models, it also includes the model meta info, which will be used by Inference SDK.
|
||||
|
||||
### Inference SDK
|
||||
|
||||
Inference SDK is developed by C/C++, wrapping the preprocessing, model forward and postprocessing modules in model inference.
|
||||
It supports FFI such as C, C++, Python, C#, Java and so on.
|
||||
|
||||
## prerequisites
|
||||
|
||||
In order to do an end-to-end model deployment, MMDeploy requires Python 3.6+ and PyTorch 1.5+.
|
||||
|
||||
**Step 0.** Download and install Miniconda from the [official website](https://docs.conda.io/en/latest/miniconda.html).
|
||||
|
||||
**Step 1.** Create a conda environment and activate it.
|
||||
|
||||
```shell
|
||||
export PYTHON_VERSION=3.7
|
||||
conda create --name mmdeploy python=${PYTHON_VERSION} -y
|
||||
conda activate mmdeploy
|
||||
```
|
||||
|
||||
**Step 2.** Install PyTorch following [official instructions](https://pytorch.org/get-started/locally/), e.g.
|
||||
|
||||
On GPU platforms:
|
||||
|
||||
```shell
|
||||
export PYTORCH_VERSION=1.8.0
|
||||
export TORCHVISION_VERSION=0.9.0
|
||||
export CUDA_VERSION=11.1
|
||||
conda install pytorch==${PYTORCH_VERSION} torchvision==${TORCHVISION_VERSION} cudatoolkit=${CUDA_VERSION} -c pytorch -c conda-forge
|
||||
```
|
||||
|
||||
On CPU platforms:
|
||||
|
||||
```shell
|
||||
export PYTORCH_VERSION=1.8.0
|
||||
export TORCHVISION_VERSION=0.9.0
|
||||
conda install pytorch==${PYTORCH_VERSION} torchvision==${TORCHVISION_VERSION} cpuonly -c pytorch
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
We recommend that users follow our best practices installing MMDeploy.
|
||||
|
||||
**Step 0.** Install [MMCV](https://github.com/open-mmlab/mmcv).
|
||||
```shell
|
||||
export MMCV_VERSION=1.5.0
|
||||
export CUDA_STRING="${CUDA_VERSION/./""}"
|
||||
python -m pip install mmcv-full==${MMCV_VERSION} -f https://download.openmmlab.com/mmcv/dist/cu${CUDA_STRING}/torch${PYTORCH_VERSION}/index.html
|
||||
```
|
||||
|
||||
**Step 1.** Install MMDeploy.
|
||||
|
||||
Since v0.5.0, MMDeploy provides prebuilt packages, which can be found from [here](https://github.com/open-mmlab/mmdeploy/releases).
|
||||
You can download them according to your target platform and device.
|
||||
|
||||
Take the MMDeploy-TensorRT package on NVIDIA for example:
|
||||
|
||||
```shell
|
||||
export MMDEPLOY_VERSION=0.5.0
|
||||
export TENSORRT_VERSION=8.2.3.0
|
||||
export PYTHON_VERSION=3.7
|
||||
export PYTHON_STRING="${PYTHON_VERSION/./""}"
|
||||
|
||||
wget https://github.com/open-mmlab/mmdeploy/releases/download/v${MMDEPLOY_VERSION}/mmdeploy-${MMDEPLOY_VERSION}-linux-x86_64-cuda${CUDA_VERSION}-tensorrt${TENSORRT_VERSION}.tar.gz
|
||||
tar -zxvf mmdeploy-${MMDEPLOY_VERSION}-linux-x86_64-cuda${CUDA_VERSION}-tensorrt${TENSORRT_VERSION}.tar.gz
|
||||
cd mmdeploy-${MMDEPLOY_VERSION}-linux-x86_64-cuda${CUDA_VERSION}-tensorrt${TENSORRT_VERSION}
|
||||
python -m pip install dist/mmdeploy-*-py${PYTHON_STRING}*.whl
|
||||
python -m pip install sdk/python/mmdeploy_python-*-cp${PYTHON_STRING}*.whl
|
||||
export LD_LIBRARY_PATH=$(pwd)/sdk/lib:$LD_LIBRARY_PATH
|
||||
cd ..
|
||||
```
|
||||
|
||||
```{note}
|
||||
If MMDeploy prebuilt package doesn meet your target platforms or devices, please build MMDeploy from its source by following the build documents
|
||||
```
|
||||
|
||||
**step 2.** Install the inference backend
|
||||
|
||||
Based on the above MMDeploy-TensorRT package, we need to download and install [TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar), including [cuDNN](https://developer.nvidia.com/cudnn).
|
||||
|
||||
**Be aware that TensorRT version and cuDNN version must matches your CUDA Toolkit version**
|
||||
|
||||
The following shows an example of installing TensorRT 8.2.3.0 and cuDNN 8.2:
|
||||
|
||||
```shell
|
||||
export TENSORRT_VERSION=8.2.3.0
|
||||
CUDA_MAJOR="${CUDA_VERSION/\.*/""}"
|
||||
|
||||
# !!! Download tensorrt package from NVIDIA that matches your CUDA Toolkit version to the current working directory
|
||||
tar -zxvf TensorRT-${TENSORRT_VERSION}*cuda-${CUDA_MAJOR}*.tar.gz
|
||||
python -m pip install TensorRT-${TENSORRT_VERSION}/python/tensorrt-*-cp${PYTHON_STRING}*.whl
|
||||
python -m pip install pycuda
|
||||
export TENSORRT_DIR=$(pwd)/TensorRT-${TENSORRT_VERSION}
|
||||
export LD_LIBRARY_PATH=${TENSORRT_DIR}/lib:$LD_LIBRARY_PATH
|
||||
|
||||
# !!! Download cuDNN package from NVIDIA that matches your CUDA Toolkit and TensorRT version to the current working directory
|
||||
tar -zxvf cudnn-${CUDA_MAJOR}.*-linux-x64*.tgz
|
||||
export CUDNN_DIR=$(pwd)/cuda
|
||||
export LD_LIBRARY_PATH=$CUDNN_DIR/lib64:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
In the next chapters, we are going to present our 'Hello, world' example based on the above settings.
|
||||
|
||||
For the installation of all inference backends supported by MMDeploy right now, please refer to:
|
||||
|
||||
- [ONNX Runtime](05-supported-backends/onnxruntime.md)
|
||||
- [TensorRT](05-supported-backends/tensorrt.md)
|
||||
- [PPL.NN](05-supported-backends/pplnn.md)
|
||||
- [ncnn](05-supported-backends/ncnn.md)
|
||||
- [OpenVINO](05-supported-backends/openvino.md)
|
||||
- [LibTorch](05-supported-backends/torchscript.md)
|
||||
|
||||
## Convert Model
|
||||
|
||||
After the installation, you can enjoy the model deployment journey starting from converting PyTorch model to backend model.
|
||||
|
||||
Based on the above settings, we provide an example to convert the Faster R-CNN in [MMDetection](https://github.com/open-mmlab/mmdetection) to TensorRT as below:
|
||||
|
||||
```shell
|
||||
# clone mmdeploy repo. We are going to use the pre-defined pipeline config from the source code
|
||||
git clone --recursive https://github.com/open-mmlab/mmdeploy.git
|
||||
python -m pip install -r mmdeploy/requirements/runtime.txt
|
||||
export MMDEPLOY_DIR=$(pwd)/mmdeploy
|
||||
|
||||
# clone mmdetection repo. We have to use the config file to build PyTorch nn module
|
||||
python -m pip install mmdet==2.24.0
|
||||
git clone https://github.com/open-mmlab/mmdetection.git
|
||||
export MMDET_DIR=$(pwd)/mmdetection
|
||||
|
||||
# download Faster R-CNN checkpoint
|
||||
export CHECKPOINT_DIR=$(pwd)/checkpoints
|
||||
wget -P ${CHECKPOINT_DIR} https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
|
||||
|
||||
# set working directory, where the mmdeploy model is saved
|
||||
export WORK_DIR=$(pwd)/mmdeploy_models
|
||||
|
||||
# run the command to start model conversion
|
||||
python ${MMDEPLOY_DIR}/tools/deploy.py \
|
||||
${MMDEPLOY_DIR}/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
|
||||
${MMDET_DIR}/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \
|
||||
${CHECKPOINT_DIR}/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
|
||||
${INPUT_IMG} \
|
||||
${MMDET_DIR}/demo/demo.jpg \
|
||||
--work-dir ${WORK_DIR} \
|
||||
--device cuda:0 \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
`${MMDEPLOY_DIR}/tools/deploy.py` is a tool that does everything you need to convert a model. Read [how_to_convert_model](./tutorials/how_to_convert_model.md) for more details. The converted model and other meta-info will be found in `${WORK_DIR}`. And they make up of MMDeploy SDK Model that can be fed to MMDeploy SDK to do model inference.
|
||||
`${MMDEPLOY_DIR}/tools/deploy.py` does everything you need to convert a model. Read [how_to_convert_model](./02-how-to-run/convert_model.md) for more details.
|
||||
The converted model and its meta info will be found in the path specified by `--work-dir`.
|
||||
And they make up of MMDeploy Model that can be fed to MMDeploy SDK to do model inference.
|
||||
|
||||
`detection_tensorrt_dynamic-320x320-1344x1344.py` is a config file that contains all arguments you need to customize the conversion pipeline. The name is formed as
|
||||
`detection_tensorrt_dynamic-320x320-1344x1344.py` is a config file that contains all arguments you need to customize the conversion pipeline. The name is formed as:
|
||||
|
||||
```bash
|
||||
<task name>_<backend>-[backend options]_<dynamic support>.py
|
||||
```
|
||||
|
||||
It is easy to find the deployment config you need by name. If you want to customize the conversion, you can edit the config file by yourself. Here is a tutorial about [how to write config](./tutorials/how_to_write_config.md).
|
||||
If you want to customize the conversion pipeline, you can edit the config file by following [this](./02-how-to-run/write_config.md) tutorial.
|
||||
|
||||
### Inference Model
|
||||
## Inference Model
|
||||
|
||||
Now you can do model inference with the APIs provided by the backend. But what if you want to test the model instantly? We have some backend wrappers for you.
|
||||
After model conversion, we can perform inference both by Model Converter and Inference SDK.
|
||||
|
||||
The former is developed by Python, while the latter is mainly written by C/C++.
|
||||
|
||||
### Inference by Model Converter
|
||||
|
||||
Model Converter provides a unified API named as `inference_model` to do the job, making all inference backends API transparent to users.
|
||||
Take the previous converted Faster R-CNN tensorrt model for example,
|
||||
|
||||
```python
|
||||
from mmdeploy.apis import inference_model
|
||||
import os
|
||||
|
||||
model_cfg = os.getenv('MMDET_DIR') + '/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py'
|
||||
deploy_cfg = os.getenv('MMDEPLOY_DIR') + '/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py'
|
||||
backend_files = os.getenv('WORK_DIR') + '/end2end.engine'
|
||||
|
||||
result = inference_model(model_cfg, deploy_cfg, backend_files, img=img, device=device)
|
||||
```
|
||||
|
||||
The `inference_model` will create a wrapper module and do the inference for you. The result has the same format as the original OpenMMLab repo.
|
||||
The data type and data layout is exactly the same with the OpenMMLab PyTorch model inference results.
|
||||
|
||||
### Evaluate Model
|
||||
```{note}
|
||||
You can certainly use the infernce backend API directly to do inference. But since MMDeploy has being developed several custom operators, it's necessary to load them first before calling the infernce backend API.
|
||||
```
|
||||
|
||||
You might wonder that does the backend model have the same precision as the original one? How fast can the model run? MMDeploy provides tools to test the model. Take the converted TensorRT Faster-RCNN as an example:
|
||||
### Inference by SDK
|
||||
|
||||
```bash
|
||||
You can use SDK API to do model inference with the mmdeploy model generated by Model Converter.
|
||||
|
||||
In the following section, we will provide examples of deploying the converted Faster R-CNN model talked above with different FFI.
|
||||
|
||||
#### Python API
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Detector
|
||||
import os
|
||||
import cv2
|
||||
|
||||
# get mmdeploy model path of faster r-cnn
|
||||
model_path = os.getenv('WORK_DIR')
|
||||
# use mmdetection demo image as an input image
|
||||
image_path = '/'.join((os.getenv('MMDET_DIR'), 'demo/demo.jpg'))
|
||||
|
||||
img = cv2.imread(image_path)
|
||||
detector = Detector(model_path, 'cuda', 0)
|
||||
bboxes, labels, _ = detector([img])[0]
|
||||
|
||||
indices = [i for i in range(len(bboxes))]
|
||||
for index, bbox, label_id in zip(indices, bboxes, labels):
|
||||
[left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
|
||||
if score < 0.3:
|
||||
continue
|
||||
cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))
|
||||
|
||||
cv2.imwrite('output_detection.png', img)
|
||||
```
|
||||
|
||||
You can find more examples from [here](https://github.com/open-mmlab/mmdeploy/demo/python).
|
||||
|
||||
```{note}
|
||||
If you build MMDeploy from the source, please add ${MMDEPLOY_DIR}/build/lib to the environment variable PYTHONPATH.
|
||||
Otherwise, you will run into an error like ’ModuleNotFoundError: No module named 'mmdeploy_python'
|
||||
```
|
||||
|
||||
#### C API
|
||||
|
||||
Using SDK C API should follow next pattern,
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[create inference handle] --> B(read image)
|
||||
B --> C(apply handle)
|
||||
C --> D[deal with inference result]
|
||||
D -->E[destroy result buffer]
|
||||
E -->F[destroy handle]
|
||||
```
|
||||
|
||||
Now let's apply this procedure on the above Faster R-CNN model.
|
||||
|
||||
```C++
|
||||
#include <cstdlib>
|
||||
#include <opencv2/opencv.hpp>
|
||||
#include "detector.h"
|
||||
|
||||
int main() {
|
||||
const char* device_name = "cuda";
|
||||
int device_id = 0;
|
||||
|
||||
// get mmdeploy model path of faster r-cnn
|
||||
std::string model_path = std::getenv("WORK_DIR");
|
||||
// use mmdetection demo image as an input image
|
||||
std::string image_path = std::getenv("MMDET_DIR") + "/demo/demo.jpg";
|
||||
|
||||
// create inference handle
|
||||
mm_handle_t detector{};
|
||||
int status{};
|
||||
status = mmdeploy_detector_create_by_path(model_path, device_name, device_id, &detector);
|
||||
assert(status == MM_SUCCESS);
|
||||
|
||||
// read image
|
||||
cv::Mat img = cv::imread(image_path);
|
||||
assert(img.data);
|
||||
|
||||
// apply handle and get the inference result
|
||||
mm_mat_t mat{img.data, img.rows, img.cols, 3, MM_BGR, MM_INT8};
|
||||
mm_detect_t *bboxes{};
|
||||
int *res_count{};
|
||||
status = mmdeploy_detector_apply(detector, &mat, 1, &bboxes, &res_count);
|
||||
assert (status == MM_SUCCESS);
|
||||
|
||||
// deal with the result. Here we choose to visualize it
|
||||
for (int i = 0; i < *res_count; ++i) {
|
||||
const auto &box = bboxes[i].bbox;
|
||||
if (bboxes[i].score < 0.3) {
|
||||
continue;
|
||||
}
|
||||
cv::rectangle(img, cv::Point{(int)box.left, (int)box.top},
|
||||
cv::Point{(int)box.right, (int)box.bottom}, cv::Scalar{0, 255, 0});
|
||||
}
|
||||
|
||||
cv::imwrite('output_detection.png', img);
|
||||
|
||||
// destroy result buffer
|
||||
mmdeploy_detector_release_result(bboxes, res_count, 1);
|
||||
// destroy inference handle
|
||||
mmdeploy_detector_destroy(detector);
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
When you build this example, try to add MMDeploy package in your CMake project as following. Then pass `-DMMDeploy_DIR` to cmake, which indicates the path where `MMDeployConfig.cmake` locates. You can find it in the prebuilt package.
|
||||
|
||||
```Makefile
|
||||
find_package(MMDeploy REQUIRED)
|
||||
mmdeploy_load_static(${YOUR_AWESOME_TARGET} MMDeployStaticModules)
|
||||
mmdeploy_load_dynamic(${YOUR_AWESOME_TARGET} MMDeployDynamicModules)
|
||||
target_link_libraries(${YOUR_AWESOME_TARGET} PRIVATE MMDeployLibs)
|
||||
```
|
||||
|
||||
For more SDK C API usages, please read these [samples](https://github.com/open-mmlab/mmdeploy/demo/csrc).
|
||||
|
||||
#### C# API
|
||||
|
||||
Due to limitations on space, we will not present a specific example. But you can find all of them [here](https://github.com/open-mmlab/mmdeploy/demo/csharp).
|
||||
|
||||
## Evaluate Model
|
||||
|
||||
You can test the performance of deployed model using `tool/test.py`. For example,
|
||||
|
||||
```shell
|
||||
python ${MMDEPLOY_DIR}/tools/test.py \
|
||||
${MMDEPLOY_DIR}/configs/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
|
||||
${MMDET_DIR}/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \
|
||||
|
|
@ -67,150 +341,8 @@ python ${MMDEPLOY_DIR}/tools/test.py \
|
|||
--device cuda:0
|
||||
```
|
||||
|
||||
Read [how to evaluate a model](./tutorials/how_to_evaluate_a_model.md) for more details about how to use `tools/test.py`
|
||||
|
||||
### Integrate MMDeploy SDK
|
||||
|
||||
Make sure to turn on `MMDEPLOY_BUILD_SDK` to build and install SDK by following [build.md](./build.md).
|
||||
After that, the structure in the installation folder will show as follows,
|
||||
|
||||
```
|
||||
install
|
||||
├── example
|
||||
├── include
|
||||
│ ├── c
|
||||
│ └── cpp
|
||||
└── lib
|
||||
```
|
||||
where `include/c` and `include/cpp` correspond to C and C++ API respectively.
|
||||
|
||||
**Caution: The C++ API is highly volatile and not recommended at the moment.**
|
||||
|
||||
In the example directory, there are several examples involving classification, object detection, image segmentation and so on.
|
||||
You can refer to these examples to learn how to use MMDeploy SDK's C API and how to link ${MMDeploy_LIBS} to your application.
|
||||
|
||||
### A From-scratch Example
|
||||
|
||||
Here is an example of how to deploy and inference Faster R-CNN model of MMDetection from scratch.
|
||||
|
||||
#### Create Virtual Environment and Install MMDetection.
|
||||
|
||||
Please run the following command in Anaconda environment to [install MMDetection](https://mmdetection.readthedocs.io/en/latest/get_started.html#a-from-scratch-setup-script).
|
||||
|
||||
```bash
|
||||
conda create -n openmmlab python=3.7 -y
|
||||
conda activate openmmlab
|
||||
|
||||
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=10.2 -c pytorch -y
|
||||
|
||||
# install mmcv
|
||||
pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.8/index.html
|
||||
|
||||
# install mmdetection
|
||||
git clone https://github.com/open-mmlab/mmdetection.git
|
||||
cd mmdetection
|
||||
pip install -r requirements/build.txt
|
||||
pip install -v -e .
|
||||
```{note}
|
||||
Regarding the --model option, it represents the converted engine files path when using Model Converter to do performance test. But when you try to test the metrics by Inference SDK, this option refers to the directory path of MMDeploy Model.
|
||||
```
|
||||
|
||||
#### Download the Checkpoint of Faster R-CNN
|
||||
|
||||
Download the checkpoint from this [link](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth) and put it in the `{MMDET_ROOT}/checkpoints` where `{MMDET_ROOT}` is the root directory of your MMDetection codebase.
|
||||
|
||||
#### Install MMDeploy and ONNX Runtime
|
||||
|
||||
Please run the following command in Anaconda environment to [install MMDeploy](./build.md).
|
||||
```bash
|
||||
conda activate openmmlab
|
||||
|
||||
git clone https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
git submodule update --init --recursive
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
Once we have installed the MMDeploy, we should select an inference engine for model inference. Here we take ONNX Runtime as an example. Run the following command to [install ONNX Runtime](./backends/onnxruntime.md):
|
||||
```bash
|
||||
pip install onnxruntime==1.8.1
|
||||
```
|
||||
|
||||
Then download the ONNX Runtime library to build the mmdeploy plugin for ONNX Runtime:
|
||||
```bash
|
||||
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz
|
||||
|
||||
tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
|
||||
cd onnxruntime-linux-x64-1.8.1
|
||||
export ONNXRUNTIME_DIR=$(pwd)
|
||||
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
|
||||
|
||||
cd ${MMDEPLOY_DIR} # To MMDeploy root directory
|
||||
mkdir -p build && cd build
|
||||
|
||||
# build ONNXRuntime custom ops
|
||||
cmake -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
|
||||
make -j$(nproc)
|
||||
|
||||
# build MMDeploy SDK
|
||||
cmake -DMMDEPLOY_BUILD_SDK=ON \
|
||||
-DCMAKE_CXX_COMPILER=g++-7 \
|
||||
-DOpenCV_DIR=/path/to/OpenCV/lib/cmake/OpenCV \
|
||||
-DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} \
|
||||
-DMMDEPLOY_TARGET_BACKENDS=ort \
|
||||
-DMMDEPLOY_CODEBASES=mmdet ..
|
||||
make -j$(nproc) && make install
|
||||
```
|
||||
|
||||
#### Model Conversion
|
||||
|
||||
Once we have installed MMDetection, MMDeploy, ONNX Runtime and built plugin for ONNX Runtime, we can convert the Faster R-CNN to a `.onnx` model file which can be received by ONNX Runtime. Run following commands to use our deploy tools:
|
||||
|
||||
```bash
|
||||
# Assume you have installed MMDeploy in ${MMDEPLOY_DIR} and MMDetection in ${MMDET_DIR}
|
||||
# If you do not know where to find the path. Just type `pip show mmdeploy` and `pip show mmdet` in your console.
|
||||
|
||||
python ${MMDEPLOY_DIR}/tools/deploy.py \
|
||||
${MMDEPLOY_DIR}/configs/mmdet/detection/detection_onnxruntime_dynamic.py \
|
||||
${MMDET_DIR}/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \
|
||||
${MMDET_DIR}/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
|
||||
${MMDET_DIR}/demo/demo.jpg \
|
||||
--work-dir work_dirs \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
If the script runs successfully, two images will display on the screen one by one. The first image is the infernce result of ONNX Runtime and the second image is the result of PyTorch. At the same time, an onnx model file `end2end.onnx` and three json files (SDK config files) will generate on the work directory `work_dirs`.
|
||||
|
||||
#### Run MMDeploy SDK demo
|
||||
|
||||
After model conversion, SDK Model is saved in directory ${work_dir}.
|
||||
Here is a recipe for building & running object detection demo.
|
||||
```Bash
|
||||
cd build/install/example
|
||||
|
||||
# path to onnxruntime ** libraries **
|
||||
export LD_LIBRARY_PATH=/path/to/onnxruntime/lib
|
||||
|
||||
mkdir -p build && cd build
|
||||
cmake -DOpenCV_DIR=path/to/OpenCV/lib/cmake/OpenCV \
|
||||
-DMMDeploy_DIR=${MMDEPLOY_DIR}/build/install/lib/cmake/MMDeploy ..
|
||||
make object_detection
|
||||
|
||||
# suppress verbose logs
|
||||
export SPDLOG_LEVEL=warn
|
||||
|
||||
# running the object detection example
|
||||
./object_detection cpu ${work_dirs} ${path/to/an/image}
|
||||
|
||||
```
|
||||
If the demo runs successfully, an image named "output_detection.png" is supposed to be found showing detection objects.
|
||||
|
||||
### Add New Model Support?
|
||||
|
||||
If the models you want to deploy have not been supported yet in MMDeploy, you can try to support them by yourself. Here are some documents that may help you:
|
||||
- Read [how_to_support_new_models](./tutorials/how_to_support_new_models.md) to learn more about the rewriter.
|
||||
|
||||
|
||||
|
||||
|
||||
Finally, we welcome your PR!
|
||||
You can read [how to evaluate a model](02-how-to-run/how_to_evaluate_a_model.md) for more details.
|
||||
|
|
|
|||
|
|
@ -4,48 +4,57 @@ Welcome to MMDeploy's documentation!
|
|||
You can switch between Chinese and English documents in the lower-left corner of the layout.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:maxdepth: 2
|
||||
:caption: Get Started
|
||||
|
||||
build.md
|
||||
supported_models.md
|
||||
get_started.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: Tutorials
|
||||
:caption: Build
|
||||
|
||||
tutorials/how_to_convert_model.md
|
||||
tutorials/how_to_write_config.md
|
||||
tutorials/how_to_evaluate_a_model.md
|
||||
tutorials/how_to_measure_performance_of_models.md
|
||||
tutorials/how_to_support_new_models.md
|
||||
tutorials/how_to_support_new_backends.md
|
||||
tutorials/how_to_add_test_units_for_backend_ops.md
|
||||
tutorials/how_to_test_rewritten_models.md
|
||||
tutorials/how_to_use_docker.md
|
||||
tutorials/how_to_install_mmdeploy_on_jetsons.md
|
||||
01-how-to-build/build_from_source.md
|
||||
01-how-to-build/build_from_docker.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: OpenMMLab Support
|
||||
:caption: Run & Test
|
||||
|
||||
codebases/mmcls.md
|
||||
codebases/mmdet.md
|
||||
codebases/mmseg.md
|
||||
codebases/mmedit.md
|
||||
codebases/mmocr.md
|
||||
02-how-to-run/how_to_convert_model.md
|
||||
02-how-to-run/how_to_evaluate_a_model.md
|
||||
02-how-to-run/how_to_measure_performance_of_models.md
|
||||
02-how-to-run/how_to_write_config.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: Benchmark
|
||||
|
||||
03-benchmark/supported_models.md
|
||||
03-benchmark/benchmark.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: OpenMMLab Codebase Support
|
||||
|
||||
04-supported-codebases/mmcls.md
|
||||
04-supported-codebases/mmdet.md
|
||||
04-supported-codebases/mmseg.md
|
||||
04-supported-codebases/mmedit.md
|
||||
04-supported-codebases/mmocr.md
|
||||
04-supported-codebases/mmpose.md
|
||||
04-supported-codebases/mmdet3d.md
|
||||
04-supported-codebases/mmrotate.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: Backend Support
|
||||
|
||||
backends/onnxruntime.md
|
||||
backends/tensorrt.md
|
||||
backends/openvino.md
|
||||
backends/ncnn.md
|
||||
backends/pplnn.md
|
||||
backends/torchscript.md
|
||||
05-supported-backends/onnxruntime.md
|
||||
05-supported-backends/tensorrt.md
|
||||
05-supported-backends/openvino.md
|
||||
05-supported-backends/ncnn.md
|
||||
05-supported-backends/pplnn.md
|
||||
05-supported-backends/torchscript.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
|
@ -57,9 +66,17 @@ You can switch between Chinese and English documents in the lower-left corner of
|
|||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: Benchmark
|
||||
:caption: Developer Guide
|
||||
|
||||
06-developer-guide/how_to_support_new_models.md
|
||||
06-developer-guide/how_to_support_new_backends.md
|
||||
06-developer-guide/how_to_add_test_units_for_backend_ops.md
|
||||
06-developer-guide/how_to_test_rewritten_models.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:caption: Tutorials on Model Deployment
|
||||
|
||||
benchmark.md
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Docker 安装
|
||||
# 使用 Docker 镜像
|
||||
|
||||
我们分别为 CPU 和 GPU 提供了两个 dockerfile。对于 CPU 用户，我们对接 ONNXRuntime、ncnn 和 OpenVINO 后端安装 MMDeploy。对于 GPU 用户，我们安装带有 TensorRT 后端的 MMDeploy。此外，用户可以在构建 docker 镜像时安装不同版本的 mmdeploy。
|
||||
|
||||
|
|
|
|||
|
|
@ -37,4 +37,4 @@ git clone -b master git@github.com:open-mmlab/mmdeploy.git --recursive
|
|||
- [Linux-x86_64](linux-x86_64.md)
|
||||
- [Windows](windows.md)
|
||||
- [Android-aarch64](android.md)
|
||||
- [NVIDIA Jetson](https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_install_mmdeploy_on_jetsons.html)
|
||||
- [NVIDIA Jetson](https://mmdeploy.readthedocs.io/en/latest/01-how-to-build/jetsons.html)
|
||||
|
|
|
|||
|
|
@ -17,19 +17,19 @@ ${MODEL_CFG} \
|
|||
[--speed-test] \
|
||||
[--warmup ${WARM_UP}] \
|
||||
[--log-interval ${LOG_INTERVERL}] \
|
||||
[--log2file ${LOG_RESULT_TO_FILE}] \
|
||||
[--log2file ${LOG_RESULT_TO_FILE}]
|
||||
```
|
||||
|
||||
## 参数详解
|
||||
|
||||
|参数| 说明 |
|
||||
| ------------ | ----------------------------------- |
|
||||
| deploy_cfg | 部署配置文件 |
|
||||
| model_cfg | codebase 中的模型配置文件 |
|
||||
| log2file | 保存日志和运行文件的路径 |
|
||||
| speed-test | 是否做速度测试 |
|
||||
| warm-up | 执行前是否 warm-up |
|
||||
| log-interval | 日志打印间隔 |
|
||||
| 参数 | 说明 |
|
||||
| ------------ | ------------------------- |
|
||||
| deploy_cfg | 部署配置文件 |
|
||||
| model_cfg | codebase 中的模型配置文件 |
|
||||
| log2file | 保存日志和运行文件的路径 |
|
||||
| speed-test | 是否做速度测试 |
|
||||
| warm-up | 执行前是否 warm-up |
|
||||
| log-interval | 日志打印间隔 |
|
||||
|
||||
|
||||
## 使用样例
|
||||
|
|
@ -41,7 +41,7 @@ python tools/test.py \
|
|||
{MMCLS_DIR}/configs/resnet/resnet50_b32x8_imagenet.py \
|
||||
--model model.onnx \
|
||||
--out out.pkl \
|
||||
--device cuda:0 \
|
||||
--device cuda:0
|
||||
```
|
||||
|
||||
profile 速度测试
|
||||
|
|
@ -51,5 +51,5 @@ python tools/test.py \
|
|||
{MMCLS_DIR}/configs/resnet/resnet50_b32x8_imagenet.py \
|
||||
--model model.onnx \
|
||||
--speed-test \
|
||||
--device cpu \
|
||||
--device cpu
|
||||
```
|
||||
|
|
|
|||
|
|
@ -179,7 +179,7 @@ GPU: ncnn, TensorRT, PPLNN
|
|||
<td align="center">30.41</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py">Faster-RCNN</a></td>
|
||||
<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py">Faster R-CNN</a></td>
|
||||
<td align="center">800x1344</td>
|
||||
<td align="center">88.08</td>
|
||||
<td align="center">26.52</td>
|
||||
|
|
@ -188,7 +188,7 @@ GPU: ncnn, TensorRT, PPLNN
|
|||
<td align="center">65.40</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py">Mask-RCNN</a></td>
|
||||
<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py">Mask R-CNN</a></td>
|
||||
<td align="center">800x1344</td>
|
||||
<td align="center">104.83</td>
|
||||
<td align="center">58.27</td>
|
||||
|
|
|
|||
|
|
@ -1,62 +1,337 @@
|
|||
## 操作概述
|
||||
# 操作概述
|
||||
|
||||
MMDeploy提供了一系列工具，帮助您更轻松的将OpenMMLab下的算法部署到各种设备与平台上。您可以使用我们设计的流程一“部”到位，也可以定制您自己的转换流程。这份指引将会向您展示MMDeploy的基本使用方式，并帮助您将 MMDeploy SDK 整合进您的应用。
|
||||
MMDeploy 提供了一系列工具，帮助您更轻松的将 OpenMMLab 下的算法部署到各种设备与平台上。
|
||||
|
||||
### 编译安装
|
||||
您可以使用我们设计的流程一“部”到位，也可以定制您自己的转换流程。
|
||||
|
||||
首先我们需要根据[安装指南](./01-how-to-build/build_from_source.md)正确安装MMDeploy。**注意！** 不同推理后端的安装方式略有不同。可以根据下面的介绍选择最适合您的推理后端：
|
||||
在接下来的章节中，我们将会向您展示 MMDeploy 的模型部署方式。并在 NVIDIA 设备上，以 [MMDetection](https://github.com/open-mmlab/mmdetection) Faster R-CNN 模型为例，演示 MMDeploy 的基本使用方法。
|
||||
|
||||
- [ONNXRuntime](https://mmdeploy.readthedocs.io/en/latest/backends/onnxruntime.html): ONNX Runtime 是一个跨平台的机器学习训练推理加速器，通过图形优化和变换以及硬件加速器提供优秀的推理性能。<span style="color:red">拥有完善的对ONNX的支持</span>。
|
||||
- [TensorRT](https://mmdeploy.readthedocs.io/en/latest/backends/tensorrt.html): NVIDIA® TensorRT™ 是一个用于高性能深度学习推理的开发工具包（SDK）。借助Nvidia的设备特性，TensorRT可以优化模型的推理，提供更低的推理延迟以及更高的吞吐量。如果您希望将模型部署在<span style="color:red">NVIDIA硬件设备</span>上，那么TensorRT就是一个合适的选择。
|
||||
- [ncnn](https://mmdeploy.readthedocs.io/en/latest/backends/ncnn.html): ncnn 是一个<span style="color:red">为手机端极致优化</span>的高性能神经网络前向计算框架。ncnn 从设计之初深刻考虑手机端的部署和使用。无第三方依赖，跨平台。基于 ncnn，开发者能够将深度学习算法轻松移植到手机端高效执行，开发出人工智能 APP，将 AI 带到您的指尖。
|
||||
- [PPLNN](https://mmdeploy.readthedocs.io/en/latest/backends/pplnn.html): PPLNN是一个为高效AI推理所开发的高性能深度学习推理引擎。可以用于各种ONNX模型的推理。并且<span style="color:red">对OpenMMLab有非常强的支持</span>。
|
||||
- [OpenVINO](https://mmdeploy.readthedocs.io/en/latest/backends/openvino.html): OpenVINO™ 是一个为优化与部署AI推理开发的开源工具集。该工具集<span style="color:red">可无缝集成到 Intel 硬件平台</span>，包括最新的神经网络加速芯片，Intel计算棒，边缘设备等。
|
||||
## 简介
|
||||
|
||||
选择最适合您的推理后端，点击对应的连接查看具体安装细节
|
||||
MMDeploy 定义的模型部署流程，如下图所示：
|
||||
|
||||
|
||||
### 模型转换
|
||||
### 模型转换（Model Converter）
|
||||
|
||||
一旦您完成了MMDeploy的安装，就可以用一条指令轻松的将OpenMMLab的PyTorch模型转换成推理后端支持的格式！以 [MMDetection](https://github.com/open-mmlab/mmdetection) 中的 `Faster-RCNN` 到 `TensorRT` 的转换为例：
|
||||
模型转换的主要功能是把输入的模型格式，转换为目标设备的推理引擎所要求的模型格式。
|
||||
|
||||
```bash
|
||||
# 本例假设 MMDeploy 所在目录为 ${MMDEPLOY_DIR}， MMDetection 所在目录为 ${MMDET_DIR}
|
||||
# 如果您不知道具体的安装位置，可以在终端通过命令 `pip show mmdeploy` 和 `pip show mmdet` 查看
|
||||
目前，MMDeploy 可以把 PyTorch 模型转换为 ONNX、TorchScript 等和设备无关的 IR 模型。也可以将 ONNX 模型转换为推理后端模型。两者相结合，可实现端到端的模型转换，也就是从训练端到生产端的一键式部署。
|
||||
|
||||
### MMDeploy 模型（MMDeploy Model）
|
||||
|
||||
模型转换结果的集合。它不仅包括后端模型，还包括模型的元信息。这些信息将用于推理 SDK 中。
|
||||
|
||||
### 推理 SDK（Inference SDK）
|
||||
|
||||
封装了模型的前处理、网络推理和后处理过程。对外提供多语言的模型推理接口。
|
||||
|
||||
## 准备工作
|
||||
|
||||
对于端到端的模型转换和推理，MMDeploy 依赖 Python 3.6+ 以及 PyTorch 1.5+。
|
||||
|
||||
**第一步**：从[官网](https://docs.conda.io/en/latest/miniconda.html)下载并安装 Miniconda
|
||||
|
||||
**第二步**：创建并激活 conda 环境
|
||||
```shell
|
||||
export PYTHON_VERSION=3.7
|
||||
conda create --name mmdeploy python=${PYTHON_VERSION} -y
|
||||
conda activate mmdeploy
|
||||
```
|
||||
|
||||
**第三步**: 参考[官方文档](https://pytorch.org/get-started/locally/)并安装 PyTorch
|
||||
|
||||
Model Converter 的 torch2onnx 功能依赖它。
|
||||
|
||||
在 GPU 环境下（这里我们以 Ubuntu 18.04 CUDA 11.1 为基础），您可以使用如下方式安装 PyTorch 1.8：
|
||||
|
||||
```shell
|
||||
export PYTHON_VERSION=3.7
|
||||
export PYTORCH_VERSION=1.8.0
|
||||
export TORCHVISION_VERSION=0.9.0
|
||||
export CUDA_VERSION=11.1
|
||||
|
||||
conda create -n mmdeploy python=${PYTHON_VERSION} -y
|
||||
conda activate mmdeploy
|
||||
|
||||
conda install pytorch==${PYTORCH_VERSION} torchvision==${TORCHVISION_VERSION} cudatoolkit=${CUDA_VERSION} -c pytorch -c conda-forge
|
||||
```
|
||||
|
||||
在 CPU 环境下，您可以执行：
|
||||
|
||||
```shell
|
||||
export PYTORCH_VERSION=1.8.0
|
||||
export TORCHVISION_VERSION=0.9.0
|
||||
conda install pytorch==${PYTORCH_VERSION} torchvision==${TORCHVISION_VERSION} cpuonly -c pytorch
|
||||
```
|
||||
|
||||
## 安装 MMDeploy
|
||||
|
||||
|
||||
**第一步**: 安装 mmcv-full
|
||||
|
||||
```shell
|
||||
export MMCV_VERSION=1.5.0
|
||||
export CUDA_STRING="${CUDA_VERSION/./""}"
|
||||
|
||||
python -m pip install mmcv-full==${MMCV_VERSION} -f https://download.openmmlab.com/mmcv/dist/cu${CUDA_STRING}/torch${PYTORCH_VERSION}/index.html
|
||||
```
|
||||
|
||||
**第二步**: 安装 MMDeploy
|
||||
|
||||
从 v0.5.0 之后，MMDeploy 开始提供预编译包。您可以根据目标软硬件平台，从[这里](https://github.com/open-mmlab/mmdeploy/releases)选择并下载预编译包。
|
||||
|
||||
在 NVIDIA 设备上，我们推荐使用 MMDeploy-TensoRT 预编译包：
|
||||
|
||||
```shell
|
||||
export MMDEPLOY_VERSION=0.5.0
|
||||
export TENSORRT_VERSION=8.2.3.0
|
||||
export PYTHON_VERSION=3.7
|
||||
export PYTHON_STRING="${PYTHON_VERSION/./""}"
|
||||
|
||||
wget https://github.com/open-mmlab/mmdeploy/releases/download/v${MMDEPLOY_VERSION}/mmdeploy-${MMDEPLOY_VERSION}-linux-x86_64-cuda${CUDA_VERSION}-tensorrt${TENSORRT_VERSION}.tar.gz
|
||||
tar -zxvf mmdeploy-${MMDEPLOY_VERSION}-linux-x86_64-cuda${CUDA_VERSION}-tensorrt${TENSORRT_VERSION}.tar.gz
|
||||
cd mmdeploy-${MMDEPLOY_VERSION}-linux-x86_64-cuda${CUDA_VERSION}-tensorrt${TENSORRT_VERSION}
|
||||
python -m pip install dist/mmdeploy-*-py${PYTHON_STRING}*.whl
|
||||
python -m pip install sdk/python/mmdeploy_python-*-cp${PYTHON_STRING}*.whl
|
||||
export LD_LIBRARY_PATH=$(pwd)/sdk/lib:$LD_LIBRARY_PATH
|
||||
cd ..
|
||||
```
|
||||
|
||||
```{note}
|
||||
如果 MMDeploy 没有您所需要的目标软硬件平台的预编译包，请参考源码安装文档，正确安装和配置
|
||||
```
|
||||
|
||||
**第三步**： 安装预编译包要求的推理后端
|
||||
|
||||
在本例中，我们需要安装 TensorRT（含 cuDNN）推理引擎。因在 NVIDIA 官网下载软件包，必须要登录认证，所以请预先登录并下载所需的 [TensorRT](https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-tar) 和 [cuDNN](https://developer.nvidia.com/cudnn)。**请注意： TensorRT 版本、cuDNN 版本要和 CUDA 版本匹配**
|
||||
|
||||
下载完毕后，您可以参考如下方法安装。这里，我们以 TensorRT 8.2.3.0、cuDNN 8.2 为例：
|
||||
|
||||
```shell
|
||||
export TENSORRT_VERSION=8.2.3.0
|
||||
CUDA_MAJOR="${CUDA_VERSION/\.*/""}"
|
||||
|
||||
# !!! 从 NVIDIA 官网下载 与 cuda toolkit 匹配的 tensorrt 到当前的工作目录
|
||||
tar -zxvf TensorRT-${TENSORRT_VERSION}*cuda-${CUDA_MAJOR}*.tar.gz
|
||||
python -m pip install TensorRT-${TENSORRT_VERSION}/python/tensorrt-*-cp${PYTHON_STRING}*.whl
|
||||
python -m pip install pycuda
|
||||
export TENSORRT_DIR=$(pwd)/TensorRT-${TENSORRT_VERSION}
|
||||
export LD_LIBRARY_PATH=${TENSORRT_DIR}/lib:$LD_LIBRARY_PATH
|
||||
|
||||
|
||||
# !!! 从 NVIDIA 官网下载与 cuda toolkit，tensorrt 匹配的 cudnn 到当前的工作目录
|
||||
tar -zxvf cudnn-${CUDA_MAJOR}.*-linux-x64*.tgz
|
||||
export CUDNN_DIR=$(pwd)/cuda
|
||||
export LD_LIBRARY_PATH=$CUDNN_DIR/lib64:$LD_LIBRARY_PATH
|
||||
```
|
||||
|
||||
在接下来的章节中，我们均以此环境为基础，演示 MMDeploy 的功能。
|
||||
|
||||
目前，对于 MMDeploy 支持各种推理后端的安装方法，您可以查阅以下文档：
|
||||
|
||||
- [ONNX Runtime](../en/05-supported-backends/onnxruntime.md)
|
||||
- [TensorRT](../en/05-supported-backends/tensorrt.md)
|
||||
- [PPL.NN](../en/05-supported-backends/pplnn.md)
|
||||
- [ncnn](../en/05-supported-backends/ncnn.md)
|
||||
- [OpenVINO](../en/05-supported-backends/openvino.md)
|
||||
- [LibTorch](../en/05-supported-backends/torchscript.md)
|
||||
|
||||
## 模型转换
|
||||
|
||||
在准备工作就绪后，我们可以使用 MMDeploy 中的工具 `deploy.py`，将 OpenMMLab 的 PyTorch 模型转换成推理后端支持的格式。
|
||||
|
||||
以 [MMDetection](https://github.com/open-mmlab/mmdetection) 中的 `Faster R-CNN` 为例，我们可以使用如下命令，将 PyTorch 模型转换成可部署在 NVIDIA GPU 上的 TenorRT 模型：
|
||||
|
||||
```shell
|
||||
# 克隆 mmdeploy 仓库。转换时，需要使用 mmdeploy 仓库中的配置文件，建立转换流水线
|
||||
git clone --recursive https://github.com/open-mmlab/mmdeploy.git
|
||||
python -m pip install -r mmdeploy/requirements/runtime.txt
|
||||
export MMDEPLOY_DIR=$(pwd)/mmdeploy
|
||||
|
||||
# 克隆 mmdetection 仓库。转换时，需要使用 mmdetection 仓库中的模型配置文件，构建 PyTorch nn module
|
||||
python -m pip install mmdet==2.24.0
|
||||
git clone https://github.com/open-mmlab/mmdetection.git
|
||||
export MMDET_DIR=$(pwd)/mmdetection
|
||||
|
||||
# 下载 Faster R-CNN 模型权重
|
||||
export CHECKPOINT_DIR=$(pwd)/checkpoints
|
||||
wget -P ${CHECKPOINT_DIR} https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
|
||||
|
||||
# 设置工作路径
|
||||
export WORK_DIR=$(pwd)/mmdeploy_models/faster-rcnn
|
||||
|
||||
# 执行转换命令，实现端到端的转换
|
||||
python ${MMDEPLOY_DIR}/tools/deploy.py \
|
||||
${MMDEPLOY_DIR}/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py \
|
||||
${MMDET_DIR}/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \
|
||||
${CHECKPOINT_DIR}/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
|
||||
${INPUT_IMG} \
|
||||
${MMDET_DIR}/demo/demo.jpg \
|
||||
--work-dir ${WORK_DIR} \
|
||||
--device cuda:0 \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
`${MMDEPLOY_DIR}/tools/deploy.py` 是一个方便模型转换的工具。可以阅读 [如何转换模型](./02-how-to-run/convert_model.md) 了解更多细节。转换后的模型以及一些其他的信息将会被保存在 `${WORK_DIR}` 中。MMDeploy SDK 可以使用这些信息进行模型推理。
|
||||
`${MMDEPLOY_DIR}/tools/deploy.py` 是一个方便模型转换的工具。您可以阅读 [如何转换模型](./02-how-to-run/convert_model.md) 了解更多细节。
|
||||
|
||||
`detection_tensorrt_dynamic-320x320-1344x1344.py` 是一个包含所有转换需要的可配置参数的配置文件。该文件的命名遵循如下规则：
|
||||
`detection_tensorrt_dynamic-320x320-1344x1344.py` 是一个参数配置文件。该文件的命名遵循如下规则：
|
||||
|
||||
```bash
|
||||
<任务名>_<推理后端>-[后端特性]_<动态模型支持>.py
|
||||
```
|
||||
|
||||
可以很容易的通过文件名来确定最适合的那个配置文件。如果您希望定制自己的转换配置，可以修改配置文件中的具体条目。我们提供了 [如何编写配置文件](https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_write_config.html) 来指导您如何进行编辑。
|
||||
可以很容易的通过文件名来确定最适合的那个配置文件。如果您希望定制自己的转换配置，可以参考[如何编写配置文件](./02-how-to-run/write_config.md)修改参数。
|
||||
|
||||
### 模型推理
|
||||
## 模型推理
|
||||
|
||||
得到了转换后的模型之后，就可以使用推理后端提供的API来进行推理。也许您想绕过API的学习与开发，确认下转换后的模型效果。我们提供了对这些API的统一封装：
|
||||
在转换完成后，您既可以使用 Model Converter 进行推理，也可以使用 Inference SDK。前者使用 Python 开发，后者主要使用 C/C++ 开发。
|
||||
|
||||
### 使用 Model Converter 的推理 API
|
||||
|
||||
Model Converter 屏蔽了推理后端接口的差异，对其推理 API 进行了统一封装，接口名称为 `inference_model`。
|
||||
|
||||
以上文中 Faster R-CNN 的 TensorRT 模型为例，您可以使用如下方式进行模型推理工作：
|
||||
|
||||
```python
|
||||
from mmdeploy.apis import inference_model
|
||||
import os
|
||||
|
||||
model_cfg = os.getenv('MMDET_DIR') + '/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py'
|
||||
deploy_cfg = os.getenv('MMDEPLOY_DIR') + '/configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py'
|
||||
backend_files = os.getenv('WORK_DIR') + '/end2end.engine'
|
||||
|
||||
result = inference_model(model_cfg, deploy_cfg, backend_files, img=img, device=device)
|
||||
```
|
||||
|
||||
`inference_model`会创建一个对后端模型的封装，通过该封装进行推理。推理的结果会保持与OpenMMLab中原模型同样的格式。
|
||||
`inference_model`会创建一个对后端模型的封装，通过该封装进行推理。推理的结果会保持与 OpenMMLab 中原模型同样的格式。
|
||||
|
||||
### 模型评估
|
||||
```{note}
|
||||
MMDeploy 转出的后端模型，您可以直接使用后端 API 进行推理。不过，因为 MMDeploy 拥有 TensorRT、ONNX Runtime 等自定义算子，
|
||||
您需要先加载对应的自定义算子库，然后再使用后端 API。
|
||||
```
|
||||
|
||||
转换后的模型是否会带来一些精度损失？推理后端是否为模型带来效率提升？我们提供了工具来帮助完成验证与评估工作。以TensorRT的Faster-RCNN模型为例：
|
||||
### 使用推理 SDK
|
||||
|
||||
您也可以使用 MMDeploy SDK 进行推理。以上文中转出的 Faster R-CNN TensorRT 模型为例，接下来的章节将介绍如何使用 SDK 的 FFI 进行模型推理。
|
||||
|
||||
#### Python API
|
||||
|
||||
```python
|
||||
from mmdeploy_python import Detector
|
||||
import os
|
||||
import cv2
|
||||
|
||||
# 获取转换后的 mmdeploy model 路径
|
||||
model_path = os.getenv('WORK_DIR')
|
||||
# 从 mmdetection repo 中，获取 demo.jpg 路径
|
||||
image_path = '/'.join((os.getenv('MMDET_DIR'), 'demo/demo.jpg'))
|
||||
|
||||
img = cv2.imread(image_path)
|
||||
detector = Detector(model_path, 'cuda', 0)
|
||||
bboxes, labels, _ = detector([img])[0]
|
||||
|
||||
indices = [i for i in range(len(bboxes))]
|
||||
for index, bbox, label_id in zip(indices, bboxes, labels):
|
||||
[left, top, right, bottom], score = bbox[0:4].astype(int), bbox[4]
|
||||
if score < 0.3:
|
||||
continue
|
||||
cv2.rectangle(img, (left, top), (right, bottom), (0, 255, 0))
|
||||
|
||||
cv2.imwrite('output_detection.png', img)
|
||||
|
||||
```
|
||||
|
||||
更多模型的 SDK Python API 应用样例，请查阅[这里](https://github.com/open-mmlab/mmdeploy/demo/python)。
|
||||
|
||||
```{note}
|
||||
如果您使用源码安装方式, 请把 ${MMDEPLOY_DIR}/build/lib 加入到环境变量 PYTHONPATH 中。
|
||||
否则会遇到错误’ModuleNotFoundError: No module named 'mmdeploy_python'
|
||||
```
|
||||
|
||||
#### C API
|
||||
|
||||
使用 C API 进行模型推理的流程符合下面的模式：
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[创建推理句柄] --> B(读取图像)
|
||||
B --> C(应用句柄进行推理)
|
||||
C --> D[处理推理结果]
|
||||
D -->E[销毁结果]
|
||||
E -->F[销毁推理句柄]
|
||||
```
|
||||
|
||||
以下是这个流程的具体应用过程：
|
||||
|
||||
```C++
|
||||
#include <cstdlib>
|
||||
#include <opencv2/opencv.hpp>
|
||||
#include "detector.h"
|
||||
|
||||
int main() {
|
||||
const char* device_name = "cuda";
|
||||
int device_id = 0;
|
||||
|
||||
// 获取转换后的 mmdeploy model 路径
|
||||
std::string model_path = std::getenv("WORK_DIR");
|
||||
// 从 mmdetection repo 中，获取 demo.jpg 路径
|
||||
std::string image_path = std::getenv("MMDET_DIR") + "/demo/demo.jpg";
|
||||
|
||||
// 创建推理句柄
|
||||
mm_handle_t detector{};
|
||||
int status{};
|
||||
status = mmdeploy_detector_create_by_path(model_path, device_name, device_id, &detector);
|
||||
assert(status == MM_SUCCESS);
|
||||
|
||||
// 读取图像
|
||||
cv::Mat img = cv::imread(image_path);
|
||||
assert(img.data);
|
||||
|
||||
// 应用句柄进行推理
|
||||
mm_mat_t mat{img.data, img.rows, img.cols, 3, MM_BGR, MM_INT8};
|
||||
mm_detect_t *bboxes{};
|
||||
int *res_count{};
|
||||
status = mmdeploy_detector_apply(detector, &mat, 1, &bboxes, &res_count);
|
||||
assert (status == MM_SUCCESS);
|
||||
|
||||
// 处理推理结果: 此处我们选择可视化推理结果
|
||||
for (int i = 0; i < *res_count; ++i) {
|
||||
const auto &box = bboxes[i].bbox;
|
||||
if (bboxes[i].score < 0.3) {
|
||||
continue;
|
||||
}
|
||||
cv::rectangle(img, cv::Point{(int)box.left, (int)box.top},
|
||||
cv::Point{(int)box.right, (int)box.bottom}, cv::Scalar{0, 255, 0});
|
||||
}
|
||||
|
||||
cv::imwrite('output_detection.png', img);
|
||||
|
||||
// 销毁结果
|
||||
mmdeploy_detector_release_result(bboxes, res_count, 1);
|
||||
// 销毁推理句柄
|
||||
mmdeploy_detector_destroy(detector);
|
||||
return 0;
|
||||
}
|
||||
```
|
||||
|
||||
在您的项目CMakeLists中，增加：
|
||||
|
||||
```Makefile
|
||||
find_package(MMDeploy REQUIRED)
|
||||
mmdeploy_load_static(${YOUR_AWESOME_TARGET} MMDeployStaticModules)
|
||||
mmdeploy_load_dynamic(${YOUR_AWESOME_TARGET} MMDeployDynamicModules)
|
||||
target_link_libraries(${YOUR_AWESOME_TARGET} PRIVATE MMDeployLibs)
|
||||
```
|
||||
|
||||
编译时，使用 -DMMDeploy_DIR，传入MMDeloyConfig.cmake所在的路径。它在预编译包中的sdk/lib/cmake/MMDeloy下。
|
||||
更多模型的 SDK C API 应用样例，请查阅[此处](https://github.com/open-mmlab/mmdeploy/demo/csrc)。
|
||||
|
||||
#### C# API
|
||||
|
||||
因篇幅所限，本文不展示具体案例。请参考[这里](https://github.com/open-mmlab/mmdeploy/demo/csharp)，了解 SDK C# API 的用法。
|
||||
|
||||
## 模型精度评估
|
||||
|
||||
为了测试部署模型的精度，推理效率，我们提供了 `tools/test.py` 来帮助完成相关工作。以上文中的部署模型为例：
|
||||
|
||||
```bash
|
||||
python ${MMDEPLOY_DIR}/tools/test.py \
|
||||
|
|
@ -67,148 +342,8 @@ python ${MMDEPLOY_DIR}/tools/test.py \
|
|||
--device cuda:0
|
||||
```
|
||||
|
||||
请阅读 [如何进行模型评估](https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_evaluate_a_model.html) 了解关于 `tools/test.py` 的使用细节。
|
||||
|
||||
### 整合 MMDeploy SDK
|
||||
|
||||
请参考[安装指南](./01-how-to-build/build_from_source.md)，在编译 MMDeploy 时开启`MMDEPLOY_BUILD_SDK`以安装 MMDeploy SDK。
|
||||
成功安装后，安装目录的文件结构将会如下所示：
|
||||
|
||||
```
|
||||
install
|
||||
├── example
|
||||
├── include
|
||||
│ ├── c
|
||||
│ └── cpp
|
||||
└── lib
|
||||
```{note}
|
||||
关于 --model 选项，当使用 Model Converter 进行推理时，它代表转换后的推理后端模型的文件路径。而当使用 SDK 测试模型精度时，该选项表示 MMDeploy Model 的路径.
|
||||
```
|
||||
|
||||
其中 `include/c` 和 `include/cpp` 分别对应 C 与 C++ API。
|
||||
|
||||
**注意：C++的API尚处于不稳定阶段，可能会在未来发生变动，暂时不推荐使用**
|
||||
|
||||
在 example 目录下有包含分类，目标检测，图像分割等数个范例。可以通过他们学习如何使用 MMDeploy SDK 以及如何将 ${MMDeploy_LIBS} 链接到应用程序。
|
||||
|
||||
### 从零开始进行模型部署
|
||||
|
||||
这章节将会展示如何从零开始进行 MMDetection 中 Faster-RCNN 的模型部署与推理。
|
||||
|
||||
#### 创建虚拟环境并安装 MMDetection
|
||||
|
||||
请运行下面的指令在Anaconda环境中[安装MMDetection](https://mmdetection.readthedocs.io/zh_CN/latest/get_started.html#a-from-scratch-setup-script)。
|
||||
|
||||
```bash
|
||||
conda create -n openmmlab python=3.7 -y
|
||||
conda activate openmmlab
|
||||
|
||||
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=10.2 -c pytorch -y
|
||||
|
||||
# 安装 mmcv
|
||||
pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.8/index.html
|
||||
|
||||
# 安装mmdetection
|
||||
git clone https://github.com/open-mmlab/mmdetection.git
|
||||
cd mmdetection
|
||||
pip install -r requirements/build.txt
|
||||
pip install -v -e .
|
||||
```
|
||||
|
||||
#### 下载 Faster R-CNN 的模型文件
|
||||
|
||||
请从[本链接](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth)下载模型文件，放在`{MMDET_ROOT}/checkpoints`目录下。其中`{MMDET_ROOT}`为您的MMDetection的根目录。
|
||||
|
||||
#### 安装 MMDeploy 以及 ONNX Runtime
|
||||
|
||||
请运行下面的指令在Anaconda环境中[安装MMDeploy](./01-how-to-build/build_from_source.md)。
|
||||
|
||||
```bash
|
||||
conda activate openmmlab
|
||||
|
||||
git clone https://github.com/open-mmlab/mmdeploy.git
|
||||
cd mmdeploy
|
||||
git submodule update --init --recursive
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
一旦我们完成MMDeploy的安装，我们需要选择一个模型的推理引擎。这里我们以ONNX Runtime为例。运行下面命令来[安装ONNX Runtime](https://mmdeploy.readthedocs.io/en/latest/backends/onnxruntime.html)：
|
||||
|
||||
```bash
|
||||
pip install onnxruntime==1.8.1
|
||||
```
|
||||
|
||||
然后下载 ONNX Runtime Library来编译 MMDeploy 中的算子插件：
|
||||
|
||||
```bash
|
||||
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-1.8.1.tgz
|
||||
|
||||
tar -zxvf onnxruntime-linux-x64-1.8.1.tgz
|
||||
cd onnxruntime-linux-x64-1.8.1
|
||||
export ONNXRUNTIME_DIR=$(pwd)
|
||||
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
|
||||
|
||||
cd ${MMDEPLOY_DIR} # 到 MMDeploy 根目录下
|
||||
mkdir -p build && cd build
|
||||
|
||||
# 编译自定义算子
|
||||
cmake -DMMDEPLOY_TARGET_BACKENDS=ort -DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} ..
|
||||
make -j$(nproc)
|
||||
|
||||
# 编译 MMDeploy SDK
|
||||
cmake -DMMDEPLOY_BUILD_SDK=ON \
|
||||
-DCMAKE_CXX_COMPILER=g++-7 \
|
||||
-DOpenCV_DIR=/path/to/OpenCV/lib/cmake/OpenCV \
|
||||
-DONNXRUNTIME_DIR=${ONNXRUNTIME_DIR} \
|
||||
-DMMDEPLOY_TARGET_BACKENDS=ort \
|
||||
-DMMDEPLOY_CODEBASES=mmdet ..
|
||||
make -j$(nproc) && make install
|
||||
```
|
||||
|
||||
#### 模型转换
|
||||
|
||||
当我们成功安装 MMDetection, MMDeploy, ONNX Runtime 以及编译ONNX Runtime的插件之后，我们就可以将 Faster-RCNN 转换成 ONNX 格式，以支持 ONNX Runtime 下的模型推理。运行下面指令来使用我们的部署工具：
|
||||
|
||||
```bash
|
||||
# 本例假设 MMDeploy 所在目录为 ${MMDEPLOY_DIR}， MMDetection 所在目录为 ${MMDET_DIR}
|
||||
# 如果您不知道具体的安装位置，可以在终端通过命令 `pip show mmdeploy` 和 `pip show mmdet` 查看
|
||||
|
||||
python ${MMDEPLOY_DIR}/tools/deploy.py \
|
||||
${MMDEPLOY_DIR}/configs/mmdet/detection/detection_onnxruntime_dynamic.py \
|
||||
${MMDET_DIR}/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \
|
||||
${MMDET_DIR}/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
|
||||
${MMDET_DIR}/demo/demo.jpg \
|
||||
--work-dir work_dirs \
|
||||
--device cpu \
|
||||
--show \
|
||||
--dump-info
|
||||
```
|
||||
|
||||
如果脚本运行成功，屏幕上将先后显示两张图片。首先是ONNX Runtime推理结果的可视化，然后是PyTorch推理结果的可视化。同时，ONNX文件 `end2end.onnx` 以及数个json文件（供SDK使用）将会被生成在`work_dirs`目录下。
|
||||
|
||||
#### 运行 MMDeploy SDK demo
|
||||
|
||||
进行模型转换后，SDK模型将被保存在`work_dirs`目录下。
|
||||
可以用如下方法编译并运行 MMDeploy SDK demo。
|
||||
|
||||
```bash
|
||||
cd build/install/example
|
||||
|
||||
# 配置ONNX Runtime库目录
|
||||
export LD_LIBRARY_PATH=/path/to/onnxruntime/lib
|
||||
mkdir -p build && cd build
|
||||
cmake -DOpenCV_DIR=path/to/OpenCV/lib/cmake/OpenCV \
|
||||
-DMMDeploy_DIR=${MMDEPLOY_DIR}/build/install/lib/cmake/MMDeploy ..
|
||||
make object_detection
|
||||
# 设置log等级
|
||||
export SPDLOG_LEVEL=warn
|
||||
# 运行目标检测范例程序
|
||||
./object_detection cpu ${work_dirs} ${path/to/an/image}
|
||||
```
|
||||
|
||||
如果demo运行成功，将会生成一张名为"output_detection.png"的图片，展示模型输出的可视化效果。
|
||||
|
||||
### 新模型的支持？
|
||||
|
||||
如果您希望使用的模型尚未被 MMDeploy 所支持，您可以尝试自己添加对应的支持。我们准备了如下的文档：
|
||||
- 请阅读[如何支持新模型](https://mmdeploy.readthedocs.io/en/latest/tutorials/how_to_support_new_models.html)了解我们的模型重写机制。
|
||||
|
||||
最后，欢迎大家踊跃提PR。
|
||||
请阅读 [如何进行模型评估](./02-how-to-run/profile_model.md) 了解关于 `tools/test.py` 的使用细节。
|
||||
|
|
|
|||
|
|
@ -8,9 +8,6 @@
|
|||
:caption: 编译
|
||||
|
||||
01-how-to-build/build_from_source.md
|
||||
01-how-to-build/android.md
|
||||
01-how-to-build/linux-x86_64.md
|
||||
01-how-to-build/windows.md
|
||||
01-how-to-build/build_from_docker.md
|
||||
|
||||
.. toctree::
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 93 KiB After Width: | Height: | Size: 189 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 140 KiB |
Loading…
Reference in New Issue