mirrors
/
mmdeploy
mirror of https://github.com/open-mmlab/mmdeploy.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

[Refactor] update tutorial about how to support new backend (#1574)

pull/1589/head
q.yao 2022-12-29 12:05:31 +08:00 committed by GitHub
parent e3f033ee5d
commit fc98472e9c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 142 additions and 138 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

130
docs/en/07-developer-guide/support_new_backend.md
View File

@ -26,56 +26,7 @@ The backends in MMDeploy must support the ONNX. The backend loads the ".onnx" fi
TENSORRT = 'tensorrt'
```
2. Add a corresponding package (a folder with `__init__.py`) in `mmdeploy/backend/`. For example, `mmdeploy/backend/tensorrt`. In the `__init__.py`, there must be a function named `is_available` which checks if users have installed the backend library. If the check is passed, then the remaining files of the package will be loaded.
**Example**:
```Python
# mmdeploy/backend/tensorrt/__init__.py
def is_available():
return importlib.util.find_spec('tensorrt') is not None
if is_available():
from .utils import from_onnx, load, save
from .wrapper import TRTWrapper
__all__ = [
'from_onnx', 'save', 'load', 'TRTWrapper'
]
```
3. Create a config file in `configs/_base_/backends` (e.g., `configs/_base_/backends/tensorrt.py`). If the backend just takes the '.onnx' file as input, the new config can be simple. The config of the backend only consists of one field denoting the name of the backend (which should be same as the name in `mmdeploy/utils/constants.py`).
**Example**:
```python
backend_config = dict(type='onnxruntime')
```
If the backend requires other files, then the arguments for the conversion from ".onnx" file to backend files should be included in the config file.
**Example:**
```Python
backend_config = dict(
type='tensorrt',
common_config=dict(
fp16_mode=False, max_workspace_size=0))
```
After possessing a base backend config file, you can easily construct a complete deploy config through inheritance. Please refer to our [config tutorial](../02-how-to-run/write_config.md) for more details. Here is an example:
```Python
_base_ = ['../_base_/backends/onnxruntime.py']
codebase_config = dict(type='mmcls', task='Classification')
onnx_config = dict(input_shape=None)
```
4. If the backend requires model files or weight files other than a ".onnx" file, create a `onnx2backend.py` file in the corresponding folder (e.g., create `mmdeploy/backend/tensorrt/onnx2tensorrt.py`). Then add a conversion function `onnx2backend` in the file. The function should convert a given ".onnx" file to the required backend files in a given work directory. There are no requirements on other parameters of the function and the implementation details. You can use any tools for conversion. Here are some examples:
2. Add a corresponding package (a folder with `__init__.py`) in `mmdeploy/backend/`. For example, `mmdeploy/backend/tensorrt`. If the backend requires model files or weight files other than a ".onnx" file, create a `onnx2backend.py` file in the corresponding folder (e.g., create `mmdeploy/backend/tensorrt/onnx2tensorrt.py`). Then add a conversion function `onnx2backend` in the file. The function should convert a given ".onnx" file to the required backend files in a given work directory. There are no requirements on other parameters of the function and the implementation details. You can use any tools for conversion. Here are some examples:
**Use Python script:**
@ -106,6 +57,65 @@ The backends in MMDeploy must support the ONNX. The backend loads the ".onnx" fi
call([onnx2ncnn_path, onnx_path, save_param, save_bin])\
```
3. Create a backend manager class and implement the interface to support model conversion, version check and other features.
**Example**:
```Python
# register the backend manager
# the backend manager derive from BaseBackendManager
@BACKEND_MANAGERS.register('tensorrt')
class TensorRTManager(BaseBackendManager):
@classmethod
def is_available(cls, with_custom_ops: bool = False) -> bool:
....
@classmethod
def get_version(cls) -> str:
....
@classmethod
def to_backend(cls,
ir_files: Sequence[str],
work_dir: str,
deploy_cfg: Any,
log_level: int = logging.INFO,
device: str = 'cpu',
**kwargs) -> Sequence[str]:
...
```
4. Create a config file in `configs/_base_/backends` (e.g., `configs/_base_/backends/tensorrt.py`). If the backend just takes the '.onnx' file as input, the new config can be simple. The config of the backend only consists of one field denoting the name of the backend (which should be same as the name in `mmdeploy/utils/constants.py`).
**Example**:
```python
backend_config = dict(type='onnxruntime')
```
If the backend requires other files, then the arguments for the conversion from ".onnx" file to backend files should be included in the config file.
**Example:**
```Python
backend_config = dict(
type='tensorrt',
common_config=dict(
fp16_mode=False, max_workspace_size=0))
```
After possessing a base backend config file, you can easily construct a complete deploy config through inheritance. Please refer to our [config tutorial](../02-how-to-run/write_config.md) for more details. Here is an example:
```Python
_base_ = ['../_base_/backends/onnxruntime.py']
codebase_config = dict(type='mmcls', task='Classification')
onnx_config = dict(input_shape=None)
```
5. Define APIs in a new package in `mmdeploy/apis`.
**Example:**
@ -123,22 +133,6 @@ The backends in MMDeploy must support the ONNX. The backend loads the ".onnx" fi
__all__ += ['onnx2ncnn', 'get_output_model_file']
```
Create a backend manager class which derive from `BackendManager`, implement its `to_backend` static method.
**Example:**
```Python
@classmethod
def to_backend(cls,
ir_files: Sequence[str],
deploy_cfg: Any,
work_dir: str,
log_level: int = logging.INFO,
device: str = 'cpu',
**kwargs) -> Sequence[str]:
return ir_files
```
6. Convert the models of OpenMMLab to backends (if necessary) and inference on backend engine. If you find some incompatible operators when testing, you can try to rewrite the original model for the backend following the [rewriter tutorial](support_new_model.md) or add custom operators.
7. Add docstring and unit tests for new code :).
@ -197,13 +191,13 @@ Although the backend engines are usually implemented in C/C++, it is convenient
self.sess.run_with_iobinding(io_binding)
```
4. Create a backend manager class which derive from `BackendManager`, implement its `build_wrapper` static method.
4. Implement `build_wrapper` method in the backend manager.
**Example:**
```Python
@BACKEND_MANAGERS.register('onnxruntime')
class ONNXRuntimeUtils(BaseBackendManager):
class ONNXRuntimeManager(BaseBackendManager):
@classmethod
def build_wrapper(cls,

150
docs/zh_cn/07-developer-guide/support_new_backend.md
View File

@ -26,56 +26,7 @@ MMDeploy 中的后端必须支持 ONNX，因此后端能直接加载“.onnx”
TENSORRT = 'tensorrt'
```
2. 在 `mmdeploy/backend/` 目录下添加相应的库(一个包括 `__init__.py` 的文件夹),例如， `mmdeploy/backend/tensorrt` 。在 `__init__.py` 中，必须有一个名为 `is_available` 的函数检查用户是否安装了后端库。如果检查通过，则将加载库的剩余文件。
**例子**:
```Python
# mmdeploy/backend/tensorrt/__init__.py
def is_available():
return importlib.util.find_spec('tensorrt') is not None
if is_available():
from .utils import from_onnx, load, save
from .wrapper import TRTWrapper
__all__ = [
'from_onnx', 'save', 'load', 'TRTWrapper'
]
```
3. 在 `configs/_base_/backends` 目录中创建一个配置文件(例如， `configs/_base_/backends/tensorrt.py` )。如果新后端引擎只是将“.onnx”文件作为输入，那么新的配置可以很简单,对应配置只需包含一个表示后端名称的字段(但也应该与 `mmdeploy/utils/constants.py` 中的名称相同)。
**例子**
```python
backend_config = dict(type='tensorrt')
```
但如果后端需要其他文件，则从“.onnx”文件转换为后端文件所需的参数也应包含在配置文件中。
**例子**
```Python
backend_config = dict(
type='tensorrt',
common_config=dict(
fp16_mode=False, max_workspace_size=0))
```
在拥有一个基本的后端配置文件后，您已经可以通过继承轻松构建一个完整的部署配置。有关详细信息，请参阅我们的[配置教程](../02-how-to-run/write_config.md)。下面是一个例子：
```Python
_base_ = ['../_base_/backends/tensorrt.py']
codebase_config = dict(type='mmcls', task='Classification')
onnx_config = dict(input_shape=None)
```
4. 如果新后端需要模型文件或权重文件而不是“.onnx”文件，则需要在相应的文件夹中创建一个 `onnx2backend.py` 文件(例如,创建 `mmdeploy/backend/tensorrt/onnx2tensorrt.py` )。然后在文件中添加一个转换函数`onnx2backend`。该函数应将给定的“.onnx”文件转换为给定工作目录中所需的后端文件。对函数的其他参数和实现细节没有要求，您可以使用任何工具进行转换。下面有些例子：
2. 在 `mmdeploy/backend/` 目录下添加相应的库(一个包括 `__init__.py` 的文件夹),例如， `mmdeploy/backend/tensorrt` 。如果新后端需要模型文件或权重文件而不是“.onnx”文件，则需要在相应的文件夹中创建一个 `onnx2backend.py` 文件(例如,创建 `mmdeploy/backend/tensorrt/onnx2tensorrt.py` )。然后在文件中添加一个转换函数`onnx2backend`。该函数应将给定的“.onnx”文件转换为给定工作目录中所需的后端文件。对函数的其他参数和实现细节没有要求，您可以使用任何工具进行转换。下面有些例子：
**使用python脚本**
@ -110,6 +61,81 @@ MMDeploy 中的后端必须支持 ONNX，因此后端能直接加载“.onnx”
**例子**
```Python
@classmethod
def to_backend(cls,
ir_files: Sequence[str],
deploy_cfg: Any,
work_dir: str,
log_level: int = logging.INFO,
device: str = 'cpu',
**kwargs) -> Sequence[str]:
return ir_files
```
3. 创建一个 BaseBackendManager 的派生类，注册并实现对应接口
**例子**:
```Python
# 注册 backend manager
# 该类派生自 BaseBackendManager
@BACKEND_MANAGERS.register('tensorrt')
class TensorRTManager(BaseBackendManager):
@classmethod
def is_available(cls, with_custom_ops: bool = False) -> bool:
....
@classmethod
def get_version(cls) -> str:
....
@classmethod
def to_backend(cls,
ir_files: Sequence[str],
work_dir: str,
deploy_cfg: Any,
log_level: int = logging.INFO,
device: str = 'cpu',
**kwargs) -> Sequence[str]:
...
```
4. 在 `configs/_base_/backends` 目录中创建一个配置文件(例如， `configs/_base_/backends/tensorrt.py` )。如果新后端引擎只是将“.onnx”文件作为输入，那么新的配置可以很简单,对应配置只需包含一个表示后端名称的字段(但也应该与 `mmdeploy/utils/constants.py` 中的名称相同)。
**例子**
```python
backend_config = dict(type='tensorrt')
```
但如果后端需要其他文件，则从“.onnx”文件转换为后端文件所需的参数也应包含在配置文件中。
**例子**
```Python
backend_config = dict(
type='tensorrt',
common_config=dict(
fp16_mode=False, max_workspace_size=0))
```
在拥有一个基本的后端配置文件后，您已经可以通过继承轻松构建一个完整的部署配置。有关详细信息，请参阅我们的[配置教程](../02-how-to-run/write_config.md)。下面是一个例子：
```Python
_base_ = ['../_base_/backends/tensorrt.py']
codebase_config = dict(type='mmcls', task='Classification')
onnx_config = dict(input_shape=None)
```
5. 在 `mmdeploy/apis/<backend>` 中添加希望给用户使用的 API
**例子**
```Python
# mmdeploy/apis/ncnn/__init__.py
@ -123,25 +149,9 @@ MMDeploy 中的后端必须支持 ONNX，因此后端能直接加载“.onnx”
__all__ += ['onnx2ncnn', 'get_output_model_file']
```
然后根据需要使用这些 APIs 为 `tools/deploy.py` 添加相关转换代码
6. 将 OpenMMLab 的模型转换后(如有必要)并在后端引擎上进行推理。如果在测试时发现一些不兼容的算子，可以尝试按照[重写器教程](support_new_model.md)为后端重写原始模型或添加自定义算子。
**例子**
```Python
@classmethod
def to_backend(cls,
ir_files: Sequence[str],
deploy_cfg: Any,
work_dir: str,
log_level: int = logging.INFO,
device: str = 'cpu',
**kwargs) -> Sequence[str]:
return ir_files
```
5. 将 OpenMMLab 的模型转换后(如有必要)并在后端引擎上进行推理。如果在测试时发现一些不兼容的算子，可以尝试按照[重写器教程](support_new_model.md)为后端重写原始模型或添加自定义算子。
6. 为新后端引擎代码添加相关注释和单元测试:).
7. 为新后端引擎代码添加相关注释和单元测试:).
## 支持后端推理
@ -198,13 +208,13 @@ MMDeploy 中的后端必须支持 ONNX，因此后端能直接加载“.onnx”
self.sess.run_with_iobinding(io_binding)
```
4. 从 `BackendManager` 派生接口类，实现 `build_wrapper` 静态方法
4. 从 `BaseBackendManager` 派生接口类，实现 `build_wrapper` 静态方法
**例子**
```Python
@BACKEND_MANAGERS.register('onnxruntime')
class ONNXRuntimeUtils(BaseBackendManager):
class ONNXRuntimeManager(BaseBackendManager):
@classmethod
def build_wrapper(cls,