diff --git a/docs/en/04-supported-codebases/mmaction2.md b/docs/en/04-supported-codebases/mmaction2.md index 4bb4a4429..96a10df40 100644 --- a/docs/en/04-supported-codebases/mmaction2.md +++ b/docs/en/04-supported-codebases/mmaction2.md @@ -1,187 +1,11 @@ -# MMAction2 Deployment - -- [MMAction2 Deployment](#mmaction2-deployment) - - [Installation](#installation) - - [Install mmaction2](#install-mmaction2) - - [Install mmdeploy](#install-mmdeploy) - - [Convert model](#convert-model) - - [Convert video recognition model](#convert-video-recognition-model) - - [Model specification](#model-specification) - - [Model Inference](#model-inference) - - [Backend model inference](#backend-model-inference) - - [SDK model inference](#sdk-model-inference) - - [Video recognition SDK model inference](#video-recognition-sdk-model-inference) - - [Supported models](#supported-models) - -______________________________________________________________________ +# MMAction2 Support [MMAction2](https://github.com/open-mmlab/mmaction2) is an open-source toolbox for video understanding based on PyTorch. It is a part of the [OpenMMLab](https://openmmlab.com) project. -## Installation - -### Install mmaction2 +## Install mmaction2 Please follow the [installation guide](https://github.com/open-mmlab/mmaction2#installation) to install mmaction2. -### Install mmdeploy - -There are several methods to install mmdeploy, among which you can choose an appropriate one according to your target platform and device. - -**Method I：** Install precompiled package - -You can download the latest release package from [here](https://github.com/open-mmlab/mmdeploy/releases) - -**Method II：** Build using scripts - -If your target platform is **Ubuntu 18.04 or later version**, we encourage you to run -[scripts](../01-how-to-build/build_from_script.md). For example, the following commands install mmdeploy as well as inference engine - `ONNX Runtime`. - -```shell -git clone --recursive https://github.com/open-mmlab/mmdeploy.git -cd mmdeploy -python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc) -export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH -export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH -``` - -**Method III:** Build from source - -If neither **I** nor **II** meets your requirements, [building mmdeploy from source](../01-how-to-build/build_from_source.md) is the last option. - -## Convert model - -You can use [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/master/tools/deploy.py) to convert mmaction2 models to the specified backend models. Its detailed usage can be learned from [here](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage). - -When using `tools/deploy.py`, it is crucial to specify the correct deployment config. We've already provided builtin deployment config [files](https://github.com/open-mmlab/mmdeploy/tree/master/configs/mmaction) of all supported backends for mmaction2, under which the config file path follows the pattern: - -``` -{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py -``` - -其中： - -- **{task}:** task in mmaction2. -- **{backend}:** inference backend, such as onnxruntime, tensorrt, pplnn, ncnn, openvino, coreml etc. -- **{precision}:** fp16, int8. When it's empty, it means fp32 -- **{static | dynamic}:** static shape or dynamic shape -- **{shape}:** input shape or shape range of a model -- **{2d/3d}:** model type - -In the next part，we will take `tsn` model from `video recognition` task as an example, showing how to convert them to onnx model that can be inferred by ONNX Runtime. - -### Convert video recognition model - -```shell -cd mmdeploy - -# download tsn model from mmaction2 model zoo -mim download mmaction2 --config tsn_r50_1x1x3_100e_kinetics400_rgb --dest . - -# convert mmaction2 model to onnxruntime model with dynamic shape -python tools/deploy.py \ - configs/mmaction/video-recognition/video-recognition_onnxruntime_static.py \ - tsn_r50_1x1x3_100e_kinetics400_rgb.py \ - tsn_r50_256p_1x1x3_100e_kinetics400_rgb_20200725-22592236.pth \ - tests/data/arm_wrestling.mp4 \ - --work-dir mmdeploy_models/mmaction/tsn/ort \ - --device cpu \ - --show \ - --dump-info -``` - -## Model specification - -Before moving on to model inference chapter, let's know more about the converted model structure which is very important for model inference. - -The converted model locates in the working directory like `mmdeploy_models/mmaction/tsn/ort` in the previous example. It includes: - -``` -mmdeploy_models/mmaction/tsn/ort -├── deploy.json -├── detail.json -├── end2end.onnx -└── pipeline.json -``` - -in which, - -- **end2end.onnx**: backend model which can be inferred by ONNX Runtime -- \***.json**: the necessary information for mmdeploy SDK - -The whole package **mmdeploy_models/mmaction/tsn/ort** is defined as **mmdeploy SDK model**, i.e., **mmdeploy SDK model** includes both backend model and inference meta information. - -## Model Inference - -### Backend model inference - -Take the previous converted `end2end.onnx` mode of `tsn` as an example, you can use the following code to inference the model and visualize the results. - -```python -from mmdeploy.apis.utils import build_task_processor -from mmdeploy.utils import get_input_shape, load_config -import numpy as np -import torch - -deploy_cfg = 'configs/mmaction/video-recognition/video-recognition_onnxruntime_static.py' -model_cfg = 'tsn_r50_1x1x3_100e_kinetics400_rgb.py' -device = 'cpu' -backend_model = ['./mmdeploy_models/mmaction/tsn/ort/end2end.onnx'] -image = 'tests/data/arm_wrestling.mp4' - -# read deploy_cfg and model_cfg -deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg) - -# build task and backend model -task_processor = build_task_processor(model_cfg, deploy_cfg, device) -model = task_processor.init_backend_model(backend_model) - -# process input image -input_shape = get_input_shape(deploy_cfg) -model_inputs, _ = task_processor.create_input(image, input_shape) - -# do model inference -with torch.no_grad(): - result = task_processor.run_inference(model, model_inputs) - -# show top5-results -result = np.array(result[0]) -top_index = np.argsort(result)[::-1] -for i in range(5): - index = top_index[i] - print(index, result[index]) -``` - -### SDK model inference - -Given the above SDK model of `tsn` you can also perform SDK model inference like following, - -#### Video recognition SDK model inference - -```python -from mmdeploy_python import VideoRecognizer -import cv2 - -# refer to demo/python/video_recognition.py -# def SampleFrames(cap, clip_len, frame_interval, num_clips): -# ... - -cap = cv2.VideoCapture('tests/data/arm_wrestling.mp4') - -clips, info = SampleFrames(cap, 1, 1, 25) - -# create a recognizer -recognizer = VideoRecognizer(model_path='./mmdeploy_models/mmaction/tsn/ort', device_name='cpu', device_id=0) -# perform inference -result = recognizer(clips, info) -# show inference result -for label_id, score in result: - print(label_id, score) -``` - -Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Interface), such as C, C++, C#, Java and so on. You can learn their usage from [demos](https://github.com/open-mmlab/mmdeploy/demo). - -> MMAction2 only API of c, c++ and python for now. - ## Supported models | Model | TorchScript | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | diff --git a/docs/en/index.rst b/docs/en/index.rst index f9ccd259b..fed409fa2 100644 --- a/docs/en/index.rst +++ b/docs/en/index.rst @@ -50,6 +50,7 @@ You can switch between Chinese and English documents in the lower-left corner of 04-supported-codebases/mmpose.md 04-supported-codebases/mmdet3d.md 04-supported-codebases/mmrotate.md + 04-supported-codebases/mmaction2.md .. toctree:: :maxdepth: 1 @@ -63,6 +64,7 @@ You can switch between Chinese and English documents in the lower-left corner of 05-supported-backends/tensorrt.md 05-supported-backends/torchscript.md 05-supported-backends/rknn.md + 05-supported-backends/coreml.md .. toctree:: :maxdepth: 1 diff --git a/docs/zh_cn/04-supported-codebases/mmaction2.md b/docs/zh_cn/04-supported-codebases/mmaction2.md index b490d6349..04e2d87c4 100644 --- a/docs/zh_cn/04-supported-codebases/mmaction2.md +++ b/docs/zh_cn/04-supported-codebases/mmaction2.md @@ -1,190 +1,11 @@ -# MMAction2 模型部署 - -- [MMAction2 模型部署](#mmaction2-模型部署) - - [安装](#安装) - - [安装 mmaction2](#安装-mmaction2) - - [安装 mmdeploy](#安装-mmdeploy) - - [模型转换](#模型转换) - - [视频分类任务模型转换](#视频分类任务模型转换) - - [模型规范](#模型规范) - - [模型推理](#模型推理) - - [后端模型推理](#后端模型推理) - - [SDK 模型推理](#sdk-模型推理) - - [视频分类 SDK 模型推理](#视频分类-sdk-模型推理) - - [模型支持列表](#模型支持列表) - -______________________________________________________________________ +# mmction2 模型支持列表 [MMAction2](https://github.com/open-mmlab/mmaction2)是一款基于 PyTorch 的视频理解开源工具箱，是[OpenMMLab](https://openmmlab.com)项目的成员之一。 -## 安装 - -### 安装 mmaction2 +## 安装 mmaction2 请参考[官网安装指南](https://github.com/open-mmlab/mmaction2#installation). -### 安装 mmdeploy - -mmdeploy 有以下几种安装方式: - -**方式一：** 安装预编译包 - -通过此[链接](https://github.com/open-mmlab/mmdeploy/releases)获取最新的预编译包 - -**方式二：** 一键式脚本安装 - -如果部署平台是 **Ubuntu 18.04 及以上版本**， 请参考[脚本安装说明](../01-how-to-build/build_from_script.md)，完成安装过程。 -比如，以下命令可以安装 mmdeploy 以及配套的推理引擎——`ONNX Runtime`. - -```shell -git clone --recursive https://github.com/open-mmlab/mmdeploy.git -cd mmdeploy -python3 tools/scripts/build_ubuntu_x64_ort.py $(nproc) -export PYTHONPATH=$(pwd)/build/lib:$PYTHONPATH -export LD_LIBRARY_PATH=$(pwd)/../mmdeploy-dep/onnxruntime-linux-x64-1.8.1/lib/:$LD_LIBRARY_PATH -``` - -**方式三：** 源码安装 - -在方式一、二都满足不了的情况下，请参考[源码安装说明](../01-how-to-build/build_from_source.md) 安装 mmdeploy 以及所需推理引擎。 - -## 模型转换 - -你可以使用 [tools/deploy.py](https://github.com/open-mmlab/mmdeploy/blob/master/tools/deploy.py) 把 mmaction2 模型一键式转换为推理后端模型。 -该工具的详细使用说明请参考[这里](https://github.com/open-mmlab/mmdeploy/blob/master/docs/en/02-how-to-run/convert_model.md#usage). - -转换的关键之一是使用正确的配置文件。项目中已内置了各后端部署[配置文件](https://github.com/open-mmlab/mmdeploy/tree/master/configs/mmaction)。 -文件的命名模式是： - -``` -{task}/{task}_{backend}-{precision}_{static | dynamic}_{shape}.py -``` - -其中： - -- **{task}:** mmaction2 中的任务 -- **{backend}:** 推理后端名称。比如，onnxruntime、tensorrt、pplnn、ncnn、openvino、coreml 等等 -- **{precision}:** 推理精度。比如，fp16、int8。不填表示 fp32 -- **{static | dynamic}:** 动态、静态 shape -- **{shape}:** 模型输入的 shape 或者 shape 范围 -- **{2d/3d}:** 表示模型的类别 - -以下，我们将演示如何把视频分类任务中 `tsn` 模型转换为 onnx 模型。 - -### 视频分类任务模型转换 - -```shell -cd mmdeploy - -# download tsn model from mmaction2 model zoo -mim download mmaction2 --config tsn_r50_1x1x3_100e_kinetics400_rgb --dest . - -# convert mmaction2 model to onnxruntime model with dynamic shape -python tools/deploy.py \ - configs/mmaction/video-recognition/video-recognition_onnxruntime_static.py \ - tsn_r50_1x1x3_100e_kinetics400_rgb.py \ - tsn_r50_256p_1x1x3_100e_kinetics400_rgb_20200725-22592236.pth \ - tests/data/arm_wrestling.mp4 \ - --work-dir mmdeploy_models/mmaction/tsn/ort \ - --device cpu \ - --show \ - --dump-info -``` - -## 模型规范 - -在使用转换后的模型进行推理之前，有必要了解转换结果的结构。 它存放在 `--work-dir` 指定的路路径下。 - -上例中的`mmdeploy_models/mmaction/tsn/ort`，结构如下： - -``` -mmdeploy_models/mmaction/tsn/ort -├── deploy.json -├── detail.json -├── end2end.onnx -└── pipeline.json -``` - -重要的是： - -- **end2end.onnx**: 推理引擎文件。可用 ONNX Runtime 推理 -- \***.json**: mmdeploy SDK 推理所需的 meta 信息 - -整个文件夹被定义为**mmdeploy SDK model**。换言之，**mmdeploy SDK model**既包括推理引擎，也包括推理 meta 信息。 - -## 模型推理 - -### 后端模型推理 - -以上述模型转换后的 `end2end.onnx` 为例，你可以使用如下代码进行推理： - -```python -from mmdeploy.apis.utils import build_task_processor -from mmdeploy.utils import get_input_shape, load_config -import numpy as np -import torch - -deploy_cfg = 'configs/mmaction/video-recognition/video-recognition_onnxruntime_static.py' -model_cfg = 'tsn_r50_1x1x3_100e_kinetics400_rgb.py' -device = 'cpu' -backend_model = ['./mmdeploy_models/mmaction/tsn/ort/end2end.onnx'] -image = 'tests/data/arm_wrestling.mp4' - -# read deploy_cfg and model_cfg -deploy_cfg, model_cfg = load_config(deploy_cfg, model_cfg) - -# build task and backend model -task_processor = build_task_processor(model_cfg, deploy_cfg, device) -model = task_processor.init_backend_model(backend_model) - -# process input image -input_shape = get_input_shape(deploy_cfg) -model_inputs, _ = task_processor.create_input(image, input_shape) - -# do model inference -with torch.no_grad(): - result = task_processor.run_inference(model, model_inputs) - -# show top5-results -result = np.array(result[0]) -top_index = np.argsort(result)[::-1] -for i in range(5): - index = top_index[i] - print(index, result[index]) -``` - -### SDK 模型推理 - -你也可以参考如下代码，对 SDK model 进行推理： - -#### 视频分类 SDK 模型推理 - -```python -from mmdeploy_python import VideoRecognizer -import cv2 - -# refer to demo/python/video_recognition.py -# def SampleFrames(cap, clip_len, frame_interval, num_clips): -# ... - -cap = cv2.VideoCapture('tests/data/arm_wrestling.mp4') - -clips, info = SampleFrames(cap, 1, 1, 25) - -# create a recognizer -recognizer = VideoRecognizer(model_path='./mmdeploy_models/mmaction/tsn/ort', device_name='cpu', device_id=0) -# perform inference -result = recognizer(clips, info) -# show inference result -for label_id, score in result: - print(label_id, score) -``` - -除了python API，mmdeploy SDK 还提供了诸如 C、C++、C#、Java等多语言接口。 -你可以参考[样例](https://github.com/open-mmlab/mmdeploy/tree/master/demo)学习其他语言接口的使用方法。 - -> mmaction2 的 C#，Java接口待开发 - ## 模型支持列表 | Model | TorchScript | ONNX Runtime | TensorRT | ncnn | PPLNN | OpenVINO | diff --git a/docs/zh_cn/index.rst b/docs/zh_cn/index.rst index 3fad0324b..8f0a61618 100644 --- a/docs/zh_cn/index.rst +++ b/docs/zh_cn/index.rst @@ -50,6 +50,7 @@ 04-supported-codebases/mmpose.md 04-supported-codebases/mmrotate.md 04-supported-codebases/mmseg.md + 04-supported-codebases/mmaction2.md .. toctree:: :maxdepth: 1 @@ -63,6 +64,7 @@ 05-supported-backends/snpe.md 05-supported-backends/tensorrt.md 05-supported-backends/torchscript.md + 05-supported-backends/coreml.md .. toctree:: :maxdepth: 1