mirrors
/
mmdeploy
mirror of https://github.com/open-mmlab/mmdeploy.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
mmdeploy/docs/zh_cn/04-developer-guide/do_regression_test.md

286 lines
14 KiB
Markdown
Raw Normal View History

[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
# 如何进行回归测试
<!-- -->
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
这篇教程介绍了如何进行回归测试。部署配置文件由`每个codebase的回归配置文件`，`推理框架配置信息`组成。
<!-- TOC -->
- [如何进行回归测试](#如何进行回归测试)
Fix test ncnn failure in regression test (#551) * fix ncnn test in regression test * update doc * fix docstring
2022-06-06 10:00:02 +08:00
- [1. 环境搭建](#1-环境搭建)
- [MMDeploy的安装及配置](#mmdeploy的安装及配置)
- [Python环境依赖](#python环境依赖)
- [2. 用法](#2-用法)
- [参数解析](#参数解析)
- [注意事项](#注意事项)
- [例子](#例子)
- [3. 回归测试配置文件](#3-回归测试配置文件)
- [示例及参数解析](#示例及参数解析)
- [4. 生成的报告](#4-生成的报告)
- [模板](#模板)
- [示例](#示例)
- [5. 支持的后端](#5-支持的后端)
- [6. 支持的Codebase及其Metric](#6-支持的codebase及其metric)
- [7. 注意事项](#7-注意事项)
- [8. 常见问题](#8-常见问题)
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
<!-- TOC -->
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
## 1. 环境搭建
### MMDeploy的安装及配置
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
本章节的内容，需要提前根据[build 文档](../01-how-to-build/build_from_source.md)将 MMDeploy 安装配置好之后，才能进行。
### Python环境依赖
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
需要安装 test 的环境
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
```shell
pip install -r requirements/tests.txt
```
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
如果在使用过程是 numpy 报错，则更新一下 numpy
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
```shell
pip install -U numpy
```
## 2. 用法
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
```shell
python ./tools/regression_test.py \
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
--codebase "${CODEBASE_NAME}" \
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
--backends "${BACKEND}" \
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
[--models "${MODELS}"] \
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
--work-dir "${WORK_DIR}" \
--device "${DEVICE}" \
--log-level INFO \
Fix test ncnn failure in regression test (#551) * fix ncnn test in regression test * update doc * fix docstring
2022-06-06 10:00:02 +08:00
[--performance 或 -p] \
[--checkpoint-dir "$CHECKPOINT_DIR"]
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
```
### 参数解析
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
- `--codebase` : 需要测试的 codebase，eg.`mmdet`, 测试多个 `mmcls mmdet ...`
Fix test ncnn failure in regression test (#551) * fix ncnn test in regression test * update doc * fix docstring
2022-06-06 10:00:02 +08:00
- `--backends` : 筛选测试的后端, 默认测全部`backend`, 也可传入若干个后端，例如 `onnxruntime tesnsorrt`。如果需要一同进行 SDK 的测试，需要在 `tests/regression/${codebase}.yml` 里面的 `sdk_config` 进行配置。
- `--models` : 指定测试的模型, 默认测试 `yml` 中所有模型, 也可传入若干个模型名称，模型名称可参考相关yml配置文件。例如 `ResNet SE-ResNet "Mask R-CNN"`。注意的是，可传入只有字母和数字组成模型名称，例如 `resnet seresnet maskrcnn`。
- `--work-dir` : 模型转换、报告生成的路径，默认是`../mmdeploy_regression_working_dir`，注意路径中不要不含空格等特殊字符。
- `--checkpoint-dir`: PyTorch 模型文件下载保存路径，默认是`../mmdeploy_checkpoints`，注意路径中不要不含空格等特殊字符。
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
- `--device` : 使用的设备，默认 `cuda`。
- `--log-level` : 设置日记的等级，选项包括`'CRITICAL'， 'FATAL'， 'ERROR'， 'WARN'， 'WARNING'， 'INFO'， 'DEBUG'， 'NOTSET'`。默认是`INFO`。
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
- `-p` 或 `--performance` : 是否测试精度，加上则测试转换+精度，不加上则只测试转换
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
### 注意事项
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
对于 Windows 用户：
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
1. 要在 shell 命令中使用 `&&` 连接符，需要下载并使用 `PowerShell 7 Preview 5+`。
2. 如果您使用 conda env，可能需要在 regression_test.py 中将 `python3` 更改为 `python`，因为 `%USERPROFILE%\AppData\Local\Microsoft\WindowsApps` 目录中有 `python3.exe`。
## 例子
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
1. 测试 mmdet 和 mmpose 的所有 backend 的 **转换+精度**
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
```shell
python ./tools/regression_test.py \
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
--codebase mmdet mmpose \
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
--work-dir "../mmdeploy_regression_working_dir" \
--device "cuda" \
--log-level INFO \
--performance
```
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
2. 测试 mmdet 和 mmpose 的某几个 backend 的 **转换+精度**
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
```shell
python ./tools/regression_test.py \
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
--codebase mmdet mmpose \
docs(do_regression_test): fix-typo (#547) * docs(do_regression_test): fix-typo * improvement(apis/__init__.py): add print * fix(CI): lint * revert(apis/__init__.py): remove logging
2022-06-02 17:07:53 +08:00
--backends onnxruntime tensorrt \
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
--work-dir "../mmdeploy_regression_working_dir" \
--device "cuda" \
--log-level INFO \
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
-p
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
```
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
3. 测试 mmdet 和 mmpose 的某几个 backend，**只测试转换**
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
```shell
python ./tools/regression_test.py \
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
--codebase mmdet mmpose \
docs(do_regression_test): fix-typo (#547) * docs(do_regression_test): fix-typo * improvement(apis/__init__.py): add print * fix(CI): lint * revert(apis/__init__.py): remove logging
2022-06-02 17:07:53 +08:00
--backends onnxruntime tensorrt \
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
--work-dir "../mmdeploy_regression_working_dir" \
--device "cuda" \
--log-level INFO
```
docs(do_regression_test): fix-typo (#547) * docs(do_regression_test): fix-typo * improvement(apis/__init__.py): add print * fix(CI): lint * revert(apis/__init__.py): remove logging
2022-06-02 17:07:53 +08:00
4. 测试 mmdet 和 mmcls 的某几个 models，**只测试转换**
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
```shell
python ./tools/regression_test.py \
--codebase mmdet mmpose \
--models ResNet SE-ResNet "Mask R-CNN" \
--work-dir "../mmdeploy_regression_working_dir" \
--device "cuda" \
--log-level INFO
```
## 3. 回归测试配置文件
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
### 示例及参数解析
```yaml
globals:
codebase_dir: ../mmocr # 回归测试的 codebase 路径
checkpoint_force_download: False # 回归测试是否重新下载模型即使其已经存在
images: # 测试使用图片
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
img_densetext_det: &img_densetext_det ../mmocr/demo/demo_densetext_det.jpg
img_demo_text_det: &img_demo_text_det ../mmocr/demo/demo_text_det.jpg
img_demo_text_ocr: &img_demo_text_ocr ../mmocr/demo/demo_text_ocr.jpg
img_demo_text_recog: &img_demo_text_recog ../mmocr/demo/demo_text_recog.jpg
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
metric_info: &metric_info # 指标参数
hmean-iou: # 命名根据 metafile.Results.Metrics
eval_name: hmean-iou # 命名根据 test.py --metrics args 入参名称
metric_key: 0_hmean-iou:hmean # 命名根据 eval 写入 log 的 key name
tolerance: 0.1 # 容忍的阈值区间
task_name: Text Detection # 命名根据模型 metafile.Results.Task
dataset: ICDAR2015 #命名根据模型 metafile.Results.Dataset
word_acc: # 同上
eval_name: acc
metric_key: 0_word_acc_ignore_case
tolerance: 0.2
task_name: Text Recognition
dataset: IIIT5K
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
convert_image_det: &convert_image_det # det转换会使用到的图片
input_img: *img_densetext_det
test_img: *img_demo_text_det
convert_image_rec: &convert_image_rec
input_img: *img_demo_text_recog
test_img: *img_demo_text_recog
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
backend_test: &default_backend_test True # 是否对 backend 进行精度测试
sdk: # SDK 配置文件
sdk_detection_dynamic: &sdk_detection_dynamic configs/mmocr/text-detection/text-detection_sdk_dynamic.py
sdk_recognition_dynamic: &sdk_recognition_dynamic configs/mmocr/text-recognition/text-recognition_sdk_dynamic.py
onnxruntime:
pipeline_ort_recognition_static_fp32: &pipeline_ort_recognition_static_fp32
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
convert_image: *convert_image_rec # 转换过程中使用的图片
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
backend_test: *default_backend_test # 是否进行后端测试，存在则判断，不存在则视为 False
sdk_config: *sdk_recognition_dynamic # 是否进行SDK测试，存在则使用特定的 SDK config 进行测试，不存在则视为不进行 SDK 测试
deploy_config: configs/mmocr/text-recognition/text-recognition_onnxruntime_static.py # 使用的 deploy cfg 路径，基于 mmdeploy 的路径
pipeline_ort_recognition_dynamic_fp32: &pipeline_ort_recognition_dynamic_fp32
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
convert_image: *convert_image_rec
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
backend_test: *default_backend_test
sdk_config: *sdk_recognition_dynamic
deploy_config: configs/mmocr/text-recognition/text-recognition_onnxruntime_dynamic.py
pipeline_ort_detection_dynamic_fp32: &pipeline_ort_detection_dynamic_fp32
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
convert_image: *convert_image_det
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
deploy_config: configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py
tensorrt:
pipeline_trt_recognition_dynamic_fp16: &pipeline_trt_recognition_dynamic_fp16
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
convert_image: *convert_image_rec
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
backend_test: *default_backend_test
sdk_config: *sdk_recognition_dynamic
deploy_config: configs/mmocr/text-recognition/text-recognition_tensorrt-fp16_dynamic-1x32x32-1x32x640.py
pipeline_trt_detection_dynamic_fp16: &pipeline_trt_detection_dynamic_fp16
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
convert_image: *convert_image_det
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
backend_test: *default_backend_test
sdk_config: *sdk_detection_dynamic
change resolution and fix regression test for mmocr (#534)
2022-06-02 18:29:41 +08:00
deploy_config: configs/mmocr/text-detection/text-detection_tensorrt-fp16_dynamic-320x320-2240x2240.py
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
openvino:
# 此处省略，内容同上
ncnn:
# 此处省略，内容同上
pplnn:
# 此处省略，内容同上
torchscript:
# 此处省略，内容同上
models:
- name: crnn # 模型名称
metafile: configs/textrecog/crnn/metafile.yml # 模型对应的 metafile 的路径，相对于 codebase 的路径
codebase_model_config_dir: configs/textrecog/crnn # `model_configs` 的父文件夹路径，相对于 codebase 的路径
model_configs: # 需要测试的 config 名称
- crnn_academic_dataset.py
pipelines: # 使用的 pipeline
- *pipeline_ort_recognition_dynamic_fp32
- name: dbnet
metafile: configs/textdet/dbnet/metafile.yml
codebase_model_config_dir: configs/textdet/dbnet
model_configs:
- dbnet_r18_fpnc_1200e_icdar2015.py
pipelines:
- *pipeline_ort_detection_dynamic_fp32
- *pipeline_trt_detection_dynamic_fp16
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
# 特殊的 pipeline 可以这样加入
- convert_image: xxx
backend_test: xxx
sdk_config: xxx
deploy_config: configs/mmocr/text-detection/xxx
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
```
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
## 4. 生成的报告
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
### 模板
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
| | Model | Model Config | Task | Checkpoint | Dataset | Backend | Deploy Config | Static or Dynamic | Precision Type | Conversion Result | metric_1 | metric_2 | metric_n | Test Pass |
| ---- | -------- | ----------------- | ---------------- | -------------- | ---------- | -------- | --------------- | ----------------- | -------------- | ----------------- | ----------- | ----------- | ----------- | ------------ |
| 序号 | 模型名称 | model config 路径 | 执行的 task name | `.pth`模型路径 | 数据集名称 | 后端名称 | deploy cfg 路径 | 动态 or 静态 | 测试精度 | 模型转换结果 | 指标 1 数值 | 指标 2 数值 | 指标 n 数值 | 后端测试结果 |
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
### 示例
这是 MMOCR 生成的报告
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
| | Model | Model Config | Task | Checkpoint | Dataset | Backend | Deploy Config | Static or Dynamic | Precision Type | Conversion Result | hmean-iou | word_acc | Test Pass |
| --- | ----- | ---------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------ | --------- | --------------- | -------------------------------------------------------------------------------------- | ----------------- | -------------- | ----------------- | --------- | -------- | --------- |
| 0 | crnn | ../mmocr/configs/textrecog/crnn/crnn_academic_dataset.py | Text Recognition | ../mmdeploy_checkpoints/mmocr/crnn/crnn_academic-a723a1c5.pth | IIIT5K | Pytorch | - | - | - | - | - | 80.5 | - |
| 1 | crnn | ../mmocr/configs/textrecog/crnn/crnn_academic_dataset.py | Text Recognition | ${WORK_DIR}/mmocr/crnn/onnxruntime/static/crnn_academic-a723a1c5/end2end.onnx | x | onnxruntime | configs/mmocr/text-recognition/text-recognition_onnxruntime_dynamic.py | static | fp32 | True | - | 80.67 | True |
| 2 | crnn | ../mmocr/configs/textrecog/crnn/crnn_academic_dataset.py | Text Recognition | ${WORK_DIR}/mmocr/crnn/onnxruntime/static/crnn_academic-a723a1c5 | x | SDK-onnxruntime | configs/mmocr/text-recognition/text-recognition_sdk_dynamic.py | static | fp32 | True | - | x | False |
| 3 | dbnet | ../mmocr/configs/textdet/dbnet/dbnet_r18_fpnc_1200e_icdar2015.py | Text Detection | ../mmdeploy_checkpoints/mmocr/dbnet/dbnet_r18_fpnc_sbn_1200e_icdar2015_20210329-ba3ab597.pth | ICDAR2015 | Pytorch | - | - | - | - | 0.795 | - | - |
| 4 | dbnet | ../mmocr/configs/textdet/dbnet/dbnet_r18_fpnc_1200e_icdar2015.py | Text Detection | ../mmdeploy_checkpoints/mmocr/dbnet/dbnet_r18_fpnc_sbn_1200e_icdar2015_20210329-ba3ab597.pth | ICDAR | onnxruntime | configs/mmocr/text-detection/text-detection_onnxruntime_dynamic.py | dynamic | fp32 | True | - | - | True |
| 5 | dbnet | ../mmocr/configs/textdet/dbnet/dbnet_r18_fpnc_1200e_icdar2015.py | Text Detection | ${WORK_DIR}/mmocr/dbnet/tensorrt/dynamic/dbnet_r18_fpnc_sbn_1200e_icdar2015_20210329-ba3ab597/end2end.engine | ICDAR | tensorrt | configs/mmocr/text-detection/text-detection_tensorrt-fp16_dynamic-320x320-2240x2240.py | dynamic | fp16 | True | 0.793302 | - | True |
| 6 | dbnet | ../mmocr/configs/textdet/dbnet/dbnet_r18_fpnc_1200e_icdar2015.py | Text Detection | ${WORK_DIR}/mmocr/dbnet/tensorrt/dynamic/dbnet_r18_fpnc_sbn_1200e_icdar2015_20210329-ba3ab597 | ICDAR | SDK-tensorrt | configs/mmocr/text-detection/text-detection_sdk_dynamic.py | dynamic | fp16 | True | 0.795073 | - | True |
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
## 5. 支持的后端
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
- [x] ONNX Runtime
- [x] TensorRT
- [x] PPLNN
- [x] ncnn
- [x] OpenVINO
- [x] TorchScript
Sync v0.7.0 to dev-1.x (#907) * make -install -> make install (#621) change `make -install` to `make install` https://github.com/open-mmlab/mmdeploy/issues/618 * [Fix] fix csharp api detector release result (#620) * fix csharp api detector release result * fix wrong count arg of xxx_release_result in c# api * [Enhancement] Support two-stage rotated detector TensorRT. (#530) * upload * add fake_multiclass_nms_rotated * delete unused code * align with pytorch * Update delta_midpointoffset_rbbox_coder.py * add trt rotated roi align * add index feature in nms * not good * fix index * add ut * add benchmark * move to csrc/mmdeploy * update unit test Co-authored-by: zytx121 <592267829@qq.com> * Reduce mmcls version dependency (#635) * fix shufflenetv2 with trt (#645) * fix shufflenetv2 and pspnet * fix ci * remove print * ' -> " (#654) If there is a variable in the string, single quotes will ignored it, while double quotes will bring the variable into the string after parsing * ' -> " (#655) same with https://github.com/open-mmlab/mmdeploy/pull/654 * Support deployment of Segmenter (#587) * support segmentor with ncnn * update regression yml * replace chunk with split to support ts * update regression yml * update docs * fix segmenter ncnn inference failure brought by #477 * add test * fix test for ncnn and trt * fix lint * export nn.linear to Gemm op in onnx for ncnn * fix ci * simplify `Expand` (#617) * Fix typo (#625) * Add make install in en docs * Add make install in zh docs * Fix typo * Merge and add windows build Co-authored-by: tripleMu <865626@163.com> * [Enhancement] Fix ncnn unittest (#626) * optmize-csp-darknet * replace floordiv to torch.div * update csp_darknet default implement * fix test * [Enhancement] TensorRT Anchor generator plugin (#646) * custom trt anchor generator * add ut * add docstring, update doc * Add partition doc and sample code (#599) * update torch2onnx tool to support onnx partition * add model partition of yolov3 * add cn doc * update torch2onnx tool to support onnx partition * add model partition of yolov3 * add cn doc * add to index.rst * resolve comment * resolve comments * fix lint * change caption level in docs * update docs (#624) * Add java apis and demos (#563) * add java classifier detector * add segmentor * fix lint * add ImageRestorer java apis and demo * remove useless count parameter for Segmentor and Restorer, add PoseDetector * add RotatedDetection java api and demo * add Ocr java demo and apis * remove mmrotate ncnn java api and demo * fix lint * sync java api folder after rebase to master * fix include * remove record * fix java apis dir path in cmake * add java demo readme * fix lint mdformat * add test javaapi ci * fix lint * fix flake8 * fix test javaapi ci * refactor readme.md * fix install opencv for ci * fix install opencv : add permission * add all codebases and mmcv install * add torch * install mmdeploy * fix image path * fix picture path * fix import ncnn * fix import ncnn * add submodule of pybind * fix pybind submodule * change download to git clone for submodule * fix ncnn dir * fix README error * simplify the github ci * fix ci * fix yapf * add JNI as required * fix Capitalize * fix Capitalize * fix copyright * ignore .class changed * add OpenJDK installation docs * install target of javaapi * simplify ci * add jar * fix ci * fix ci * fix test java command * debugging what failed * debugging what failed * debugging what failed * add java version info * install openjdk * add java env var * fix export * fix export * fix export * fix export * fix picture path * fix picture path * fix file name * fix file name * fix README * remove java_api strategy * fix python version * format task name * move args position * extract common utils code * show image class result * add detector result * segmentation result format * add ImageRestorer result * add PoseDetection java result format * fix ci * stage ocr * add visualize * move utils * fix lint * fix ocr bugs * fix ci demo * fix java classpath for ci * fix popd * fix ocr demo text garbled * fix ci * fix ci * fix ci * fix path of utils ci * update the circleci config file by adding workflows both for linux, windows and linux-gpu (#368) * update circleci by adding more workflows * fix test workflow failure on windows platform * fix docker exec command for SDK unittests * Fixed tensorrt plugin not found in Windows (#672) * update introduction.png (#674) * [Enhancement] Add fuse select assign pass (#589) * Add fuse select assign pass * move code to csrc * add config flag * remove bool cast * fix export sdk info of input shape (#667) * Update get_started.md (#675) Fix backend model assignment * Update get_started.md (#676) Fix backend model assignment * [Fix] fix clang build (#677) * fix clang build * fix ndk build * fix ndk build * switch to `std::filesystem` for clang-7 and later * Deploy the Swin Transformer on TensorRT. (#652) * resolve conflicts * update ut and docs * fix ut * refine docstring * add comments and refine UT * resolve comments * resolve comments * update doc * add roll export * check backend * update regression test * bump version to 0.6.0 (#680) * bump vertion to 0.6.0 * update version * pass img_metas while exporting to onnx (#681) * pass img_metas while exporting to onnx * remove try-catch in tools for beter debugging * use get * fix typo * [Fix] fix ssd ncnn ut (#692) * fix ssd ncnn ut * fix yapf * fix passing img_metas to pytorch2onnx for mmedit (#700) * fix passing img_metas for mmdet3d (#707) * [Fix] Fix android build (#698) * fix android build * fix cmake * fix url link * fix wrong exit code in pipeline_manager (#715) * fix exit * change to general exit errorcode=1 * fix passing wrong backend type (#719) * Rename onnx2ncnn to mmdeploy_onnx2ncnn (#694) * improvement(tools/onnx2ncnn.py): rename to mmdeploy_onnx2ncnn * format(tools/deploy.py): clean code * fix(init_plugins.py): improve if condition * fix(CI): update target * fix(test_onnx2ncnn.py): update desc * Update init_plugins.py * [Fix] Fix mmdet ort static shape bug (#687) * fix shape * add device * fix yapf * fix rewriter for transforms * reverse image shape * fix ut of distance2bbox * fix rewriter name * fix c4 for torchscript (#724) * [Enhancement] Standardize C API (#634) * unify C API naming * fix demo and move apis/c/* -> apis/c/mmdeploy/* * fix lint * fix C# project * fix Java API * [Enhancement] Support Slide Vertex TRT (#650) * reorgnize mmrotate * fix * add hbb2obb * add ut * fix rotated nms * update docs * update benchmark * update test * remove ort regression test, remove comment * Fix get-started rendering issues in readthedocs (#740) * fix mermaid markdown rendering issue in readthedocs * fix error in C++ example * fix error in c++ example in zh_cn get_started doc * [Fix] set default topk for dump info (#702) * set default topk for dump info * remove redundant docstrings * add ci densenet * fix classification warnings * fix mmcls version * fix logger.warnings * add version control (#754) * fix satrn for ORT (#753) * fix satrn for ORT * move rewrite into pytorch * Add inference latency test tool (#665) * add profile tool * remove print envs in profile tool * set cudnn_benchmark to True * add doc * update tests * fix typo * support test with images from a directory * update doc * resolve comments * [Enhancement] Add CSE ONNX pass (#647) * Add fuse select assign pass * move code to csrc * add config flag * Add fuse select assign pass * Add CSE for ONNX * remove useless code * Test robot Just test robot * Update README.md Revert * [Fix] fix yolox point_generator (#758) * fix yolox point_generator * add a UT * resolve comments * fix comment lines * limit markdown version (#773) * [Enhancement] Better index put ONNX export. (#704) * Add rewriter for tensor setitem * add version check * Upgrade Dockerfile to use TensorRT==8.2.4.2 (#706) * Upgrade TensorRT to 8.2.4.2 * upgrade pytorch&mmcv in CPU Dockerfile * Delete redundant port example in Docker * change 160x160-608x608 to 64x64-608x608 for yolov3 * [Fix] reduce log verbosity & improve error reporting (#755) * reduce log verbosity & improve error reporting * improve error reporting * [Enhancement] Support latest ppl.nn & ppl.cv (#564) * support latest ppl.nn * fix pplnn for model convertor * fix lint * update memory policy * import algo from buffer * update ppl.cv * use `ppl.cv==0.7.0` * document supported ppl.nn version * skip pplnn dependency when building shared libs * [Fix][P0] Fix for torch1.12 (#751) * fix for torch1.12 * add comment * fix check env (#785) * [Fix] fix cascade mask rcnn (#787) * fix cascade mask rcnn * fix lint * add regression * [Feature] Support RoITransRoIHead (#713) * [Feature] Support RoITransRoIHead * Add docs * Add mmrotate models regression test * Add a draft for test code * change the argument name * fix test code * fix minor change for not class agnostic case * fix sample for test code * fix sample for test code * Add mmrotate in requirements * Revert "Add mmrotate in requirements" This reverts commit 043490075e6dbe4a8fb98e94b2b583b91fc5038d. * [Fix] fix triu (#792) * fix triu * triu -> triu_default * [Enhancement] Install Optimizer by setuptools (#690) * Add fuse select assign pass * move code to csrc * add config flag * Add fuse select assign pass * Add CSE for ONNX * remove useless code * Install optimizer by setup tools * fix comment * [Feature] support MMRotate model with le135 (#788) * support MMRotate model with le135 * cse before fuse select assign * remove unused import * [Fix] Support macOS build (#762) * fix macOS build * fix missing * add option to build & install examples (#822) * [Fix] Fix setup on non-linux-x64 (#811) * fix setup * replace long to int64_t * [Feature] support build single sdk library (#806) * build single lib for c api * update csharp doc & project * update test build * fix test build * fix * update document for building android sdk (#817) Co-authored-by: dwSun <dwsunny@icloud.com> * [Enhancement] support kwargs in SDK python bindings (#794) * support-kwargs * make '__call__' as single image inference and add 'batch' API to deal with batch images inference * fix linting error and typo * fix lint * improvement(sdk): add sdk code coverage (#808) * feat(doc): add CI * CI(sdk): add sdk coverage * style(test): code format * fix(CI): update coverage.info path * improvement(CI): use internal image * improvement(CI): push coverage info once * [Feature] Add C++ API for SDK (#831) * add C++ API * unify result type & add examples * minor fix * install cxx API headers * fix Mat, add more examples * fix monolithic build & fix lint * install examples correctly * fix lint * feat(tools/deploy.py): support snpe (#789) * fix(tools/deploy.py): support snpe * improvement(backend/snpe): review advices * docs(backend/snpe): update build * docs(backend/snpe): server support specify port * docs(backend/snpe): update path * fix(backend/snpe): time counter missing argument * docs(backend/snpe): add missing argument * docs(backend/snpe): update download and using * improvement(snpe_net.cpp): load model with modeldata * Support setup on environment with no PyTorch (#843) * support test with multi batch (#829) * support test with multi batch * resolve comment * import algorithm from buffer (#793) * [Enhancement] build sdk python api in standard-alone manner (#810) * build sdk python api in standard-alone manner * enable MMDEPLOY_BUILD_SDK_MONOLITHIC and MMDEPLOY_BUILD_EXAMPLES in prebuild config * link mmdeploy to python target when monolithic option is on * checkin README to describe precompiled package build procedure * use packaging.version.parse(python_version) instead of list(python_version) * fix according to review results * rebase master * rollback cmake.in and apis/python/CMakeLists.txt * reorganize files in install/example * let cmake detect visual studio instead of specifying 2019 * rename whl name of precompiled package * fix according to review results * Fix SDK backend (#844) * fix mmpose python api (#852) * add prebuild package usage docs on windows (#816) * add prebuild package usage docs on windows * fix lint * update * try fix lint * add en docs * update * update * udpate faq * fix typo (#862) * [Enhancement] Improve get_started documents and bump version to 0.7.0 (#813) * simplify commands in get_started * add installation commands for Windows * fix typo * limit markdown and sphinx_markdown_tables version * adopt html <details open> tag * bump mmdeploy version * bump mmdeploy version * update get_started * update get_started * use python3.8 instead of python3.7 * remove duplicate section * resolve issue #856 * update according to review results * add reference to prebuilt_package_windows.md * fix error when build sdk demos * fix mmcls Co-authored-by: Ryan_Huang <44900829+DrRyanHuang@users.noreply.github.com> Co-authored-by: Chen Xin <xinchen.tju@gmail.com> Co-authored-by: q.yao <yaoqian@sensetime.com> Co-authored-by: zytx121 <592267829@qq.com> Co-authored-by: Li Zhang <lzhang329@gmail.com> Co-authored-by: tripleMu <gpu@163.com> Co-authored-by: tripleMu <865626@163.com> Co-authored-by: hanrui1sensetime <83800577+hanrui1sensetime@users.noreply.github.com> Co-authored-by: lvhan028 <lvhan_028@163.com> Co-authored-by: Bryan Glen Suello <11388006+bgsuello@users.noreply.github.com> Co-authored-by: zambranohally <63218980+zambranohally@users.noreply.github.com> Co-authored-by: AllentDan <41138331+AllentDan@users.noreply.github.com> Co-authored-by: tpoisonooo <khj.application@aliyun.com> Co-authored-by: Hakjin Lee <nijkah@gmail.com> Co-authored-by: 孙德伟 <5899962+dwSun@users.noreply.github.com> Co-authored-by: dwSun <dwsunny@icloud.com> Co-authored-by: Chen Xin <irexyc@gmail.com>
2022-08-19 09:30:13 +08:00
- [x] SNPE
[Feature] Regression test for mmdeploy (#302) * Add regression test script * Add doc * Add test yaml for mmdet * evaluate_outputs Add return result (cherry picked from commit c8c9cd75df7916aa4d80a7b8bfb8a78e04446cad) * object_detection return metric after eval (cherry picked from commit 1b8dcaa39ed14f016bf51d1aee65c2c416cd7c33) * move `deploy_config_dir` to `global_info` in test yaml * fix path error * Improve test yaml structure * Add test env for saving regression report * Fix SDK test report will crash * Get SDK FPS * Add mmcls regression test yaml * Using CMD to test the backend result * Get metric from log file * Imporve coding * Imporve coding * restructure test yaml * resturcture the test yaml and coding, using pipeline style * Fixed wont saving into report when cant find `backend_test` and `sdk_config` * set `metric_info` in test yaml * improve test yaml * Fixed will get black checkpoint file name * Fix lint * Fix yaml * Add common in test yaml * Resturcture mmcla test yaml * Resturcture mmcla test yaml * Improve mmcls test yaml * mmcls test success * Improve tes yaml field * Add `--test-img` only when `test_img_path` is not None * Add `precision_type` in report * Not saving pkl result file any more * Add 'x' install of '-' when script crash * Fix some field in mmcls test yaml * Add mmseg test yaml * Add unknown backend final file name * Improve backend file dict * Add mmseg success * unify the checkpoint path to relate path * unify the checkpoint path to relate path * Add mmpose, need to test * Support backend file list to `--model` when test the backend * Fix lint * Add some common * FPS get from log always get 1:10 line * Add dataset in report when test backend * Get dataset type from model config file * Replace pipeline.json topk * SDK report add backend name * Add txt report, it will save each test * update mmcls config * Add `calib-dataset-cfg` in cmd when it exist in tset yaml * make model path shorter by cutting the work_dir_root * Add `task_name` in test yaml * Add `task_name` in report * Improve test yaml * Add mmocr test yaml * Get mmocr fps metric success * Add `dataset` feild in test yaml * Report will skip when the dataset name not in test yaml * Add dbnet in mmocr test yaml and success get metric * Add mmedit test yaml * Improve some common * Add mmedit success * Fix lint * Fix isort lint * Fix yapf lint * Undo some changes in `evaluate_outputs` * Undo some changes in `evaluate_outputs` * Improve test requirement.txt * Undo some changes in `evaluate_outputs` * Improve doc * Improve mmedit test yaml * Using `--divice` * Fix lint * Using `--performance` replace `--test-type` * Fix lint * Fix page link * Fix backend name * Using `logger` instead of `print` * Fix lint * Add TorchScript in the doc * Add type hint for all the funcs * Fix docformatter lint * Fix path in report have the root of work dir * mmdet add other backend in tast yaml * mmdet add other backend in tast yaml * mmocr add other backend in tast yaml * mmedit add other backend in tast yaml * mmpose add other backend in tast yaml * Delete filed `codebase_model_config_dir` in test yaml * Using `Config` in metafile.yml instal of `Name` from allmodel config files. * Fix yapf lint * update mmpose mmseg config * Fix lint * Imporve mmcls test yaml * Imporve mmedit test yaml * Imporve mmedit test yaml * Imporve mmseg test yaml * update mmdet yml * Not using pth when conver sucess when in only convert mode * Using metafile dataset when can not get `model_cfg.dataset_type` * Fixed `model_name` incorrect in some codebase * Improve mmcls test yaml image * Improve mmedit test yaml image * Improve mmocr test yaml image * Improve mmseg test yaml image * Fix test yaml bug * Support overwirte `metric_tolerance` * Add `metric_tolerance` in mmcls * Fixed yaml bug * mmcls add all models, which had already supported, in test yaml * Fix report will not replace to ${WORK_DIR} * Add metric tolerance in mmcls test yaml * Modefied mmcls global metric tolreance * remove `metric_tolerance` in each pipeline * Improve mmcls test yaml * mmcls add TODO * imporve test yaml `pp`l -> `pplnn` * mmdet add all models, which had already supported, in test yaml * mmedit add all models, which had already supported, in test yaml * mmocr add all models, which had already supported, in test yaml * mmpose add all models, which had already supported, in test yaml * Replace `sdk_xxx_fp32` to `sdk_xxx` * mmseg add all models, which had already supported, in test yaml * Import the backend order of test yaml * Fix mmseg test yaml pplnn static config * Add notes for Windows user in the doc * Fix get metric from different metric name but correct dataset name. * Fix mmedit dataset incorrect * Fix test yaml of mmedit * Fix lint * Fix mmpose can't find metric * Improve mmseg trt testing config * Add dataset in mmdet test yaml * Add logs when continue in the code * Fix mmpose get metric log error * Fix mmdet can not get metric * Add merge report into one * Fix lint * Add version in the report * log2file for each backend * Fix lint * Improve report merge * update mmseg yml to shape 1024x2048 * Fix dependences for merging report * Imporve mmcls yaml, add `ShuffleNetV1` and `ShuffleNetV2` * Fix name of model in test yaml with a space will crash when convert * Add commons for test yaml metric tolerance * Add mmdet seg detail config in test yaml * Improve mmdet test yaml * Fix mmdet mskrcnn metric Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-04-28 08:03:59 +08:00
- [x] MMDeploy SDK
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
## 6. 支持的Codebase及其Metric
[Docs] Replace markdownlint with mdformat and configure myst-parser (#610) * replace mdlint with mdformat * set myst_heading_anchor for doc * update precommit yml * format md files
2022-06-17 09:19:10 +08:00
| Codebase | Metric | Support |
| -------- | -------- | ------------------ |
| mmdet | bbox | :heavy_check_mark: |
| | segm | :heavy_check_mark: |
| | PQ | :x: |
| mmcls | accuracy | :heavy_check_mark: |
| mmseg | mIoU | :heavy_check_mark: |
| mmpose | AR | :heavy_check_mark: |
| | AP | :heavy_check_mark: |
| mmocr | hmean | :heavy_check_mark: |
| | acc | :heavy_check_mark: |
| mmedit | PSNR | :heavy_check_mark: |
| | SSIM | :heavy_check_mark: |
[Enhancement] Improve MMDeploy Regression test (#425) * make regression test as a module under project * using `--codebase` instead of `--deploy-yml` * Improve doc for `--codebase` * Add shorter arg `-p` for `--performance` * make `checopoint-dir` as an arg for the script * Gen error log when convert fail. * Improve res code for testing * Doc add sdk test detail * Doc add env setup * Fix lint * Fix doc lint * Improve model path in report * Improve report title * Improve report checkpoint path * Fix lint * move test yaml under `tests/regression` * Improve the test yaml path * Fix lint * Improve doc * make func `update_report` code better * move doc to new location * Fix arg * Update arg details * Use cpu when openvino and onnxruntime cpu package * Fix word * Fix func of openpyxl 3.0.9 * Add some info * Fix lint * Fix filename * Fix doc link * Fix dir name with space when is not sdk * Add args `--models` for test specific model(s) * not saving report when no model in codebase when using `--models` * Fix doc * Fix lint * Add table for metric in doc * Improve table for doc * Using `None` install of `['all']` * Improce doc * set device type properly * Increate popen bufsize * Add `precision_type` in `work-dir` * Fix popen stuck * Fix lint * Fix lint * Fix popen stuck by using file handler * Make metric dataset as a list * Update mmseg.yml * Remove 'FPS' in the report * Update do_regression_test.md * Improve log * Fix codespell * Fix doc * ncnn only save `xxx.param` as checkpoint name in the report Co-authored-by: maningsheng <mnsheng@yeah.net>
2022-05-27 17:08:32 +08:00
## 7. 注意事项
暂无
## 8. 常见问题
暂无