262 Commits

Author SHA1 Message Date
Mashiro
ad1b43faf2
[Fix] Fix Config cannot parse base config when there is . in tmp path (#856)
* [Fix] Fix config cannot parse tmp path like

* Add comments

* Add comments

* Apply suggestions from code review

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-12-30 14:56:14 +08:00
Zaida Zhou
646927f62f
[Enhance] Ensure metrics is not empty when saving best ckpts (#849)
* [Enhance] Ensure metrics is not empty when saving best ckpts

* fix warn to warning

* delete a unnecessary method
2022-12-28 11:34:08 +08:00
Mashiro
a9b6753fbe
Make TTAModel compatible with FSDP (#611)
* Add build_runner_with_tta and PrepareTTAHook

* rename hook file

* support build tta runner with runner type

* add unit test

* Add build_runner_with_tta to index.rst

* minor refine

* Add runner test cast

* Fix unit test

* fix unit test

* tmp save

* pop None if key does not exist

* Fix is_model_wrapper and force register class in test_runner

* [Fix] Fix is_model_wrapper

* destroy group after ut

* register module in testcase

* pass through unit test

* fix as comment

* remove breakpoint

* remove mmengine/testing/runner_test_cast.py

* minor refine

* minor refine

* minor refine

* set default data preprocessor for model

* minor refine

* minor refine

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* fix lint

* Fix unit test

* replace  with  in ImgDataPreprocessor

* Fix as comment

* add inference tutorial in advanced tutorial

* update index.rst

* add tta example

* refine tta tutorial

* Add english tutorial

* add note for build_runner_with_tta

* Fix as comment

* add examples

* remove chinese comment

* Update docs/en/advanced_tutorials/test_time_augmentation.md

Co-authored-by: RangiLyu <lyuchqi@gmail.com>

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
Co-authored-by: RangiLyu <lyuchqi@gmail.com>
2022-12-27 21:39:43 +08:00
BayMax_BHL
16589ce386
[Feature] Add ProfilerHook (#768)
* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* [Feature] Add profiler hook functionality

* Apply suggestions from code review

* Update mmengine/hooks/profiler_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-12-27 18:58:05 +08:00
Mashiro
c382f8a5d5
[Fix] Ignore the distributed tests for macOS platform (#821)
* only test timer in linux

* Fix merge_stage_test.yml

* remove install ffmpeg

* test_ci

* test_ci

* Fix python -m pip install pip --upgrade to pip install

* don't test unittest with sleep 1 in windows

* debug with tmate

* increase timeout

* increase timeout

* skip test gloo

* skip test gloo

* fix synteax error

* skip test gloo in torch 1.13

* skip test gloo in torch 1.13

* skip testing setup_env in macOS

* skip test test_dist

* skip test setup

* restore triggered during push

* clean the code

* debug macos

* refine macOS CI

* test merge stage test

* trigger CI during push to main

* Fix as comment
2022-12-27 16:18:46 +08:00
RangiLyu
e83ac944b6
[Feature] Registry supports import modules automatically. (#643)
* [Feature] Support registry auto import modules.

* update

* rebase and fix ut

* add docstring

* remove count_registered_modules

* update docstring

* resolve comments

* resolve comments

* rename ut

* fix warning

* avoid BC breaking

* update doc

* update doc

* resolve comments
2022-12-23 15:46:29 +08:00
Xiangxu-0103
591024e533
[Docs] Update docstring of structures (#840)
* Update docstring of `structures`

* update docs

* add `import torch` to `examples`
2022-12-21 20:07:18 +08:00
Zaida Zhou
b44048aeb7
[Fix] Fix failed unit tests caused by the new released numpy (#838)
* [Fix] Fix failed unit tests caused by the new released numpy

* minor fix
2022-12-20 11:00:47 +08:00
Qian Zhao
4147e976a6
[Fix] ZeroRedundancyOptimizer ambiguous error with param groups when pytorch < 1.12.0 (#818)
* fix zero_optimizer error with param groups when pytorch < 1.12.0

* add docstring

* fix docstring

* add unittest

* change ut to use a valid paramwise_cfg

* modify ut

* fix as comments
2022-12-19 17:07:58 +08:00
RangiLyu
3582b4c787
[Enhance] Support setting decay multiplier for flatten parameter (#771)
* [Fix] Fix bias decay mult of depth-wise conv.

* support flatten param weight decay multiplier

* add unit tests

* REMOVE TODO

* update doc
2022-12-16 17:37:22 +08:00
Mashiro
95c71c82d0
[Fix] Only test Timer in Linux platform (#823) 2022-12-13 19:11:09 +08:00
Mashiro
d876d4e0f8
[Enhance] Add support of TorchVision's Model Registration API (#793)
* enhance get_torchvision_model

* remove mmcv
2022-12-11 22:08:51 +08:00
Ma Zerun
fe26c6559e
[Enhance] Support multiple --cfg-options. (#759) 2022-12-07 10:44:25 +08:00
Qian Zhao
bd6791382f
[Fix] BaseModel & BaseDataPreprocessor to method to be consistent with torch.nn.Module (#783)
* fix BaseModel `to` method to be consistent with torch.nn.Module

* fix data_preprocessor as well

* fix docstring alignment

* fix docstring alignment
2022-12-05 18:04:47 +08:00
SUNYC
7e787d8ffa
[Enhance] Add a new parameter conf_path in PetrelBackend (#774)
* feat: add conf_path in PetrelBackend

* Update mmengine/fileio/backends/petrel_backend.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>
2022-11-29 14:23:16 +08:00
liukuikun
5f58517069
[Enchance] remove redundant attributes in BaseDataElement (#690)
* remove redundant attribute in dump or print

* optimize efficiency

Co-authored-by: HAOCHENYE <21724054@zju.edu.cn>
2022-11-24 21:36:19 +08:00
Mashiro
ded73f3a56
[Enhance] Enhance compatibility of revert_sync_batchnorm (#695)
* [Enhance] Enhance revert_sync_batchnorm and convert_sync_batchnorm

* [Enhance] Enhance revert_sync_batchnorm and convert_sync_batchnorm

* Fix unit test

* Add coments

* Refine comments

* clean the code

* revert convert_sync_batchnorm

* revert convert_sync_batchnorm

* refine comment

* fix CI

* fix CI
2022-11-24 16:41:54 +08:00
cir7
0e6bb48b12
[Enhance] Support eta_min_ratio in CosineAnnealingParamScheduler (#725)
* [Enhance] support eta_min_ratio in CosineAnnealingParamScheduler

* [doc] fix docstring

* [Enhance] add ut for eta_min_ratio

* [doc] update docstring

* avoid bc-breaking of eta_min

* [doc] add docstring in CosineAnnealingParamScheduler and CosineAnnealingMomentum

* Apply suggestions from code review

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-11-22 20:19:16 +08:00
songyuc
6636f07cfe
[Feature] Add get_hooks_info() to print hooks messages (#672)
* Add test of get_hooks_info()

* Change to use original Runner for get_hook_info() test

* Change to test after_train_iter hooks for get_hook_info()

* Complement the stages list

* Add logging hooks information in Runner.__init__()

* Rearrange the stages list

* Restore the stages to tuple type

* Clean the unnecessary changes

* Replace  statement with TestCase's methods

* add test stages in method_stages_map

* change the hooks info into a f-string

* return list(trigger_stages) directly

* change keys of method_stages_map

* Fix previous changes to method_stages_map.keys
2022-11-22 20:02:29 +08:00
Mashiro
b06234cfcd
[Enhance] Right align the log (#436)
* right allign the log

* fix as comment

* Add comments for magic number 3

* remove max_len_str

* Update mmengine/runner/log_processor.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-11-21 11:55:18 +08:00
Mashiro
c478bdca27
[Enhance] enhance runner test case (#631)
* Add runner test cast

* Fix unit test

* fix unit test

* pop None if key does not exist

* Fix is_model_wrapper and force register class in test_runner

* [Fix] Fix is_model_wrapper

* destroy group after ut

* register module in testcase

* fix as comment

* minor refine

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* fix lint

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-11-21 11:54:05 +08:00
RangiLyu
a9a575866f [Fix] Fix offline_evaluate index error (#630)
* [Fix] Fix offline eval dataset index error.

* update

* update
2022-11-18 01:17:12 +08:00
RangiLyu
f2b0540f58 [Enhance] Raise warning for abnormal momentum (#655) 2022-11-18 01:17:12 +08:00
Yanhong Zeng
618a063ddc
[Docs] Fix docstring and documentation used for hub.get_model (#659)
* fix docstring and documentation used for hub.get_model

* fix config name in test_hub

* fix docstring mentioned faster-rcnn in utils
2022-11-02 10:16:56 +08:00
BigDong
376251961d
[Enhance] Add ignore_keys in ConcatDataset (#556)
* [Fix] Fix ConcatDataset error in VOCDataset

* minor fix

* minor fix

* minor fix

* add UT

* minor fix

* minor fix

* minor fix

* Update mmengine/dataset/dataset_wrapper.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* minor fix

* Update mmengine/dataset/dataset_wrapper.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-11-01 17:06:02 +08:00
Z-Fran
090104df21
[Fix] Fix the calculation error of eta_min in CosineRestart (#639)
* [Fix] fix CosineRestart eta_min

* add ut case

* Enhance unit test

Enhance unit test

* remove unused code

Co-authored-by: HAOCHENYE <21724054@zju.edu.cn>
2022-11-01 15:48:39 +08:00
Mashiro
d1dd240796
[Fix] Fix BaseDataPreprocessor.cast_data cound not handle string data (#602)
* [Fix] Fix  cound not handle string data

* Minor refine

* Refine type hint

Refine type hint

* fix as comment

* Minor refine

* Update mmengine/model/base_model/data_preprocessor.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-10-28 15:17:27 +08:00
Hakjin Lee
0857f9fb40
[Feature] Support torch ZeroRedundancyOptimizer (#551)
* [Feature] Support torch ZeRORedundancyOptimizer

Co-authored-by: Junhwa Song <ethan9867@gmail.com>
Signed-off-by: Junhwa Song <ethan9867@gmail.com>
Signed-off-by: Hakjin Lee <nijkah@gmail.com>

* lint

* Fix saving optimizer state_dict

* Fix handling import error

* Add test case

* fix UT

* Revert "fix UT"

This reverts commit dd64538960ff7440c6020f533d43945ffc23f2d2.

* fix handling import in UT

* Fix saving zero checkpoint and delete redundant master_only

* lint

* test unittest

* Fix handling impor error

* Fix UT condition

* Edit docstrings

* Fix typo

* Skip redundant procudure in checkpoint hook

* fix typo again

* Update mmengine/optim/optimizer/zero_optimizer.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Add api info

* lint

* Fix lint

* Handling AmpOptimWrapper case

* handling overlap_with_ddp

* Fix error

Signed-off-by: Junhwa Song <ethan9867@gmail.com>
Signed-off-by: Hakjin Lee <nijkah@gmail.com>
Co-authored-by: Junhwa Song <ethan9867@gmail.com>
Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-10-27 20:31:50 +08:00
Mashiro
c1447734c2
[Enhance] Call register_all_modules in Registry.get() (#541)
* call register_all_modules  in Registry.get()

* Fix ci

* fix scope bug (scope_name -> scope), for temp sync

* Fix unit test

* Refine log information

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Fix typo

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-10-24 20:15:14 +08:00
Mashiro
bda92b49e7
[Feature] Add a function to mark the deprecated function. (#609)
* [Enhancement] Add a deprecated function

* Refine and add ut

* tmp save

* minor refine

* Fix unit test

* Fix unit test

* fix docstring

* Update utils.rst
2022-10-24 14:49:48 +08:00
wangjiangben-hw
601db12d38
[Feature] Support engine with NPU backend. (#572)
* init npu

* Update mmengine/optim/optimizer/amp_optimizer_wrapper.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/dist/dist.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* change to is_hccl_backend

* Update mmengine/optim/optimizer/amp_optimizer_wrapper.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* add comment with AmpOptimWrapper

* Update mmengine/runner/amp.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/runner/amp.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* add npu fn in base_model

* Update mmengine/optim/optimizer/amp_optimizer_wrapper.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* clean lint

* Update mmengine/optim/optimizer/amp_optimizer_wrapper.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/model/base_model/base_model.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* add is_npu_available

* try to fix

* Add comments

* Refine grammar

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>
Co-authored-by: HAOCHENYE <21724054@zju.edu.cn>
2022-10-24 11:42:29 +08:00
shenmishajing
d270516fe8
[Fix]: fix add_config func is not called bug (#613)
* fix add_config func is not called bug
fix wandb add_config bug
add wandb log_code feature

* move log_code_name param to the last one
add log_code_name param to docstring

* add config only when there is a cfg

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* add unit test for log_code_name param of WandbVisBackend

* Update mmengine/visualization/vis_backend.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* use log_code_kwargs instead of only log_code_name
add PATH_FILTERS registry

* use log_code_kwargs instead of only log_code_name
add PATH_FILTERS registry

* fix add config unit test

* roll back to log_code_name version

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-10-24 09:58:10 +08:00
Mashiro
aaba1d8871
[Fix] Fix error format of log message (#508)
* Fix error format of log message

* Fix unit test

* remove unnecessary comment
2022-10-18 18:04:15 +08:00
Mashiro
abe56651db
[Fix] Make autocast compatible with mps (#587)
* [Fix] Make autocast compatible with mps

* Enhance unit test

* fix unit test

* clean the code

* fix unit test
2022-10-18 18:03:17 +08:00
Mashiro
6073d9ebd8
[Enhance] add documents for clip_grad , and support clip grad by value. (#513)
* [Enhance] add documents for , and support clip grad by value

* refine docstring

* fix as comment

* Fix as comment

* minor refine

* minor refine

* remove error comment for clip grad

* refine docstring
2022-10-18 18:02:46 +08:00
shenmishajing
c2982723c9
[Feats]: add non_blocking feature to BaseDataPreprocessor (#618)
* add non_blocking feature to BaseDataPreprocessor

* Update mmengine/model/base_model/data_preprocessor.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* move new parameters to the last to avoid bc issue

* Update mmengine/model/base_model/data_preprocessor.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/model/base_model/data_preprocessor.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-10-18 17:58:51 +08:00
Mashiro
62456217f9
[Feature] Add test time augmentation base model. (#538)
* First commit

* add BaseTestTimeAugModel

* Add unit test

* move loop logic to test_step

* fix ddp test

* rename model to module

* optim __init__

* Fix as comment

* Fix as comment

* make val_step should not be called

* make tta do not inherit base model

* Fix unit test

* Enhance docstring

* Fix as comment

* Fix as comment

* minor refine

* minor refine

* minor refine

* fix unit test

* minor refine

* minor refine

* minor refine

* minor refine

* minor refine

* minor refine

* fix unit test
2022-10-10 14:29:33 +08:00
Qian Zhao
e73c4bf135
[Fix] some out-of-date unittests (#586)
* fix dist ut

* fix config & hub ut break due to mmdet faster-rcnn rename
2022-10-08 19:48:51 +08:00
Mashiro
89146a5b90
[Fix] Fix detect_anomalous_params (#588) 2022-10-08 19:48:35 +08:00
Zaida Zhou
ed84dfd34d
[Refactor] Refactor fileio without breaking back compatibility (#533)
* [Refactor] Refactor fileio but without breaking bc

* handle compatibility

* fix format

* modify io functions

* fix ut

* fix ut

* rename method names

* refine

* refine docstring

* fix ut in windows

* update ut

* minor fix

* ensure client is not None when closing it

* add more examples for list_dir_or_file interface

* refine docstring

* refine deprecated info

* fix ut

* add a description for lmdb docstring
2022-09-26 14:30:40 +08:00
Qian Zhao
c64243aa9e
[Fix] CheckpointHook behavior incorrect if given filename_tmpl argument (#518) 2022-09-22 12:47:45 +08:00
Mashiro
5d7527d75e
[Fix ] Fix unit test could may fail caused by MultiProcessTestCase (#535)
* fix merge ci

* make timeout larger

* tmp test merge

* debug

* sleep 1 in optimizer ut

* skip test clip grad

* skip test clip grad

* fix merge ci
2022-09-15 18:08:56 +08:00
Mashiro
8ee31dbc3b
[Feature] Support convert BN to SyncBN by config (#506)
* [Feature] Support convert BN to SyncBN by config

* make unit test compatible with cpu

* Fix as comment

* fix unit test

* change signature of convert_sync_batchnorm: rename sync_bn to implemention

* fix unit test

* fix unit test
2022-09-15 18:08:36 +08:00
Mashiro
6b1b8a3751
[Fix] Fix unit test in windows (#515) 2022-09-13 11:46:21 +08:00
Mashiro
0fb2b8ca8c
[Fix] Fix circle import in EMAHook (#523) 2022-09-09 19:33:27 +08:00
takuoko
a6f5297727
[fix] EMAHook load state dict (#507)
* fix ema load_state_dict

* fix ema load_state_dict

* fix for test

* fix by review

* fix resume and keys
2022-09-09 11:41:12 +08:00
Zaida Zhou
19d4eba866
[Refactor] Update Github Action CI and CircleCI (#512)
* [CI] Update Github Action CI and CircleCI

* clear outdated commands in CI

* clear outdated commands in CI

* add '-r' when installing packages from txt

* update

* update circleci

* fix mismatch conflicts

* update

* install system dependencies

* update

* set env

* fix ut

* update cuda version on Windows

* update push action
2022-09-07 20:24:11 +08:00
Mashiro
2d3bff4418
[Fix] Fix unit test of Config will install mmdet and mmcls. (#492)
* fix unit test install mmdet and mmcls

* raise error when mmdet is not installed

* rename check_and_install to install_package

* split test case

Co-authored-by: C1rN09 <zhaoqian@pjlab.org.cn>
2022-08-31 21:18:05 +08:00
liukuikun
1fea82aad5
[Docs] update data element tutorials (#431)
* structure tutorials

* refine data element docs

* modify introduce

* fix comment

* fix comment

* fix comment
2022-08-31 20:39:08 +08:00
Qian Zhao
94412f72d8
[Fix] Exception while building registered function (#491) 2022-08-31 13:53:32 +08:00