Commit Graph

331 Commits (1ab646de1707c80529fc9104cded58cee375fba9)

Author SHA1 Message Date
shenmishajing d270516fe8
[Fix]: fix add_config func is not called bug (#613)
* fix add_config func is not called bug
fix wandb add_config bug
add wandb log_code feature

* move log_code_name param to the last one
add log_code_name param to docstring

* add config only when there is a cfg

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* add unit test for log_code_name param of WandbVisBackend

* Update mmengine/visualization/vis_backend.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* use log_code_kwargs instead of only log_code_name
add PATH_FILTERS registry

* use log_code_kwargs instead of only log_code_name
add PATH_FILTERS registry

* fix add config unit test

* roll back to log_code_name version

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-10-24 09:58:10 +08:00
Mashiro aaba1d8871
[Fix] Fix error format of log message (#508)
* Fix error format of log message

* Fix unit test

* remove unnecessary comment
2022-10-18 18:04:15 +08:00
Mashiro abe56651db
[Fix] Make autocast compatible with mps (#587)
* [Fix] Make autocast compatible with mps

* Enhance unit test

* fix unit test

* clean the code

* fix unit test
2022-10-18 18:03:17 +08:00
Mashiro 6073d9ebd8
[Enhance] add documents for `clip_grad` , and support clip grad by value. (#513)
* [Enhance] add documents for , and support clip grad by value

* refine docstring

* fix as comment

* Fix as comment

* minor refine

* minor refine

* remove error comment for clip grad

* refine docstring
2022-10-18 18:02:46 +08:00
shenmishajing c2982723c9
[Feats]: add non_blocking feature to BaseDataPreprocessor (#618)
* add non_blocking feature to BaseDataPreprocessor

* Update mmengine/model/base_model/data_preprocessor.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* move new parameters to the last to avoid bc issue

* Update mmengine/model/base_model/data_preprocessor.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/model/base_model/data_preprocessor.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2022-10-18 17:58:51 +08:00
Mashiro 62456217f9
[Feature] Add test time augmentation base model. (#538)
* First commit

* add BaseTestTimeAugModel

* Add unit test

* move loop logic to test_step

* fix ddp test

* rename model to module

* optim __init__

* Fix as comment

* Fix as comment

* make val_step should not be called

* make tta do not inherit base model

* Fix unit test

* Enhance docstring

* Fix as comment

* Fix as comment

* minor refine

* minor refine

* minor refine

* fix unit test

* minor refine

* minor refine

* minor refine

* minor refine

* minor refine

* minor refine

* fix unit test
2022-10-10 14:29:33 +08:00
Qian Zhao e73c4bf135
[Fix] some out-of-date unittests (#586)
* fix dist ut

* fix config & hub ut break due to mmdet faster-rcnn rename
2022-10-08 19:48:51 +08:00
Mashiro 89146a5b90
[Fix] Fix detect_anomalous_params (#588) 2022-10-08 19:48:35 +08:00
Zaida Zhou ed84dfd34d
[Refactor] Refactor fileio without breaking back compatibility (#533)
* [Refactor] Refactor fileio but without breaking bc

* handle compatibility

* fix format

* modify io functions

* fix ut

* fix ut

* rename method names

* refine

* refine docstring

* fix ut in windows

* update ut

* minor fix

* ensure client is not None when closing it

* add more examples for list_dir_or_file interface

* refine docstring

* refine deprecated info

* fix ut

* add a description for lmdb docstring
2022-09-26 14:30:40 +08:00
Qian Zhao c64243aa9e
[Fix] CheckpointHook behavior incorrect if given `filename_tmpl` argument (#518) 2022-09-22 12:47:45 +08:00
Mashiro 5d7527d75e
[Fix ] Fix unit test could may fail caused by `MultiProcessTestCase` (#535)
* fix merge ci

* make timeout larger

* tmp test merge

* debug

* sleep 1 in optimizer ut

* skip test clip grad

* skip test clip grad

* fix merge ci
2022-09-15 18:08:56 +08:00
Mashiro 8ee31dbc3b
[Feature] Support convert `BN` to `SyncBN` by config (#506)
* [Feature] Support convert BN to SyncBN by config

* make unit test compatible with cpu

* Fix as comment

* fix unit test

* change signature of convert_sync_batchnorm: rename sync_bn to implemention

* fix unit test

* fix unit test
2022-09-15 18:08:36 +08:00
Mashiro 6b1b8a3751
[Fix] Fix unit test in windows (#515) 2022-09-13 11:46:21 +08:00
Mashiro 0fb2b8ca8c
[Fix] Fix circle import in EMAHook (#523) 2022-09-09 19:33:27 +08:00
takuoko a6f5297727
[fix] EMAHook load state dict (#507)
* fix ema load_state_dict

* fix ema load_state_dict

* fix for test

* fix by review

* fix resume and keys
2022-09-09 11:41:12 +08:00
Zaida Zhou 19d4eba866
[Refactor] Update Github Action CI and CircleCI (#512)
* [CI] Update Github Action CI and CircleCI

* clear outdated commands in CI

* clear outdated commands in CI

* add '-r' when installing packages from txt

* update

* update circleci

* fix mismatch conflicts

* update

* install system dependencies

* update

* set env

* fix ut

* update cuda version on Windows

* update push action
2022-09-07 20:24:11 +08:00
Mashiro 2d3bff4418
[Fix] Fix unit test of `Config` will install `mmdet` and `mmcls`. (#492)
* fix unit test install mmdet and mmcls

* raise error when mmdet is not installed

* rename check_and_install to install_package

* split test case

Co-authored-by: C1rN09 <zhaoqian@pjlab.org.cn>
2022-08-31 21:18:05 +08:00
liukuikun 1fea82aad5
[Docs] update data element tutorials (#431)
* structure tutorials

* refine data element docs

* modify introduce

* fix comment

* fix comment

* fix comment
2022-08-31 20:39:08 +08:00
Qian Zhao 94412f72d8
[Fix] Exception while building registered function (#491) 2022-08-31 13:53:32 +08:00
Mashiro 33d7c81bc4
[Refactor]: Avoid creating instances with the same name (#483) 2022-08-30 16:28:18 +08:00
wxDai fba9a94f52
[Refactor] add testing utils (#475)
* add testing utils

* fix ut

* add blank line betweeen `Args` and `Returns`
2022-08-29 22:59:20 +08:00
Zaida Zhou 2f09342663
[Docs] Refine registry docs (#443)
* [Docs] Refine registry docs

* explain how to use _scope_

* refine
2022-08-29 14:50:48 +08:00
Mashiro 576e5c8f91
[Fix] Regist pytorch ddp and dp to `MODEL_WRAPPERS`, add unit test to `is_model_wrapper` (#474)
* regist pytorch ddp and dp, add unit test

* minor refine

* Support check custom wrapper

* enhance ut
2022-08-28 17:09:41 +08:00
RangiLyu 8d25dbdeda
[Fix] Fix EMAHook trigger train loop and AveragedModel sync buffer. (#467)
* [Fix] Fix EMAHook trigger train loop init during testing.

* fix sync buffer

* update ut

* fix sync buffer

* fix sync buffer
2022-08-26 14:21:56 +08:00
Mashiro 8770c6c7fc
[Refactor] Refactor data flow to make the interface more natural (#468)
* [Refactor]: modify interface of Visualizer.add_datasample (#365)

* [Refactor] Refactor data flow: refine `data_preprocessor`. (#359)

* refine data_preprocessor

* remove unused BATCH_DATA alias

* Fix type hints

* rename move_data to cast_data

* [Refactor] Refactor data flow: collate data in `collate_fn` of `DataLoader`  (#323)

* acollate data in dataloader

* fix docstring

* refine comment

* fix as comment

* refactor default collate and psedo collate

* foramt test file

* fix docstring

* fix as comment

* rename elem to data_item

* minor fix

* fix as comment

* [Refactor] Refactor data flow: `data_batch` argument of `Evaluator.process is a `dict` (#360)

* refine evaluator and metric

* compatible with new default collate

* replace default collate with pseudo

* Handle data_batch in metric

* fix unit test

* fix unit test

* fix unit test

* minor refine

* make data_batch optional

make data_batch optional

* rename outputs to predictions

* fix ut

* rename predictions to outputs

* fix docstring

* fix docstring

* fix unit test

* make outputs and data_batch to kwargs

* fix unit test

* keep signature of metric

* fix ut

* rename pred_sample arguments to data_sample(Visualizer)

* fix loop and ut

* [refactor]: Refactor model dataflow (#398)

* [Refactor] Refactor data flow: refine `data_preprocessor`. (#359)

* refine data_preprocessor

* remove unused BATCH_DATA alias

* Fix type hints

* rename move_data to cast_data

* refactor model data flow

tmp_commt

tmp commit

* make val_cfg and test_cfg optional

* roll back runner

* pass test mmdet

* fix as comment

fix as comment

fix ci in DataPreprocessor

* fix ut

* fix ut

* fix rebase main

* [Fix]: Fix test val ddp (#462)

* [Fix] Fix docstring and type hint of data flow (#463)

* Fix docstring of data flow

* change signature of hook

* fix unit test

* resolve conflicts

* fix lint
2022-08-24 22:04:55 +08:00
Zaida Zhou 7e1d7af2d9
[Refactor] Refactor code structure (#395)
* Rename data to structure

* adjust the way to import module

* adjust the way to import module

* rename Structure to Data Structures in docs api

* rename structure to structures

* support using some modules of mmengine without torch

* fix circleci config

* fix circleci config

* fix registry ut

* minor fix

* move init method from model/utils to model/weight_init.py

* move init method from model/utils to model/weight_init.py

* move sync_bn to model

* move functions depending on torch to dl_utils

* format import

* fix logging ut

* add weight init in model/__init__.py

* move get_config and get_model to mmengine/hub

* move log_processor.py to mmengine/runner

* fix ut

* Add TimeCounter in dl_utils/__init__.py
2022-08-24 19:14:07 +08:00
Zaida Zhou 486d8cda56
[Refactor] Refactor the import rule (#459)
* [Refactor] Refactor the import rule

* minor refinement

* add a comment
2022-08-23 18:58:36 +08:00
Zaida Zhou a9ad09bded
[Fix] Fix utils ut (#458) 2022-08-23 16:56:47 +08:00
liukuikun bb56cf42ab
[Visualizer] use FigureManager to manage figure to avoid affecting plt.show() outside Visualizer(#440)
* figure in Visualizer is not managed by plt

* encapsulate code and remove unused code
2022-08-22 17:16:14 +08:00
Zaida Zhou b75962a660
Migrate utils from mmcv (#447) 2022-08-22 11:51:56 +08:00
Mashiro e907931fb8
Fix unit tests (#449) 2022-08-21 14:54:24 +08:00
Zaida Zhou 429bb27972
[Refactor]: migrate LMDB backends and modifications from open-mmlab/mmcv#1764 and open-mmlab/mmcv#1747 (#426)
* Add fileio ut

* pick modification from master branch

* pick modification from mmcv #1764

* test fileio without mmcv

* remove unnecessary checking
2022-08-19 16:06:49 +08:00
Mashiro 4abf1a0454
[Enhance] Support build evaluator from list of built metric (#423)
* Support build evaluator from list of built metric

* regist evaluator

* fix as comment

* add unit test
2022-08-19 10:56:51 +08:00
Mashiro d6ad01a4cf
[Fix]: fix ci (#441) 2022-08-18 14:04:19 +08:00
Mashiro e08b9031fc
[Enhance] Support building optimizer wrapper from built Optimizer instance (#422)
* support build optimizer wrapper from built Optimizer instance

* refine comments
2022-08-17 19:17:00 +08:00
Mashiro a706bbc018
[Fix]: fix error and add unit test (#429) 2022-08-17 19:16:00 +08:00
Zaida Zhou 4637c49640
[Refactor] Rename test_hook to test_hooks (#425) 2022-08-15 11:45:07 +08:00
Mashiro 438e8e7498
BaseModel support recursively set the device of data_preprocessor (#387) 2022-08-15 11:04:24 +08:00
Zaida Zhou f98ba60629
[Enhancement] Improve unit tests of mmengine/runner (#182)
* [Enhancement] Add unit test for get_priority

* fix priority ut

* fix typo

Co-authored-by: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com>
2022-08-15 10:57:58 +08:00
Mashiro 2708b7ed48
fix ci (#424) 2022-08-13 09:15:08 +08:00
RangiLyu 813f49bf23
[Feature] Support CosineRestartParamScheduler. (#397)
* [Feature] Support CosineRestartParamScheduler.

* add ut and docstring

* add docstring
2022-08-11 17:57:35 +08:00
Mashiro b14cbc2576
[Fix] Fix wrong epoch and iter when saving best ckpt (#400)
* fix wrong epoch andd iter when save bbest ckpt

* fix ut

* fix resume best ckpt unexpectedly

* minor refine

* fix unit test
2022-08-11 14:52:38 +08:00
Mashiro ee56f151f6
[Fix] Support training with data without `metainfo`. (#417)
* support training with data without metainfo

* clean the code

* clean the code
2022-08-11 14:51:11 +08:00
Ma Zerun 9b2a0e02da
[Enhance] Add `data_preprocessor` config as an argument of runner. (#343)
* [Enhance] Add `preprocess_cfg` as an argument of runner.

* Rename `preprocess_cfg` to `data_preprocessor`

* Fix docstring
2022-08-09 11:25:29 +08:00
Mashiro cd257c9047
[Enhance] Support log enviroment information during initiate runner (#384)
* Support log enviroment information when initiate runner

* Fix unit test

* fix as comment, save world_size

* log gpu num

* clear code and reformat log

* minor refine

* fix as comment

* minor refine

* clean the code

* clean the code

* remove save world_size in meta
2022-08-08 22:23:52 +08:00
Mashiro 7e2302388f
[Feature] add config new feature (#105) 2022-08-08 21:01:06 +08:00
Mashiro a07a063306
[Enhance] Add build function for scheduler. (#372)
* add build function for scheduler

* add unit test

add unit test

* handle convert_to_iter in build_scheduler_from_cfg

* restore deleted code

* format import

* fix lint
2022-08-08 20:34:16 +08:00
Mashiro 99de0951af
[Enhance] EMAHook support does not load checkpoint strictly (#352)
* BaseAveragedModel support load ckpt without module prefix

* refine docstring

* allow EMAHook does not load ckpt strictly

* add unit test for strict argument of EMAHook

* sync remote

* sync remote

* clean the code

* ema hook supports setting start iter

* fix unit test

* fix as comment

* fix as comment

* describe kwargs
2022-08-08 20:26:16 +08:00
LeoXing1996 08602a2385
[Enhancement] Support save best based on multi metrics (#349)
* support save best based on multi metrics

* add unit test

* resolve bugs after rebasing

* revise docstring

* revise docstring

* fix as comment

* revise as comment
2022-08-08 20:17:17 +08:00
Mashiro 6ebb7ed481
[Fix] Fix config cannot change(assign) the value of the dict in config(without base config) (#378)
* Support changing value of dict in config(without base config)

* minor refine
2022-08-08 17:07:47 +08:00
Mashiro 5580542666
[Fix] Fix build multiple list of scheduler for multiple optimizers (#383)
* fix build multiple scheduler

* add new unit test

* fix comment and error message

* fix comment and error message

* extract _parse_scheduler_cfg

* always call build_param_scheduler during train and resume. If there is only one optimizer, the defaut value for sheduler will be a list, otherwise there is multiple optimizer, the default value of sheduler will be a dict

* minor refine

* rename runner test exp name

* fix as comment

* minor refine

* fix ut

* only check parameter scheduler

* minor refine
2022-08-08 17:05:27 +08:00
Mashiro 1a8f013937
[Refine] Make scheduler default to None (#396)
* make scheduler default to None

* fix bc breaking

* refine warning message

* fix as comment

* fix as comment

* fix lint
2022-08-04 20:13:13 +08:00
Mashiro df4e6e3294
[Fix] Fix resume `message_hub` and save `metainfo` in message_hub. (#394)
* Fix resume message hub and save metainfo in messagehub

* fix as comment
2022-08-02 19:06:35 +08:00
Mashiro f850de71c3
[Fix] Support use 'global variable' in config function (#390)
* Support use 'global var' in config function

* upload test file
2022-08-01 20:10:10 +08:00
RangiLyu 4432e54c97
[Fix] Fix gpu tensors in results list are not on the same device. (#385)
* [Fix] Fix gpu tensors in results list are not on the same device.

* cast all tensor to cpu
2022-08-01 10:20:57 +08:00
Miao Zheng 39e7efb04d
[Fix] Revise UT of OneCycle schedulor (#388) 2022-07-27 16:22:00 +08:00
Mashiro 6bd548d8bb
[FIx] Logging time in LoggerHook.after_val_epoch has no effect (#376) 2022-07-20 16:51:11 +08:00
Mashiro 6b47035fdf
[Fix] Fix save scheduler state dict with optim wrapper (#375)
* fix save scheduler state dict with optim wrapper

* remove for loop and inherit TestParameterScheduler

* remove for loop and inherit TestParameterScheduler

* minor refine
2022-07-20 16:32:48 +08:00
Tong Gao 5b065b10fd
[Enhance] Support Compose(None) (#373)
* [Enhance] Allow Compose(None)

* add typehint

* fix
2022-07-20 16:04:24 +08:00
Mashiro ea61bf6bb7
Fix: avoid modification of scalar_dict in LocalVisBackend (#377) 2022-07-20 16:03:01 +08:00
RangiLyu 1241c21296
[Fix] Fix weight initializing in test and refine registry logging. (#367)
* [Fix] Fix weight initializing and registry logging.

* sync params

* resolve comments
2022-07-19 18:28:57 +08:00
Ma Zerun 3da66d1f87
[Enhance] Auto set the `end` of param schedulers. (#361)
* [Enhance] Auto set the `end` of param schedulers.

* Add log output and unit test

* Update docstring

* Update unit tests of `CosineAnnealingParamScheduler`.
2022-07-15 19:53:28 +08:00
Mashiro 78fad67d0d
[Fix] fix resume message_hub (#353)
* fix resume message_hub

* add unit test

* support resume from messagehub

* minor refine

* add comment

* fix typo

* update docstring
2022-07-14 20:13:22 +08:00
Mashiro 45001a1f6f
[Enhance] Support using variables in base config directly as normal variables. (#329)
* first commit

* Support modify base config and add unit test

* remove import mmengine in config

* add unit test

* fix lint

* add unit test

* move RemoveAssignFromAST to config utils

* git add utils

* fix format issue in test file

* refine unit test

* refine unit test
2022-07-14 13:05:55 +08:00
Mashiro 6b608b4ef1
[Enhance] Add `build_model_from_cfg` (#328)
* clean code

* fix as comment

* fix as comment

* add get_registry_by_scope method

* add unit test and docstring example

* rename get_registry_by_scope to switch_scope_and_registry

* move build function to registry/builder

* fix docstring

* rename builder->registry_builder, move build_from_cfg to registry_builder

rename builder->registry_builder, move build_from_cfg to registry_builder

* rename registry_builder to build_function

rename registry_builder to build_function

* fix docstring and type hint

* rename build_function to build_functions
2022-07-13 19:01:59 +08:00
Mashiro b2ee9f8b11
[Fix] Fix loss could be nan in optimizer wrapper (#345)
* fix optimizer wrapper counts

* fix ut
2022-07-06 16:42:49 +08:00
RangiLyu a3d2916790
[Enhance] Support scheduling betas with MomentumScheduler. (#346)
* [Enhance] Support scheduling betas with MomentumScheduler.

* enhance ut

* test adam betas

* enhance ut

* enhance ut
2022-07-05 20:37:23 +08:00
Mashiro 2853045e96
[Fix] Fix build multiple runners error (#348)
* fix build multiple runner error

* fix comments

* fix cpu ci
2022-07-05 20:35:06 +08:00
Mashiro 38e78d5549
[Fix] Fix ema hook and add unit test (#327)
* Fix ema hook and add unit test

* save state_dict of ema.module

save state_dict of ema.module

* replace warning.warn with MMLogger.warn

* fix as comment

* fix bug

* fix bug
2022-07-04 14:23:23 +08:00
Cedric Luo 9c55b4300c
[Enhance] Support dynamic interval (#342)
* support dynamic interval in iterbasedtrainloop

* update typehint

* update typehint

* add dynamic interval in epochbasedtrainloop

* update

* fix

Co-authored-by: luochunhua.vendor <luochunhua@pjlab.org.cn>
2022-06-30 15:08:56 +08:00
LeoXing1996 d65350a9da
[Fix] Fix bug of not save-best in iteration-based training (#341)
* fix bug of not save-best in iteration-based training

* revise the unit test
2022-06-30 14:51:31 +08:00
Mashiro 59b0ccfe6f
[Fix] Fix pytorch version compatibility of autocast (#339)
* fix unit test of autocast

* fix compatiblity of unit test of optimizerwrapper

* clean code

* fix as comment

* fix docstring
2022-06-29 20:30:53 +08:00
Mashiro 5ac3c23338
[Fix]: fix MMSeparateDistributedDataParallel (#338) 2022-06-28 22:20:20 +08:00
Mashiro d624fa9191
[Enhance] assert image shape before forward (#300)
* assert image shape before forward

* add unit test

* enhance error message

* allow gray image input

* fix as comment

* fix unit test

* fix unit test
2022-06-28 11:46:12 +08:00
Mashiro 2fd6beb972
[Fix] Fix UT of optimizer wrapper failed in pytorch1.6 (#340) 2022-06-28 10:31:14 +08:00
Jiazhen Wang 3af3d40541
[Enhance] Refine BaseDataset (#303)
* refine data_root and data_prefix params

* modify unittest
2022-06-27 14:59:56 +08:00
Yuan Liu 03d5c17ba6
[Feature]: Set different seed to different rank (#298)
* [Feature]: Set different seed for diff rank

* [Feature]: Add log

* [Fix]: Fix lint

* [Fix]: Fix docstring

* [Fix]: Fix sampler seed

* [Fix]: Fix log bug

* [Fix]: Change diff_seed to diff_rank_seed

* [Fix]: Fix lint
2022-06-24 14:28:16 +08:00
Alex Yang 2994195be2
[Feat] Support training on MPS (#331)
* [Feat] Support mps

* fix docstring
2022-06-23 16:53:19 +08:00
Haian Huang(深度眸) 2b8a32eca0
[Fix]: fix RuntimeError of SyncBuffersHook (#309)
* fix RuntimeError of SyncBuffersHook

* add UT
2022-06-22 20:00:46 +08:00
Alex Yang e18832f046
[Feat] Support revert syncbn (#326)
* [Feat] Support revert syncbn

* use logger.info but not warning

* fix info string
2022-06-22 19:50:54 +08:00
Mashiro 312f264ecd
[Feature] Add autocast wrapper (#307)
* add autocast wrapper

* fix docstring

* fix docstring

* fix compare version

* fix unit test

* fix incompatible arguments

* fix as comment

* fix unit test

* rename auto_cast to autocast
2022-06-22 19:49:20 +08:00
Alex Yang 216521a936
[Feat] Support save best ckpt (#310)
* [Feat] Support save best ckpt

* reformat code

* rename function and reformat code

* fix logging info
2022-06-22 19:48:46 +08:00
Mashiro 7154df2618
[Enhance] LogProcessor support custom significant digit (#311)
* LogProcessor support custom significant digit

* rename to num_digits
2022-06-22 19:35:52 +08:00
Mashiro afeac1c098
[Feature]: support to dump result in LoggerHook.after_test_epoch (#321) 2022-06-22 19:10:58 +08:00
Alex Yang dceef1f66f
[Refactor] Refactor `after_val_epoch` to make it output metric by epoch (#278)
* [Refactor]:Refactor `after_val_epoch` to make it output metric by epoch

* add an option for user to choose the way of outputing metric

* rename variable

* reformat docstring

* add type alias

* reformat code

* add test function

* add comment and test code

* add comment and test code
2022-06-21 15:39:59 +08:00
Alex Yang ef946404e6
[Feat] Support FSDP Training (#304)
* [Feat] Support FSDP Training

* fix version comparison

* change param format and move `FSDP_WRAP_POLICY` to wrapper file

* add docstring and type hint,reformat code

* fix type hint

* fix typo, reformat code
2022-06-21 15:32:56 +08:00
Mashiro 4a4d6b1ab2
[Enhance] dump messagehub in runner.resume (#237)
* [Enhance] dump messagehub in runner.resume

* delete unnecessary code

* delete debugging code

Co-authored-by: imabackstabber <312276423@qq.com>
2022-06-17 11:10:37 +08:00
Mashiro 7129a98e36
[Fix]: fix log processor to log average time and grad norm (#292) 2022-06-17 10:54:20 +08:00
Jiazhen Wang 7b55c5bdbf
[Feature] Support resume from Ceph (#294)
* support resume from ceph

* move func and refine

* delete symlink

* fix unittest

* perserve _allow_symlink and symlink
2022-06-17 10:37:19 +08:00
Jiazhen Wang d0d7174274
[Feature] Support MLU Devices (#288)
* support mlu

* add ut and refine docstring
2022-06-16 20:28:09 +08:00
Mashiro 7d3224bf46
[Fix] Fix setLevel of MMLogger (#297)
* Fix setLevel of MMLogger

Fix setLevel of MMLogger

* add docstring and comment
2022-06-14 14:54:25 +08:00
RangiLyu 1c18f30854
[Enhance] Support infinite dataloader iterator wrapper for IterBasedTrainLoop. (#289) 2022-06-14 14:52:59 +08:00
Alex Yang 5016332588
[Feat] support registering function (#302) 2022-06-14 14:50:24 +08:00
RangiLyu 4cd91ffe15
[Feature] Dump predictions to a pickle file for offline evaluation. (#293)
* [Feature] Dump predictions to pickle file for offline evaluation.

* print_log
2022-06-14 14:48:21 +08:00
Mashiro b7866021c4
[Refactor] Refactor the accumulate gradient implemention of OptimWrapper (#284)
* merge context

* update unit test

* add docstring

* fix bug in AmpOptimWrapper

* add docstring for backward

* add warning and docstring for accumuate gradient

* fix docstring

* fix docstring

* add params_group method

* fix as comment

* fix as comment

* make default_value of loss_scale to dynamic

* Fix docstring

* decouple should update and should no sync

* rename attribute in OptimWrapper

* fix docstring

* fix comment

* fix comment

* fix as comment

* fix as comment and add unit test
2022-06-13 23:20:53 +08:00
Miao Zheng fd295741ca
[Features]Add OneCycleLR (#296)
* [Features]Add OnecycleLR

* [Features]Add OnecycleLR

* yapf disable

* build_iter_from_epoch

* add epoch

* fix args

* fix according to comments;

* lr-param

* fix according to comments

* defaults -> default to

* remove epoch and steps per step

* variabel names
2022-06-13 21:23:59 +08:00
Mashiro 8b0c9c5f6f
[Fix] fix build train_loop during test (#295)
* fix build train_loop during test

* fix build train_loop during test

* fix build train_loop during test

* fix build train_loop during test

* Fix as comment
2022-06-13 21:23:46 +08:00
RangiLyu 819e10c24c
[Fix] Fix image dtype when enable_normalize=False. (#301)
* [Fix] Fix image dtype when enable_normalize=False.

* update ut

* move to collate

* update ut
2022-06-13 21:21:19 +08:00
Mashiro bcab813242
[Feature] Add ModuleList Sequential and ModuleDict (#299)
* add module list

* add module list

* fix docstring
2022-06-13 13:51:07 +08:00
Alex Yang df0c510444
[Feat]:support customizing evaluator (#287)
* [Feat]:support customizing evaluator

* fix keyname of determining using default evaluator or not

* add assertion

* fix typo
2022-06-10 15:34:10 +08:00
liukuikun c90b95a44b
[Fix]: fix label data and support empty tensor in label_to_onehot (#291) 2022-06-10 15:12:41 +08:00
RangiLyu 2f16ec69fb
[Feature] Support overwrite default scope with "_scope_". (#275)
* [Feature] Support overwrite default scope with "_scope_".

* add ut

* add ut
2022-06-09 20:16:31 +08:00
jbwang1997 7a5d3c83ea
[Fix] Replace auto_scale_lr_cfg to auto_scale_lr (#286)
* Replace auto_scale_lr_cfg to auto_scale_lr

* Update
2022-06-09 20:15:36 +08:00
Mashiro 931db99005
[Enhance] Enhance img data preprocessor (#290)
* fix BaseDataPreprocessor

* fix BaseDataPreprocessor

* change device type to torch.device

* change device type to torch.device

* fix cpu method of base model

* Allow ImgDataPreprocessor do not normalize

* remove unnecessary type ignore

* make mean and std optional

* refine docstring
2022-06-09 20:12:15 +08:00
Mashiro a9afdad7a8
[Fix] Fix BaseDataPreprocessor and BaseModel (#285)
* fix BaseDataPreprocessor

* fix BaseDataPreprocessor

* change device type to torch.device

* change device type to torch.device

* fix cpu method of base model
2022-06-09 11:45:19 +08:00
Mashiro 6ee675430f
[Refactor]: change order of BaseModel arguments (#282) 2022-06-08 13:28:00 +08:00
Mashiro f04fec736d
[Feature]: add base model, ddp model wrapper and unit test (#268)
* add base model, ddp model and unit test

* add unit test

* fix unit test

* fix docstring

* fix cpu unit test

* refine base data preprocessor

* refine base data preprocessor

* refine interface of ddp module

* remove optimizer hook

* add forward

* fix as comment

* fix unit test

* fix as comment

* fix build optimizer wrapper

* rebase main and fix unit test

* stack_batch support stacking ndim tensor, add docstring for merge dict

* fix lint

* fix test loop

* make precision_context effective to data_preprocessor

* fix as comment

* fix as comment

* refine docstring

* change collate_data output typehints

* rename to_rgb to bgr_to_rgb and rgb_to_bgr

* support build basemodel with built DataPreprocessor

* fix as comment

* fix docstring
2022-06-07 22:13:53 +08:00
RangiLyu ad965a5309
[Enhance] Enhance checkpoint meta info. (#279) 2022-06-07 18:48:50 +08:00
Mashiro 538ff48aec
[Fix] Rename data_list and support loading from ceph in dataset (#240)
* rename datalist and support load ceph

* rename datalist and support load ceph

* remove check disk file path in _load_metainfo

* fix rename error

* fix rename error

* unit test error

* fix rename error

* remove unnecessary code

* fix lint
2022-06-07 17:09:33 +08:00
jbwang1997 bd3c53b385
[Fix] Fix CI after merging support auto scale lr and support custom runner (#280) 2022-06-07 16:03:51 +08:00
jbwang1997 8f3fcee301
[Feature] Add auto scale lr fucntion (#270)
* Add auto scale lr fucntion

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

* Update

Co-authored-by: wangjiabao1.vendor <wangjiabao@pjlab.org.cn>
2022-06-06 22:27:15 +08:00
Jiazhen Wang 65bc95036c
[Enhance] Support Custom Runner (#258)
* support custom runner

* change build_runner_from_cfg

* refine docstring

* refine docstring
2022-06-06 14:33:32 +08:00
RangiLyu 70c4ea191f
[Refactor]: Modify val_interval and val_begin to be the attributes of TrainLoop. (#274)
* Modify val_interval and val_begin to be the attributes of TrainLoop.

* update doc

* fix lint

* type hint
2022-06-06 11:13:25 +08:00
Alex Yang 13606040ac
[Feat]:Add base module (#277) 2022-06-06 10:51:23 +08:00
Mashiro 80a46c4848
[Fix] fix build optimizer wrapper without type (#272)
* fix build optimizer wrapper without type

* refine logic

* fix as comment

* fix optim_wrapper config error in docstring and unit test

* refine docstring of build_optim_wrapper
2022-06-05 22:35:16 +08:00
Mashiro 3e3866c1b9
[Feature] Add optimizer wrapper (#265)
* Support multiple optimizers

* minor refinement

* improve unit tests

* minor fix

* Update unit tests for resuming or saving ckpt for multiple optimizers

* refine docstring

* refine docstring

* fix typo

* update docstring

* refactor the logic to build multiple optimizers

* resolve comments

* ParamSchedulers spports multiple optimizers

* add optimizer_wrapper

* fix comment and docstirng

* fix unit test

* add unit test

* refine docstring

* RuntimeInfoHook supports printing multi learning rates

* resolve comments

* add optimizer_wrapper

* fix mypy

* fix lint

* fix OptimizerWrapperDict docstring and add unit test

* rename OptimizerWrapper to OptimWrapper, OptimWrapperDict inherit OptimWrapper, and fix as comment

* Fix AmpOptimizerWrapper

* rename build_optmizer_wrapper to build_optim_wrapper

* refine optimizer wrapper

* fix AmpOptimWrapper.step, docstring

* resolve confict

* rename DefaultOptimConstructor

* fix as comment

* rename clig grad auguments

* refactor optim_wrapper config

* fix docstring of DefaultOptimWrapperConstructor

fix docstring of DefaultOptimWrapperConstructor

* add get_lr method to OptimWrapper and OptimWrapperDict

* skip some amp unit test

* fix unit test

* fix get_lr, get_momentum docstring

* refactor get_lr, get_momentum, fix as comment

* fix error message

Co-authored-by: zhouzaida <zhouzaida@163.com>
2022-06-01 18:04:38 +08:00
Zaida Zhou f1da9a1d7f
[Feature] Support multiple optimizers (#235)
* Support multiple optimizers

* minor refinement

* improve unit tests

* minor fix

* Update unit tests for resuming or saving ckpt for multiple optimizers

* refine docstring

* refine docstring

* fix typo

* update docstring

* refactor the logic to build multiple optimizers

* resolve comments

* ParamSchedulers spports multiple optimizers

* refine docstring

* RuntimeInfoHook supports printing multi learning rates

* resolve comments

* fix typo
2022-05-31 16:54:39 +08:00
Jiazhen Wang f2190de787
[Enhance] Improve Exception in call_hook (#247)
* improve exception in call_hook

* refine unit test

* add test_call_hook

* refine

* update docstring and ut
2022-05-31 11:34:30 +08:00
jbwang1997 38b22d9e68
[Enhance] Enhance error report when a module has been registered in registery. (#264)
* Update

* Add unittest
2022-05-31 11:31:04 +08:00
RangiLyu 172b9ded4a
[Fix] Fix ema state dict swapping in EMAHook and torch1.5 ut. (#266)
* [Fix] Fix ema state dict swapping in EMAHook.

* fix pt1.5 ut

* add more comments
2022-05-30 16:51:06 +08:00
Jingwei Zhang 40daf46a45
Support validation only after some epoch/iteration in ValLoop (#257)
* add the epoch/iter that begins validating

* fix lint

* add property and fix unit test

* minor changes

* fix typos and add unit test

* add unit test about begin

* fix docstring
2022-05-27 15:10:12 +08:00
RangiLyu 4705e1fe3d
[Enhance] Add RuntimeInfoHook to update runtime information. (#254)
* [Enhance] Add RuntimeInfoHook to update runtime information.

* move lr to runtime info

* docstring

* resolve comments

* update ut and doc
2022-05-26 14:35:37 +08:00
Jiazhen Wang 4cbbbc0c31
[Enhance] Refine sync random seed (#256)
* refine sync random seed

* cancel seed param in batch-sampler
2022-05-25 19:18:03 +08:00
Haian Huang(深度眸) c197bdf359
[Feature] Profiling tools (#241)
* Add profiling tools

* fix docstr

* fix docstr

* update

* fix bug

* update

* update

* fix error

* fix mypy

* uodate

* merge main

* fix UT
2022-05-25 10:55:07 +08:00
Haian Huang(深度眸) 8d3bd4dfef
Move get_max_cuda_memory and set_multi_processing to public function (#250)
* move get_max_cuda_memory and set_multi_processing to a public function

* fix lint

* fix lint

* fix lint

* delete _set_multi_processing

* fix error

* rename
2022-05-24 19:36:55 +08:00
Jiazhen Wang a976257ca9
[Enhance] Support Custom LogProcessor (#251)
* support custom log processor

* supplementary docs

* format code
2022-05-24 17:17:35 +08:00
RangiLyu 11688507ba
[Fix] Fix some bugs in hooks and runner. (#242)
* [Fix] Fix some bugs in hooks and runner.

* fix markdown

* fix latex formula

* resolve comments
2022-05-20 17:18:24 +08:00
RangiLyu 0279ac2e8d
[Feature] Support EMA and SWA. (#239)
* [Feature] Support EMA and SWA.

* add ema hook

* add avg model ut

* add more unit tests

* resolve comments

* fix warmup ema

* rename

* fix comments

* add assert

* fix typehint

* add comments
2022-05-19 18:53:04 +08:00
Zaida Zhou 86ffc19c9c
Add pyupgrade pre-commit hook (#232)
* Add pyupgrade pre-commit hook

* fix ut

* remove comments
2022-05-19 17:56:31 +08:00
Zaida Zhou 98c85529b1
[Refactor] Replace torch distributed with mmengine dist module (#196)
* [Fix] Replace torch distributed with mmengine dist module

* minor refinement

* move all_reduce_params to dist.py

* add unit tests

* update unit tests

* fix test_logger.py

* add examples
2022-05-19 17:40:01 +08:00
RangiLyu e37f1f905b
[Refactor] Make loop-related attributes to be runner's properties. (#236)
* [Enhance] Make loop related attributes to be runner's properties.

* move iter and epoch to loop

* resolve comments
2022-05-18 22:35:10 +08:00
Mashiro cc8a6b86e1
[Fix] Fix BaseDataset: join prefix in parse_data_info (#226)
* implement parse_data_info

* add unit test

* fix join prefix of ann_file

* fix docstring
2022-05-17 20:53:13 +08:00
Mashiro fd962437e9
[Fix] Support Runner dump cfg without filename (#228)
* fix runner dump cfg

* convert dict cfg to Config
2022-05-17 17:32:10 +08:00
liukuikun 689837d2b8
[Enhancement] add pixel data and label data (#224) 2022-05-13 18:23:25 +08:00
RangiLyu 1912660db9
[Feature] Support convert epoch-based schedulers to iter-based. (#221)
* [Feature] Support convert epoch-based schedulers to iter-based.

* Support convert and refactor LR and Momentum to mixin.

* Add unit tests

* fix args and add runner ut

* resolve comments
2022-05-10 15:17:51 +08:00
Mashiro 9087956f70
[Enhance] Update config with newest mmcv and show custom imports error explicitly (#192)
* add import error information

* Update config with newest mmcv

* add empty line to test config
2022-05-06 22:59:54 +08:00
Mashiro 859f4d1580
[Fix] Fix some code and docstring in LogProcessor and LoggerHook (#213)
* fix logger hook

* fix to record multi-level logs, log train/a/b to a/b

* fix loggerhook json path

* fix uunit test

* change runner.train_dataloader to runner.train_loop.dataloader

* clean debug code

* refie comments and docstring

* fix unit test
2022-05-06 14:33:33 +08:00
liukuikun 5c5c03e648
[Enchance] cat empty instancedata, support torch.bool for more type (#209)
* refactor instancedata

* fix docs

* fix comment
2022-05-06 14:00:51 +08:00
Zaida Zhou 16058fdb18
[Fix] Fix RecursionError when key is invalid (#220)
* [Fix] Fix RecursionError when key is invalid

* refine comments
2022-05-05 20:09:03 +08:00
Mashiro 5007825619
[Fix] change CheckPointHook before_run to before train (#214)
* change CheckPointHook before_run to before train

* using tmp_path in each checkpointhook test case
2022-05-05 20:08:07 +08:00
RangiLyu 4742544b25
[Feature] Support collect all registered module info. (#193)
* [Feature] Support collect all registered module info.

* update

* update

* add unit tests

* add to runner

* resolve comments
2022-05-05 11:59:51 +08:00
RangiLyu 29342c4eb2
[Fix] Fix to_dict in BaseDataElement and add metainfo property. (#215)
* [Fix] Fix to_dict in BaseDataElement and add metainfo property.

* Update mmengine/data/base_data_element.py

Co-authored-by: liukuikun <24622904+Harold-lkk@users.noreply.github.com>

Co-authored-by: liukuikun <24622904+Harold-lkk@users.noreply.github.com>
2022-04-29 10:19:19 +08:00
Mashiro 05fd5e81a3
[Enhance] Refine MMLogger and change save dir of MMLogger and Visualizer (#205)
* MMLogger can call get_current_instance without get_instance, change log path

* fix docstring

* fix docstring and update UT

* Fix runner

* fix docstring and lint

* fix ut below python3.8

* resolve circle import
2022-04-27 20:47:14 +08:00
Mashiro db2b45b7ac
[Fix] Fix the probable failure of the basedataset unit test (#210) 2022-04-27 19:48:28 +08:00
Zaida Zhou 17dbac1812
[Enhancement] Handle the device type of inputs in functions (#137)
* [Enhancement] Handle the device type of inputs in functions

* rename and move three fucntions to dist/utils.py

* minor refinement

* rename dist to torch_dist in utils.py

* update unit tests

* refine unit tests

* add unit tests

* fix unit tests

* replace Sequence with list and tuple

* rename get_backend_device to get_comm_device

* fix unit tests

* fix unit tests

* refactor and add more unit tests

* cast_data_device does not support set type
2022-04-27 19:46:13 +08:00
Zaida Zhou 661e759063
[Fix] param_scheduler can not None when training models (#208)
* [Fix] param_scheduler can not None when training models

* update unit tests

* fix unit tests

* refactor ParamSchedulerHook

* refactor unit tests

* param_schedulers can be an empty list
2022-04-27 19:45:27 +08:00
Haian Huang(深度眸) 6996bdc892
Update visualizer code (#184)
* update vis backend

* update vis backend

* update vis backend

* update visualizer

* update visualizer

* update visualizer

* update featmap

* update featmap

* update visualizer and unitest

* add draw points unitest and refactor vis_backend

* fix typo and close unitest

* fix comment

* add docstring

* fix comment

* add master only

* fix comment

Co-authored-by: liukuikun <641417025@qq.com>
2022-04-27 19:44:40 +08:00
Mashiro 452b3656a1
[Fix] Fix dump config without self.filename (#202)
* fix config

* add docstring and unit test

* update tutorial

* update tutorial

* fix markdown format

* fix markdown format
2022-04-27 19:43:12 +08:00
Mashiro 4dcbd269aa
[Enhance] register dataset wrapper (#185)
* register dataset wrapper

* fix as comment
2022-04-26 16:52:13 +08:00
Wenwei Zhang 96f3d97fc4
Try to fix lint issue (#199)
* try to fix lint

* upgrade yapf version

* use another way to bypass yapf

* update docstring
2022-04-26 13:53:00 +08:00