Commit Graph

687 Commits (5b9a1544b0ae8ce3980c98e5844e5d762637e1f0)
 

Author SHA1 Message Date
luomaoling 5b9a1544b0
[Feature] Add torch_npu optimizer (#1079) 2023-04-21 15:15:10 +08:00
Mashiro f1aca8e307
[Fix] Failed to remove the previous best checkpoints (#1086)
* [Fix] Only reserve one best checkpoint

* [Fix] Only reserve one best checkpoint

* Fix unit test

* shutdown logging

* clean the save_checkpoint logic
2023-04-20 21:28:56 +08:00
Luo Yihang 6ebb6f838a
[Fix] Call SyncBufferHook before validation in IterBasedTrainLoop (#982)
* [Fix] Call SyncBufferHook before validation in IterBasedTrainLoop

* Add before_val_epoch in SyncBuffersHook

* Fix white space format

* Add comments for SyncBuffersHook

* Add comments for SyncBuffersHook

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Add comments for SyncBuffersHook

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Fix white space format

* Add before_test_epoch

* Remove before_test_epoch

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-04-20 16:00:42 +08:00
Mashiro 0e5f9da68b
[Fix] Fix merge ci (#1089)
* fix main ci

* Fix

* Fix

* Fix
2023-04-20 15:48:08 +08:00
Mashiro be347df770
[Fix] KeyError is thrown in _collect_scalars when log_with_hierarchy is True (#1085)
* Fix log processor

* Fix custom key
2023-04-20 10:52:32 +08:00
Mashiro a7d4b7c742
[Enhance] Support configuring directory used to synchronize results in BaseMetric (#1074)
* [Enhance] Support configuring synchronize directory for BaseMetric

* Raise error if tmpdir is not an shared dirctory for ann ranks

* Raise error if tmpdir is not an shared dirctory for ann ranks

* Update mmengine/evaluator/metric.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* refine

* Update mmengine/evaluator/metric.py

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-04-14 19:15:04 +08:00
黄启元 60b4c199fc
[Feature] Support MLU backend (#1075)
* support mlu device

* support mlu device

* fix lint error

* fix lint error builder.py

* fix lint error in amp.py

* fix lint errors

* fix data type in instance_data.py
2023-04-14 19:06:19 +08:00
t chan f22002ec08
Fix] Fix three typos in runner (#1068) 2023-04-12 14:09:55 +08:00
LEFTeyes 6b366f236c
[Docs] Translate tutorials/evaluation.md (#1053)
* [Docs] Translate tutorials/evaluation.md
2023-04-12 12:53:14 +08:00
SheffieldCao 7877f8515f
[Docs] Translate the migration/hook.md to English (#1054)
* [Docs] add en version doc migration/hook.md

* [Docs] modifications to en/migration/hook.md
2023-04-12 11:03:26 +08:00
Junwei Zheng d41906fa15
[Fix] Fix publish multiple checkpoints when using multiple GPUs (#1059) (#1070) 2023-04-12 10:38:48 +08:00
Zaida Zhou 9207e84aa0
[Docs] Introduce the use of wandb and tensorboard (#912)
* [Docs] Introduce the use of wandb and tensorboard

* fix link

* Update docs/en/common_usage/visualize_training_log.md
2023-04-11 12:31:05 +08:00
sjiang95 5da24ed102
[Enhance] Complement type hint of get_model_complexity_info() (#1064)
* Complement type hint of get_model_complexity_info()

The type of `inputs` should be one of `torch.Tensor`,
`tuple[torch.Tensor, ...]` and `None`.

Signed-off-by: Shengjiang QUAN <qsj287068067@126.com>

* Update print_helper.py

---------

Signed-off-by: Shengjiang QUAN <qsj287068067@126.com>
Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-04-10 19:57:54 +08:00
Qian Zhao b2ad2210b5
[Feature] Support registering partial functions and more (#595)
* support registering partial functions

* Update mmengine/registry/build_functions.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/registry/registry.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Revert unit test and refine

* add current logger and set log level

---------

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>
Co-authored-by: HAOCHENYE <21724054@zju.edu.cn>
2023-04-10 19:42:04 +08:00
sjiang95 f76218a489
[Enhance] Make the parameters of get_model_complexity_info() friendly (#1056)
* print_helper: optimize inputs of get_model_complexity_info

Signed-off-by: Shengjiang QUAN <qsj287068067@126.com>

* directly throw error

When "input_shape" and "inputs" are both `None` or both set,
throw ValueError.

Signed-off-by: Shengjiang QUAN <qsj287068067@126.com>

---------

Signed-off-by: Shengjiang QUAN <qsj287068067@126.com>
2023-04-10 18:45:55 +08:00
shufan wu 5e1ed7aaf0
[Enhance] Allow users to customize worker_init_fn of Dataloader (#1038)
* customize worker init fn function

* add assert

* narrow worker_init_fn type
2023-04-10 17:32:36 +08:00
Mashiro eea2c278f4
[Enhance] Remove useless variable declaration (#1052) 2023-04-10 17:27:06 +08:00
Yinlei Sun 1c67f9eb22
[Enhancement] Support BoolTensor and LongTensor on Ascend NPU (#1011) 2023-04-10 16:31:31 +08:00
sung-hwa kim 8bf1ecad38
[Feature] Add vis backend for MLflow. (#878)
* add vis mlflow backend
2023-04-07 16:35:41 +08:00
Mashiro 5762b28847
[Refactor] Refactor logger hook unit tests (#797)
* Enhance config

* add unit test data

* reafactor unittest of loggerhook

* fix rebase error

* Fix permission error in windows

* Fix CI

* Fix windows ci

* Fix windows ci

* Fix windows ci

* Fix windows CI

* Apply suggestions from code review

Co-authored-by: Qian Zhao <112053249+C1rN09@users.noreply.github.com>

* clean the code

* Refine as comment

* Refine error rasing

* Update mmengine/hooks/logger_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* replace assert_called_with with assert_has_calls

* Fix as comment

* Do not remove filehandler and fix unit test

---------

Co-authored-by: Qian Zhao <112053249+C1rN09@users.noreply.github.com>
Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-04-07 16:20:38 +08:00
Mashiro f291212b01
Bump version to v0.7.2 (#1051)
* Bump version to v0.7.2

* minor refine

* Update docs/en/notes/changelog.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-04-06 17:12:00 +08:00
Mashiro 2dbc8ed253
[Refactor] Refactor checkpointhook unit tests (#789)
* Enhance config

* add unit test data

* Refacotr unitest of checkpointhook

* add comments

* Fix unit test

* remove _get_metric_scope

* tmp save

* Revert "remove _get_metric_scope"

This reverts commit eeb7a8c5ed.

* Revert "Revert "remove _get_metric_scope""

This reverts commit 5398255f6f.

* Revert "tmp save"

This reverts commit cdc9919be8.

* clean the code

* Fix ut

* minor fix

* use str.replace
2023-04-06 10:55:16 +08:00
Mashiro dc931fd2c0
[Fix] Initialize nested modules in ddp which define 'init_weights' method (#1045) 2023-04-05 10:33:24 +08:00
BigDong fd84c210e5
[Fix] Update the ClassBalancedDataset logic to keep len(repeat_factors) = len(dataset) (#1048) 2023-04-04 14:27:27 +08:00
zhouhui 093068e4ff
[Enhancement] Align the evaluation result in log (#1034)
* align the evaluation result in log

* align the evaluation result in log

* align the evaluation result in log

* align the evaluation result in log

* fix test log_processor
2023-04-04 00:17:42 +08:00
Mashiro 0139e6eb15
[Enhance] Suppress registry warning (#1043) 2023-04-03 17:32:12 +08:00
Mashiro 4d7a29342c
Bump version to 0.7.1 (#1007)
* bump version to 0.7.1

bump version to 0.7.1

* update change log and readme

* Update docs/en/notes/changelog.md

* refine highlight

refine highlight

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-04-03 13:17:25 +08:00
Mashiro 83c4f3e643
[Enhance] Make sure the FileHandler still alive after `torch.compile` (#1021)
* [Enhance] Make sure the FileHandler still alive after

* Resume filter

* avoid bc

* Fix unit test

* clean the code

* revert changes and set mode from 'm' to 'a'

* mode to file_mode

* add comments

* refine comments

* Fix duplicated the
2023-03-30 17:41:26 +08:00
Zaida Zhou b3b1e11b4e
[Enhancement] Do not print config if it is empty (#1028) 2023-03-29 19:54:57 +08:00
Mashiro 26a7a92a13
[Enhance] Do no catch unnecessary exceptions in registry (#1010) 2023-03-29 16:15:42 +08:00
Mashiro 30c5996af2
Skip test_dist in Github Action (#1027) 2023-03-29 14:25:35 +08:00
Mashiro eb79d64af1
[Fix] Add PyTorch 2.0 CI and fix unit tests (#1026)
* [Enhance] Make sure the FileHandler still alive after

* minor refine

* minor refine

* refine unit test

* update CI

* update CI

* Fix CI

* fix build_windows

* fix build_windows

* fix build_windows

* fix build_windows

* debug

* debug

* debug

* debug

* debug

* debug

* debug

* test windows CI

* Fix

* Debug

* Fix unit test

* Fix unit test

* Fix CI

* update image version

* update action/checkout and action/setup-python

* add condition to skip test compile

* [Fix] Update CI and fix unit test

* check compiling by attempting compilation

* check compiling by attempting compilation

* check compiling by attempting compilation

* use windows-2022 in runs on

* Apply suggestions from code review

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* update yml

* remove unnecessary assert

* assert grad is None according to the PyTorch version

* Fix code

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-03-29 13:09:23 +08:00
KerwinKai 5b35c5b6ad
[Feature] Publish models after training if published_keys is set in CheckpointHook (#987)
* add publish keys in checkpointhook and update hook.md file

* Update checkpoint_hook.py

To avoid `mypy` warning `mmengine/hooks/checkpoint_hook.py:358: error: Unsupported right operand type for in ("Optional[List[str]]") Found 1 error in 1 file (checked 224 source files)`

* Update hook.md

Try to avoid trim trailing whitespace waring in hook.md

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update checkpoint_hook.py

* Update docs/en/tutorials/hook.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update hook.md

add 自动发布最好的和最后的权重

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update checkpoint_hook.py

add condition when the best checkpoints more than 1.

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update checkpoint_hook.py

delete re judge

* Update checkpoint_hook.py

* Update checkpoint_hook.py

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update checkpoint_hook.py

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update checkpoint_hook.py

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Add Test for publish model

* Update checkpoint_hook.py

* Update test_checkpoint_hook.py

* Fix file to pass pre-commit check

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Fix mypy warning

* rm not necessary line in checkpoint_hook.py

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* rm unnecessary messages add to message_hub

* Update mmengine/hooks/checkpoint_hook.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update docs/zh_cn/tutorials/hook.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update docs/zh_cn/tutorials/hook.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* update checkpoint hook and hook.md file

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update mmengine/hooks/checkpoint_hook.py

---------

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>
Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-03-29 10:25:14 +08:00
Mashiro cd3fc65c76
[Docs] Add more social networking links (#1024)
* [Docs] Add more social networking links

* change discord channel to mmengine
2023-03-28 19:39:44 +08:00
Mashiro d174952a3e
[Fix] Fix there is no space between data_time and metric (#1025) 2023-03-28 19:20:56 +08:00
Evan f356b3c207
[Docs] Minor fixes in docs to remove or replace unicode chars with ascii chars (#1018)
* Remove CN comments in EN docs code
* Replace some full-width unicode chars with half-width ascii chars
2023-03-26 13:59:26 +08:00
Qian Zhao cbb671403f
[Refactor] Refactor the `get_device` and `is_npu_available` (#1004)
* refactor get_device and is_npu_available

* Update mmengine/device/utils.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-03-19 15:55:27 +08:00
LEFTeyes ff27b723db
[Enhancement] Unify the use of print_log and logger (#997) 2023-03-17 12:31:51 +08:00
Ma Zerun 2955ddb01b
[Enhance] Add `mmpretrain` to the `MODULE2PACKAGE` (#1002) 2023-03-17 11:03:44 +08:00
Mashiro ef4c68deb7
Bump version to v0.7.0 (#999)
* Bump version to v0.7.0

* Minor refine

* Fix as comment

* Add docs link

* update release date
2023-03-16 21:03:09 +08:00
liukuikun 67ce04d918
[Fix] Fix pytorch2.0 does not support sys._getframe used to remove private keys (#979)
* [Fix] fix pytorch2.0 not support sys._getframe to remove private key

* Update mmengine/structures/base_data_element.py

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-03-15 00:37:12 +08:00
Mashiro ad33a7d0e5
[Fix] Fix inferencer gets wrong configs path (#996)
* [Fix] Fix inferencer get wrong configs path

* Update CI

* Fix indent

* Fix CI arguments

* gpu test in CI

gpu test in CI

* require lint

* Adjust pytorch version and cuda version

* Fix docker

* Fix docker syntax

* Use bach -c

* Use bach -c

* Replace is_installed with is_imported

* Fix

* Fix PYTHONPATH
2023-03-14 18:28:33 +08:00
Zaida Zhou 789330e2ac
Add citation in README (#995) 2023-03-13 19:09:45 +08:00
luomaoling 8177ef2aef
[Fix] Fix AMP in Ascend and support using NPUJITCompile environment (#994)
* add npu device support

* add npu device support
2023-03-13 19:08:50 +08:00
Zaida Zhou 60872c38d4
fix lint (#993) 2023-03-13 14:46:59 +08:00
Mashiro 395ebf8d82
[Enhancement] Support dumping logs of different ranks (#968)
* Add hostname

* Update mmengine/logging/logger.py

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Fix lint

* support record device id

* Fix unit test

* Clean the code

* Fix as comment

* Fix as comment

* Fix unit test

* Update doc

* Fix unit test

* Adjust sequence

* Replace \ with ()

* remove unnecessary ()

* does not change filename in single gpu training

* Fix ci

* fix docs

* Fix as comment

---------

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-03-13 14:35:31 +08:00
Yijie Zheng 330985d6c1
[Docs] Translate "Model Complexity Analysis" to Chinese (#969)
* [Doc] Translate model complexity analysis into Chinese.

* [Doc] Translate model complexity analysis into Chinese.

* [Docs] fix the description of the interface

* update  introduction

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update description of FLOPs

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update activation

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update model description

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Beautify code style

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Modify examples

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Upadate output description

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Update docs/zh_cn/advanced_tutorials/model_analysis.md

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

* Replace FLOPs with flop; fix typo

* Fix typo

* fix lint error

* Update docs/zh_cn/advanced_tutorials/model_analysis.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update docs/zh_cn/advanced_tutorials/model_analysis.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update docs/zh_cn/advanced_tutorials/model_analysis.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update docs/zh_cn/advanced_tutorials/model_analysis.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update docs/zh_cn/advanced_tutorials/model_analysis.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update docs/zh_cn/advanced_tutorials/model_analysis.md

Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>

* Update model_analysis.md

* Update model_analysis.md

* Apply suggestions from code review

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>

---------

Co-authored-by: Mashiro <57566630+HAOCHENYE@users.noreply.github.com>
Co-authored-by: Zaida Zhou <58739961+zhouzaida@users.noreply.github.com>
2023-03-13 14:31:06 +08:00
Mashiro 8063d2cce7
[Enhancement] Support writing data to `vis_backend` with prefix (#972)
* Log with prefix

* Fix test of loggerhook

* minor refine

* minor refine

* Fix unit test

* clean the code

* deepcopy in method

* replace regex

* Fix as comment

* Enhance readable

* rename reserve_prefix to remove_prefix

* Fix as comment

* Refine unit test

* Adjust sequence

* clean the code

* clean the code

* revert renaming reserve prefix

* Count the dataloader length in _get_dataloader_size
2023-03-13 13:07:37 +08:00
Qian Zhao 0d25625ba2
[Feature] Support torch.compile since PyTorch2.0 (#976)
* enable compile configurations to support torch.compile in Runner

* enable compilation in train, val and test

* fix as comments

* add docstring to illustrate usage

* minor refine error message

* add unittests

* fix ut skip

* add logging message to inform users

* compile `train_step`, `val_step`, `test_step` instead

* fix as comments

* revert to compile `train_step` only due to pt2 issue

* add documentation about torch.compile
2023-03-12 18:26:43 +08:00
Mashiro 6ea23a2f71
[Fix] Fix duplicated warning (#992)
* [Fix] Fix repeated warning

* Add type hint

* Fix unit test

* Rename recorder_dict to seen

* Fix as comment
2023-03-10 19:27:36 +08:00