Commit Graph

30 Commits (fd6eab697d9f84dd232de60f6311253b49be29bc)

Author SHA1 Message Date
Zaida Zhou 32e09f4933
[Feature] Upload checkpoints and logs to ceph (#1375)
* [Feature] Choose storage backend by the prefix of filepath

* refactor FileClient and add unittest

* support loading from different backends

* polish docstring

* fix unittet

* rename attribute str_like_obj to is_str_like_obj

* [Docs] Upload checkpoint to petrel oss

* add infer_client method

* Support uploading checkpoint to petrel oss

* add check_exist method

* refactor CheckpointHook

* support uploading logs to ceph

* rename var client to file_client

* polish docstring

* enhance load_from_ceph

* refactor load_from_ceph

* refactor TextLoggerHook

* change the meaning of out_dir argument

* fix test_checkpoint_hook.py

* add join_paths method

* remove join_paths and add _format_path

* enhance unittest

* refactor unittest

* add a unittest for EvalHook when file backend is petrel

* singleton pattern

* fix test_clientio.py

* deprecate CephBackend

* add warning in load_from_ceph

* fix type of out_suffix

* enhance docstring

* refactor unittest for petrel

* refactor unittest for disk backend

* update io.md

* add concat_paths method

* fix CI

* mock check_exist

* improve docstring

* improve docstring

* improve docstring

* improve docstring

* add isdir and copyfile for file backend

* delete copyfile and add get_local_path

* remove isdir method of petrel

* fix typo

* rename check_exists to exists

* refactor code and polish docstring

* fix windows ci

* add comment and polish docstring

* polish docstring

* polish docstring

* rename _path_mapping to _map_path

* polish docstring and fix typo

* refactor get_local_path

* add list_dir_or_file for FileClient

* add list_dir_or_file for PetrelBackend

* fix windows ci

* Add return docstring

* polish docstring

* fix typo

* fix typo

* fix typo

* fix error when mocking PetrelBackend

* deprecate the conversion from Path to str

* add docs for loading checkpoints with FileClient

* rename keep_log to keep_local

* refactor map_path

* add _ensure_methods to ensure methods have been implemented

* fix list_dir_or_file

* rename _ensure_method_implemented to has_method

* refactor

* polish information

* format information
2021-10-24 14:26:52 +08:00
Wenwei Zhang 979a355d2a
[Feature] Add windows CI (#1023)
* add windows CI

* clean versions

* only allow pt1.7 on windows

* fix windows install issue

* add win cpu

* fix win command

* clean unnecessary command

* resolve turbojpeg & tempfile on win

* replace os.readlink with os.path.realpath

* fix windows ci

* close file before removing it

* fix windows ci

* fix symlink on windows

* fix windows ci

* fix windows ci

* fix windows ci

* fix windows ci

* fix windows ci

* fix windows ci

* fix windows ci

* fix windows ci

* fix windows ci

* modify according to comment

Co-authored-by: zhouzaida <zhouzaida@163.com>
2021-08-24 10:07:31 +08:00
Ma Zerun 357b0dfb36
[WIP] continue PR #784 (#1221)
* Add gradient cumulative optimizer

fixes #190

* Update optimizer.py

* Update optimizer.py

* fix loss scale improperly in last equivalent_iter

* Add `GradientCumulativeOptimizerHook` in `__init__.py`.

* Add docstring of `GradientCumulativeOptimizerHook`.

* Add type check, BN warning and resume warning. And fix typo, lint the
code.

* Add unit test

* Update docstring example.

* Change GradientCumulativeOptimizerHook `__init__` arguments.

* Add GradientCumulativeOptimzierHook unit tests with IterBasedRunner.

* Add GradientCumulativeFp16OptimizerHook.

* Add unit tests of GradientCumulativeFp16OptimizerHook

* Use '!=' instead of '>' to determine resume

Co-authored-by: Zhiyuan Chen <this@zyc.ai>
2021-08-23 11:18:36 +08:00
Gu Wang 025ee9f5e6
[Feature] Add flat cosine lr updater (#1066)
* add flat cosine lr updater

* add test

* add doc

* update doc

* reformat

* update unittest

* update test flat cos

* remove momentum hook test

* update test

* change assert to ValueError

* fix unittest

* add by_epoch=True unittest

* change to start_percent

* change to start_percent in test
2021-07-20 17:16:50 +08:00
Ma Zerun 1b15f02220
support print hooks before running. (#1123)
* support print using hooks before running.

* Support to print hook trigger stages.

* Print stage-wise hook infos. And make `stages` as class attribute of
`Hook`

* Add util function `is_method_overriden` and use it in
`Hook.get_trigger_stages`.

* Add unit tests.

* Move `is_method_overriden` to `mmcv/utils/misc.py`

* Improve hook info text.

* Add base_class argument type assertion, and fix some typos.

* Remove `get_trigger_stages` to `get_triggered_stages`

* Use f-string.
2021-06-25 20:49:24 +08:00
Ma Zerun 6fe3722510
Refine default hooks and custom hooks priority rank. (#1120)
* Refine default hooks and custom hooks priority rank.

* Add unit tests for custom hooks with string priority.

* Use priority `ABOVE_NORMAL` and `BELOW_NORMAL` instead of `HIGHER` and
`LOWER`.

And add unit tests for custom hook with the same priority as
default hooks.
2021-06-25 15:07:32 +08:00
David de la Iglesia Castro bdd7022143
Add DvcliveLoggerHook (#1075)
* Add dvclive logger hook

* Move docstring to class

* docstring updates
2021-06-08 22:20:33 +08:00
fcakyon e9f2a02b47
add neptune.ai logger hook (#1025)
* add neptune.ai logger hook

* add docstring

* move docstring

* update docstr

* update before_run
2021-05-23 15:28:21 +08:00
Ma Zerun 15bcaa9c05
Add custom hook by config file (#970)
* Assign different priority to default hooks, and add custom hook register in base runner.

* Add custom hook register in example train file

* Add unittest of custom hook

* Code format
2021-05-13 20:29:17 +08:00
Ziyi Wu c77e95a65f
[Enhance] Add step momentum updater to support MMDet3D (#1011)
* add StepMomentumUpdaterHook

* add unit test

* fix typos

* refactor step updater

* replace stage with exp

* fix linting error

* use all() operation
2021-05-11 20:06:04 +08:00
gengenkai 1a5bf76237
[Enhancement] LR triangular (#1001)
* [Fix] lr add triangular

* [Fix] lr_triangular

* lr add docstring

* lr add docstring

* add unittest of triangular lr update
2021-05-11 13:25:43 +08:00
Ziyi Wu 04346fc7d0
[Feature] Support clipping min_lr in StepLrUpdaterHook to help MMDet3D (#995)
* support clipping min_lr in StepLrUpdaterHook

* add docstring for StepLrUpdaterHook

* fix small bugs

* add unit test for StepLrUpdaterHook

* fix linting error
2021-04-27 20:53:29 +08:00
Miao Zheng 89efc607ef
[Fix] OneCycleLrUpdaterHook adds total_steps (#929)
* [Fix] OneCycleLrUpdaterHook interface

* revise according to comments

* revise according to comments

* add test

* fix lint

* revise according to comments

* minors

* add pytest param

* fix lint

* ci
2021-04-10 21:19:45 +08:00
zhouzaida d525cfde10
[Fix] Fix bug of lr updater hook (#907)
* [Fix] fix bug of lr update hook

* [Fix] fix bug of lr update hook

* [Fix] Fix bug of lr updater hook
2021-04-09 10:13:32 +08:00
zhouzaida f0c43fdc32
[Feature] Add OneCycleLrUpdaterHook (#906)
* [Feature] Add OneCycleLrUpdaterHook

* fix docstring

* fix docstring

* Remove redundant code
2021-04-02 09:44:18 +08:00
Ycr e076c8b042
[Feature] Support revise_keys in load_checkpoint(). (#829)
* Simplified the code.

* Improved chkpt compatibility.

* One may modify the checkpoint via adding keywords.

* Tiny.

* Following reviewer's suggestion.

* Added unit_test.

* Fixed.

* Modify the state_dict  with  construction.

* Added test.

* Modified。

* Mimimalised the modification.

* Added the docstring.

* Format.

* Improved.

* Tiny.

* Temp file.

* Added assertion.

* Doc string.

* Fixed.
2021-03-03 10:59:11 +08:00
Jerry Jiarui XU 987cb58341
[Enhance] Refactor logger (#659)
* [Enhance] Refactor logger

* fixed test

* make commit optional

* remove debug info

* fixed test
2020-11-23 10:50:18 +08:00
Jerry Jiarui XU bb06f354d5
[Feature] Add hook msg for checkpoint hook (#635)
* [Feature] Add hook msg for ckpt_hook

* [Feature] Add hook msg for ckpt_hook

* add test
2020-11-06 19:56:50 +08:00
Kuro Latency bcf85026c3
Update lr_updater.py (#574)
* Update lr_updater.py

since epoch/iteration in  runner starts with 0, we shouldn't leave the latter iteration to former (12th epoch for example, with first period equal to 12) period.

* Update lr_updater.py

* Update test_hooks.py
2020-11-05 22:20:37 +08:00
David de la Iglesia Castro 2e6c8ec803
Refactor logger hooks (#605)
* Refactor tags for consistency

* Fix missing runner

* Fix missing runner

* Fix missing runner

* Fix missing runner

* Fix momentum runner hook inner iter

* Fix tests

* pre-commit run
2020-10-28 12:43:50 +08:00
David de la Iglesia Castro 6b52e9b55f
Add runner builder (#570)
* Add build_runner

* Parametrize test_runner

* Add imports to runner __init__

* Refactor max_iters and max_epochs from run to init

* Add assertion error messages

* Add test_builder

* Make change retro-compatible

* Raise ValueError if max_epochs and max_iters
2020-09-25 10:25:29 +08:00
Wang Xinjiang eacaf475f7
fix some pavi logger hooks (#481)
* fix some pavi logger hooks

* fix unittest

* fix small bugs

* small change

* fix unittest

* Add EpochBasedRunner conditions

* Add session text

* fix small bug

* fetch runner mode from log buffer

* Add max_iter to pavi session text

* change yaml.dump to yamp.dump(yaml.load(mmcv.dump))

* Directly use by_epoch

* fix unittest

* add comments

* Use runner.epoch + 1 in pavi log

* fix runner.epoch issue for runner.mode=='val'

* fix runner.epoch issue for runner.mode=='val'

* Use abspath instead of realpath

* Add meta dump unittest

* small change

* Add comments
2020-08-16 01:20:08 +08:00
shilong 1830347f8b
Ema (#421)
* add ema hook

* add ema hook resume

* add ema hook test

* fix typo

* fix according to comment

* delete logger

* fix according to comment

* fix unitest

* fix typo

* fix according to comment

* change to resume_from

* typo

* fix isort
2020-07-30 22:06:19 +08:00
Wang Xinjiang d4da3daa7e
Syncbuf (#447)
* More robust sync buffer hook

* More robust sync buffer hook

* Reformat
2020-07-25 12:51:46 +08:00
Wang Xinjiang 66604e83de
Add syncbuffer hook (#443)
* reformat

* reformat

* Add register hook from cfg

* docstring

* change according to comments
2020-07-24 14:15:44 +08:00
Yawei Li 7730a79fcd
fix typo of annealing (#433) 2020-07-17 23:48:22 +08:00
Kai Chen 63b7aa31b6
Fix docstring formats (#383)
* update doc formats

* update docstring
2020-07-04 00:55:25 +08:00
Harry f28a7c7ed7
Add CosineRestartLrUpdaterHook (#319)
* feat: add CosineRestartLrUpdaterHook

* style: rename period to periods

* fix: bug in period 0

* feat: rename eta_min to min_lr and add min_lr_ratio

* docs: fix docstring of restart lr updater

* refactor: use annealing_cos

* docs: add docstring to annealing_cos

* feat: cosine restart lr update hook

* refactor: modify code order for unittest
2020-06-15 23:01:26 +08:00
Kai Chen 35ba152821
Add a BaseRunner and rename Runner to EpochBasedRunner (#290)
* add a BaseRunner and rename Runner to EpochBasedRunner

* fix the train/val step

* bug fix

* update unit tests

* fix unit tests

* raise an error if both batch_processor and train_step are set

* add a unit test
2020-06-02 22:23:21 +08:00
Kai Chen a338d43d78
Refactor unittests (#241)
* refactor unittests

* split test_video.py to two files
2020-04-26 22:54:27 +08:00