50 Commits

Author SHA1 Message Date
Victor Sonck
378bde4bba
ClearML experiment tracking integration (#8620)
* Add titles to matplotlib plots

* Add ClearML Experiment Tracking integration.

* Add ClearML Data Version Management automatic download when requested

* Add ClearML Hyperparameter Optimization

* ClearML save period integration

* Fix wandb breaking when used with ClearML dataset

* Fix wandb breaking when used with ClearML resume and dataset

* Add ClearML documentation

* fixed small bug in clearml integration that misreports epoch number

* Final ClearMl additions before refactor

* Add correct epoch reporting

* Add remote execution and autoscaling docs for ClearML integration

* Added images to clearml integration docs

* fixed logo alignment bug and added hpo screenshot clearml

* Fixed small epoch number bug in clearml integration

* Remove saved model flush clearml

* Cleanup clearml readme section

* Cleaned up clearml logger docstring

* Remove resume readme section clearml

* Clearml integration cleanup

* Updated ClearML documentation

* Added dark vs light icons ClearML Readme

* Clearml Readme styling

* Add better gifs

* Fixed gif file size

* Add better images in tutorial notebook

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Addressed comments in PR #8620

* Fixed circular import

* Fixed circular import

* Update tutorial.ipynb

* Update tutorial.ipynb

* Inline comment

* Restructured tutorial notebook

* Add correct ClearML link to README

* Update tutorial.ipynb

* Update general.py

* Update __init__.py

* Update __init__.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update __init__.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update __init__.py

* Update README.md

* Update __init__.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* spelling

* Update tutorial.ipynb

* notebook cutt.ly links

* Update README.md

* Update README.md

* cutt.ly links in tutorial

* Removed labels as they show up on last subplot only

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2022-08-05 20:50:49 +02:00
Glenn Jocher
9d8ed37df7
Rename utils/datasets.py > utils/dataloaders.py (#7799) 2022-05-13 14:34:16 +02:00
Glenn Jocher
b53917de8d
Remove tqdm.auto (#7599) 2022-04-26 15:00:01 -07:00
Ayush Chaurasia
4cac1cfc8f
Add --noplots flag to suppress figures and images logging (#7534)
* support nomedia

* support nomedia for validation

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update train.py

* Revert no plot evolve

evolve plots do not contain any images

* Revert plot_results

contains no media

* Update wandb_utils.py

* sync-bn cleanup

* Cleanup

* Rename nomedia -> noplots

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2022-04-22 15:04:08 -07:00
Zengyf-CVer
c264795f50
Add mdformat to precommit checks and update other version (#7529)
* Update .pre-commit-config.yaml

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update .pre-commit-config.yaml

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update CONTRIBUTING.md

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update README.md

* Update README.md

* Update README.md

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2022-04-22 13:36:27 -07:00
Glenn Jocher
446e6f563a
Rename 'MacOS' to 'macOS' (#7349) 2022-04-08 23:05:15 +02:00
Nick Martin
a88a81469a
Copy wandb param dict before training to avoid overwrites (#7317)
* Copy wandb param dict before training to avoid overwrites.

Copy the hyperparameter dict retrieved from wandb configuration before passing it to `train()`. Training overwrites parameters in the dictionary (eg scaling obj/box/cls gains), which causes the values reported in wandb to not match the input values. This is confusing as it makes it hard to reproduce a run, and also throws off wandb's Bayesian sweep algorithm.

* Cleanup

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2022-04-06 18:35:33 +02:00
Glenn Jocher
f735458987
Use tqdm.auto (#7311) 2022-04-06 12:20:24 +02:00
pre-commit-ci[bot]
7882950577
[pre-commit.ci] pre-commit suggestions (#7279)
* [pre-commit.ci] pre-commit suggestions

updates:
- [github.com/asottile/pyupgrade: v2.31.0 → v2.31.1](https://github.com/asottile/pyupgrade/compare/v2.31.0...v2.31.1)
- [github.com/pre-commit/mirrors-yapf: v0.31.0 → v0.32.0](https://github.com/pre-commit/mirrors-yapf/compare/v0.31.0...v0.32.0)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update yolo.py

* Update activations.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update activations.py

* Update tf.py

* Update tf.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2022-04-04 22:47:00 +02:00
Jirka Borovec
c3d5ac151e
precommit: yapf (#5494)
* precommit: yapf

* align isort

* fix

# Conflicts:
#	utils/plots.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update setup.cfg

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update setup.cfg

* Update setup.cfg

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update wandb_utils.py

* Update augmentations.py

* Update setup.cfg

* Update yolo.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update val.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* simplify colorstr

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* val run fix

* export.py last comma

* Update export.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update hubconf.py

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* PyTorch Hub tuple fix

* PyTorch Hub tuple fix2

* PyTorch Hub tuple fix3

* Update setup

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2022-03-31 16:52:34 +02:00
Glenn Jocher
4728840745
Update --cache disk deprecate *_npy/ dirs (#6876)
* Updates

* Updates

* Updates

* Updates

* Updates

* Updates

* Updates

* Updates

* Updates

* Updates

* Cleanup

* Cleanup
2022-03-06 16:16:17 +01:00
Louis Combaldieu
bcc92e2169
Update sweep.yaml (#6825)
* Update sweep.yaml

Changed focal loss gamma search range between 1 and 4

* Update sweep.yaml

lowered the min value to match default
2022-03-04 09:39:23 +01:00
imyhxy
ca0a00784d
Fixed wandb logger KeyError (#6637) 2022-02-14 17:08:36 +01:00
Ayush Chaurasia
96d8f86085
W&B: don't log media in evolve (#6617) 2022-02-12 13:04:50 +01:00
Ayush Chaurasia
51f7750aad
W&B: Improve resume stability (#6611)
* log best.pt metrics at train end

* update

* Update __init__.py

* flush callbacks when using evolve

* remember batch size on resuming

* Update train.py

* improve stability of resume

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2022-02-12 13:03:09 +01:00
or-toledano
cb2ad9f685
Fix ROOT / data when running W&B log_dataset() (#6606)
* Fix missing data folder when running log_dataset

* Use ROOT/'data'

* PEP8 whitespace
2022-02-10 16:17:47 +01:00
Jirka Borovec
cba4303d32
Fix 6 Flake8 issues (#6541)
* F541

* F821

* F841

* E741

* E302

* E722

* Apply suggestions from code review

* Update general.py

* Update datasets.py

* Update export.py

* Update plots.py

* Update plots.py

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2022-02-05 15:22:59 +01:00
Ayush Chaurasia
3f152e5807
Update wandb_utils.py (#5908) 2021-12-07 16:09:11 +01:00
Ayush Chaurasia
f2ca30a407
W&B: refactor W&B tables (#5737)
* update

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* reformat

* Single-line argparser argument

* Update README.md

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update README.md

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-11-25 17:19:21 +01:00
Jirka Borovec
eb51ffdcac
Prune unused imports (#5711)
* prune unused imports

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2021-11-19 13:32:53 +01:00
Ayush Chaurasia
540ef0dd30
W&B refactor, handle exceptions, CI example (#5618)
* handle exceptions| attempt CI

* update

* Pre-commit manual run

* yaml one-liner

* Update ci-testing.yml

* Comment W&B CI

Leave as example for future separate CI

* Update ci-testing.yml

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-11-14 13:26:53 +01:00
Jirka Borovec
0155548384
precommit: isort (#5493)
* precommit: isort

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update isort config

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update name

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-11-04 17:24:25 +01:00
Ayush Chaurasia
620b535f85
Update sweep.py (#5402) 2021-10-29 19:21:59 +02:00
Jirka Borovec
ed887b5976
Add pre-commit CI actions (#4982)
* define pre-commit

* add CI code

* configure

* apply pre-commit

* fstring

* apply MD

* pre-commit

* Update torch_utils.py

* Update print strings

* notes

* Cleanup code-format.yml

* Update setup.cfg

* Update .pre-commit-config.yaml

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-10-28 18:35:01 +02:00
Ayush Chaurasia
692be757b6
W&B: Media panel fix (#5317) 2021-10-25 12:06:12 +02:00
Glenn Jocher
13f7275555
Update/inplace ops (#5233)
* Clip Objects365 autodownload labels (#5214)

Fixes out of bounds labels that seem to affect ~10% of images in dataset.

* Inplace ops
2021-10-18 14:24:48 +02:00
Ayush Chaurasia
1172ac15fb
W&B: DDP fix (#5176)
* fix dpp with wandb disabled

* PyCharm reformat

* Fix

* fix

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-10-13 20:11:04 -07:00
Ayush Chaurasia
d0bfeb3980
W&B: fix DDP with wandb disabled (#5163)
* fix dpp with wandb disabled

* PyCharm reformat

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-10-13 16:17:15 -07:00
Glenn Jocher
ba99092304
Update W&B README (#5006) 2021-09-30 00:35:00 -07:00
Diego Montes
0c87478713
Update wandb_utils.py (#4953)
`is_valset_wandb_artifact` and `is_trainset_wandb_artifact` were referenced before assignment causing wandb to be unusable.
2021-09-28 12:22:31 -07:00
Ayush Chaurasia
59aae85a7e
W&B: Fix dataset check (#4879)
* evolve fix

* Enable login timeout

* fix pkg

* check rank

* don't relogin

* fix

* reformat
2021-09-21 20:02:32 +02:00
Ayush Chaurasia
9febea79de
W&B: Login only in master processes (#4866)
* evolve fix

* Enable login timeout

* fix pkg

* check rank

* don't relogin
2021-09-20 16:19:34 +02:00
Glenn Jocher
4d1a2ac87e
Update sys.path.append(str(ROOT)) (#4852)
* Update `sys.path.append(str(ROOT))`

* Cleanup
2021-09-18 15:02:08 +02:00
Ayush Chaurasia
302a1b0bb0
W&B: Enable login timeout (#4843)
* evolve fix

* Enable login timeout

* fix pkg
2021-09-17 18:59:34 +02:00
Glenn Jocher
c47be26f34
Replace path.absolute() with path.resolve() (#4763) 2021-09-11 22:46:33 +02:00
Josh Veitch-Michaelis
19e28e3bfe
Add callbacks to train function in W&B sweep (#4742)
* add callbacks to train function in wandb sweep

Fix following https://github.com/ultralytics/yolov5/pull/4688 which modified the function signature to `train`

* Cleanup

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-09-10 16:28:52 +02:00
Ayush Chaurasia
11e7c7b48d
W&B: Don't log models in evolve operation (#4611) 2021-08-30 17:07:20 +02:00
Ayush Chaurasia
7316b78e36
W&B: Refactor the wandb_utils.py file (#4496)
* Improve docstrings and run names

* default wandb login prompt with timeout

* return key

* Update api_key check logic

* Properly support zipped dataset feature

* update docstring

* Revert tuorial change

* extend changes to log_dataset

* add run name

* bug fix

* bug fix

* Update comment

* fix import check

* remove unused import

* Hardcore .yaml file extension

* reduce code

* Reformat using pycharm

* Remove redundant try catch

* More refactoring and bug fixes

* retry

* Reformat using pycharm

* respect LOGGERS include list

* Fix

* fix

* refactor constructor

* refactor

* refactor

* refactor

* PyCharm reformat

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-08-23 13:10:07 +02:00
Glenn Jocher
24bea5e4b7
Standardize headers and docstrings (#4417)
* Implement new headers

* Reformat 1

* Reformat 2

* Reformat 3 - math

* Reformat 4 - yaml
2021-08-14 21:17:51 +02:00
Glenn Jocher
63e09fdc48
Remove encoding='ascii' (#4413)
* Remove `encoding='ascii'`

* Reinstate `encoding='ascii'` in emojis()
2021-08-14 13:47:20 +02:00
Ayush Chaurasia
a4e4553040
W&B: Fix for 4360 (#4388)
* Improve docstrings and run names

* default wandb login prompt with timeout

* return key

* Update api_key check logic

* Properly support zipped dataset feature

* update docstring

* Revert tuorial change

* extend changes to log_dataset

* add run name

* bug fix

* bug fix

* Update comment

* fix import check

* remove unused import

* Hardcore .yaml file extension

* reduce code

* Reformat using pycharm

* Remove redundant try catch

* More refactoring and bug fixes

* retry

* Reformat using pycharm

* respect LOGGERS include list

* Fix

* fix

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-08-11 21:56:25 +02:00
Ayush Chaurasia
75d323dd8b
W&B: Add advanced features tutorial (#4384)
* Improve docstrings and run names

* default wandb login prompt with timeout

* return key

* Update api_key check logic

* Properly support zipped dataset feature

* update docstring

* Revert tuorial change

* extend changes to log_dataset

* add run name

* bug fix

* bug fix

* Update comment

* fix import check

* remove unused import

* Hardcore .yaml file extension

* reduce code

* Reformat using pycharm

* Remove redundant try catch

* More refactoring and bug fixes

* retry

* Reformat using pycharm

* respect LOGGERS include list

* Initial readme update

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-08-11 18:35:43 +02:00
Ayush Chaurasia
621caea53c
W&B: Fix sweep bug (#4276)
* Improve docstrings and run names

* default wandb login prompt with timeout

* return key

* Update api_key check logic

* Properly support zipped dataset feature

* update docstring

* Revert tuorial change

* extend changes to log_dataset

* add run name

* bug fix

* bug fix

* Update comment

* fix import check

* remove unused import

* Hardcore .yaml file extension

* reduce code

* Reformat using pycharm

* Remove redundant try catch

* More refactoring and bug fixes

* retry

* Reformat using pycharm

* respect LOGGERS include list

* call wandblogger.log instead of wandb.log

* Fix Sweep bug

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-08-02 18:41:52 +02:00
Glenn Jocher
b60b62e874
PyCharm reformat (#4209)
* PyCharm reformat

* YAML reformat

* Markdown reformat
2021-07-28 23:35:14 +02:00
Ayush Chaurasia
750465edae
W&B: More improvements and refactoring (#4205)
* Improve docstrings and run names

* default wandb login prompt with timeout

* return key

* Update api_key check logic

* Properly support zipped dataset feature

* update docstring

* Revert tuorial change

* extend changes to log_dataset

* add run name

* bug fix

* bug fix

* Update comment

* fix import check

* remove unused import

* Hardcore .yaml file extension

* reduce code

* Reformat using pycharm

* Remove redundant try catch

* More refactoring and bug fixes

* retry

* Reformat using pycharm

* respect LOGGERS include list

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-07-28 23:25:15 +02:00
Ayush Chaurasia
e88e8f7a98
W&B: Restructure code to support the new dataset_check() feature (#4197)
* Improve docstrings and run names

* default wandb login prompt with timeout

* return key

* Update api_key check logic

* Properly support zipped dataset feature

* update docstring

* Revert tuorial change

* extend changes to log_dataset

* add run name

* bug fix

* bug fix

* Update comment

* fix import check

* remove unused import

* Hardcore .yaml file extension

* reduce code

* Reformat using pycharm

Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
2021-07-28 17:40:08 +02:00
Glenn Jocher
5d66e48723
Train from --data path/to/dataset.zip feature (#4185)
* Train from `--data path/to/dataset.zip` feature

* Update dataset_stats()

* cleanup

* cleanup2
2021-07-28 02:04:10 +02:00
Ayush Chaurasia
63a1971e31
Improve docstrings and run names (#4174) 2021-07-27 15:13:32 +02:00
Glenn Jocher
0ad6301c96
Update script headers (#4163)
* Update download script headers

* cleanup

* bug fix attempt

* bug fix attempt2

* bug fix attempt3

* cleanup
2021-07-26 15:23:33 +02:00
Glenn Jocher
efe60b5681
Refactor train.py and val.py loggers (#4137)
* Update loggers

* Config

* Update val.py

* cleanup

* fix1

* fix2

* fix3 and reformat

* format sweep.py

* Logger() class

* cleanup

* cleanup2

* wandb package import fix

* wandb package import fix2

* txt fix

* fix4

* fix5

* fix6

* drop wandb into utils/loggers

* fix 7

* rename loggers/wandb_logging to loggers/wandb

* Update message

* Update message

* Update message

* cleanup

* Fix x axis bug

* fix rank 0 issue

* cleanup
2021-07-25 01:18:39 +02:00