f35d6ea57b
Add multi-tensor (foreach) version of Lion in style of upcoming PyTorch 2.0 optimizers
2023-02-16 15:48:00 -08:00
709d5e0d9d
Add Lion optimizer
2023-02-14 23:55:05 -08:00
927f031293
Major module / path restructure, timm.models.layers -> timm.layers, add _ prefix to all non model modules in timm.models
2022-12-06 15:00:06 -08:00
b1b024dfed
Scheduler update, add v2 factory method, support scheduling on updates instead of just epochs. Add LR to summary csv. Add lr_base scaling calculations to train script. Fix #1168
2022-10-07 10:43:04 -07:00
2a296412be
Add Adan optimizer
2022-09-23 16:05:52 -07:00
33e30f8c8b
Remove layer-decay print
2022-09-18 21:33:03 -07:00
0557c8257d
Fix bug introduced in non layer_decay weight_decay application. Remove debug print, fix arg desc.
2022-02-28 17:06:32 -08:00
372ad5fa0d
Significant model refactor and additions:
...
* All models updated with revised foward_features / forward_head interface
* Vision transformer and MLP based models consistently output sequence from forward_features (pooling or token selection considered part of 'head')
* WIP param grouping interface to allow consistent grouping of parameters for layer-wise decay across all model types
* Add gradient checkpointing support to a significant % of models, especially popular architectures
* Formatting and interface consistency improvements across models
* layer-wise LR decay impl part of optimizer factory w/ scale support in scheduler
* Poolformer and Volo architectures added
2022-02-28 13:56:23 -08:00
c207e02782
MOAR optimizer changes. Woo!
2021-08-18 22:20:35 -07:00
a426511c95
More optimizer cleanup. Change all to no longer use .data. Improve (b)float16 use with adabelief. Add XLA compatible Lars.
2021-08-18 17:21:56 -07:00
a6af48be64
add madgradw optimizer
2021-08-17 22:19:27 -07:00
55fb5eedf6
Remove experiment from lamb impl
2021-08-17 21:48:26 -07:00
8a9eca5157
A few optimizer comments, dead import, missing import
2021-08-17 18:01:33 -07:00
ac469b50da
Optimizer improvements, additions, cleanup
...
* Add MADGRAD code
* Fix Lamb (non-fused variant) to work w/ PyTorch XLA
* Tweak optimizer factory args (lr/learning_rate and opt/optimizer_name), may break compat
* Use newer fn signatures for all add,addcdiv, addcmul in optimizers
* Use upcoming PyTorch native Nadam if it's available
* Cleanup lookahead opt
* Add optimizer tests
* Remove novograd.py impl as it was messy, keep nvnovograd
* Make AdamP/SGDP work in channels_last layout
* Add rectified adablief mode (radabelief)
* Support a few more PyTorch optim, adamax, adagrad
2021-08-17 17:51:20 -07:00
1042b8a146
Add non fused LAMB optimizer option
2021-08-09 13:13:43 -07:00
cd3dc4979f
Fix adabelief imports, remove prints, preserve memory format is the default arg for zeros_like
2021-04-12 08:25:31 -07:00
addfc7c1ac
adabelief
2021-04-04 23:48:15 -04:00
37c71a5609
Some further create_optimizer_v2 tweaks, remove some redudnant code, add back safe model str. Benchmark step times per batch.
2021-04-01 22:34:55 -07:00
288682796f
Update benchmark script to add precision arg. Fix some downstream (DeiT) compat issues with latest changes. Bump version to 0.4.7
2021-04-01 16:40:12 -07:00
0e16d4e9fb
Add benchmark.py script, and update optimizer factory to be more friendly to use outside of argparse interface.
2021-02-23 15:38:12 -08:00
7c56c718f3
Configure create_optimizer with args.opt_args
...
Closes #301
2020-12-08 00:03:09 -06:00
30ab4a1494
Fix issue in optim factory with sgd / eps flag. Bump version to 0.3.1
2020-10-31 18:05:30 -07:00
f944242cb0
Fix #262 , num_classes arg mixup. Make vision_transformers a bit closer to other models wrt get/reset classfier/forward_features. Fix torchscript for ViT.
2020-10-29 13:58:28 -07:00
477a78ed81
Fix optimizer factory regressin for optimizers like sgd/momentum that don't have an eps arg
2020-10-22 15:59:47 -07:00
a4d8fea61e
Add model based wd skip support. Improve cross version compat of optimizer factory. Fix #247
2020-10-13 12:49:47 -07:00
80078c47bb
Add Adafactor and Adahessian optimizers, cleanup optimizer arg passing, add gradient clipping support.
2020-10-09 17:24:43 -07:00
7995295968
Merge branch 'logger' into features. Change 'logger' to '_logger'.
2020-07-27 18:00:46 -07:00
6c17d57a2c
Fix some attributions, add copyrights to some file docstrings
2020-07-27 13:44:56 -07:00
e93e571f7a
Add `adamp` and 'sgdp' optimizers.
...
Update requirements.txt
Update optim_factory.py
Add `adamp` optimizer
Update __init__.py
copy files of adamp & sgdp
Create adamp.py
Update __init__.py
Create sgdp.py
Update optim_factory.py
Update optim_factory.py
Update requirements.txt
Update adamp.py
Update sgdp.py
Update sgdp.py
Update adamp.py
2020-07-25 15:33:20 -07:00
e6f24e5578
Add 'momentum' optimizer (SGD w/o nesterov) for stable EfficientDet training defaults
2020-04-25 19:42:13 -07:00
64966f61f7
Add Nvidia's NovogGrad impl from Jasper (cleaner/faster than current) and Apex Fused optimizers
2019-08-29 15:21:38 -07:00
fac58f609a
Add RAdam, NovoGrad, Lookahead, and AdamW optimizers, a few ResNet tweaks and scheduler factory tweak.
...
* Add some of the trendy new optimizers. Decent results but not clearly better than the standards.
* Can create a None scheduler for constant LR
* ResNet defaults to zero_init of last BN in residual
* add resnet50d config
2019-08-28 00:14:10 -07:00
aa4354f466
Big re-org, working towards making pip/module as 'timm'
2019-06-19 17:20:51 -07:00
Ross Wightman
juntang
Jasha
Sangdoo Yun