Commit Graph

30 Commits (a667d3d8f0f13b0759dc919246a130b6a9fc9772)

Author SHA1 Message Date
Ross Wightman 5f10450235 Some more kron work. Figured out why some tests fail, implemented a deterministic rng state load but too slow so skipping some tests for now. 2025-01-27 21:02:26 -08:00
Ross Wightman 303f7691a1 Add cautious mars, improve test reliability by skipping grad diff for first step 2024-12-02 11:29:02 -08:00
Ross Wightman 7cf683628f Cautious optimizer impl plus some typing cleanup. 2024-11-28 15:11:58 -08:00
Ross Wightman aeb1ed7a15 Keep basic optim test LR range closer to before w/ updated code 2024-11-26 15:10:15 -08:00
Ross Wightman 7a165fcb62 Remove rogue import, thanks IDE :/ 2024-11-26 15:10:15 -08:00
Ross Wightman 73d10ab482 Update tests, need handling for radamw with older PyTorch, need to back-off basic test LR in mars? 2024-11-26 15:10:15 -08:00
Ross Wightman a024ab3170 Replace radam & nadam impl with torch.optim ver, rename legacy adamw, nadam, radam impl in timm. Update optim factory & tests. 2024-11-26 15:10:15 -08:00
Ross Wightman 0b5264a108 Missing optimizers in __init__.py, add bind_defaults=False for unit tests 2024-11-13 10:50:46 -08:00
Ross Wightman dde990785e More fixes for new factory & tests, add back adahessian 2024-11-12 20:49:01 -08:00
Ross Wightman 45490ac52f Post merge fix reference of old param groups helper fn locations 2024-11-12 20:49:01 -08:00
Ross Wightman ee5f6e76bb A bit of an optimizer overhaul, added an improved factory, list_optimizers, class helper and add info classes with descriptions, arg configs 2024-11-12 20:49:01 -08:00
Ross Wightman 94e0560aba Remove an indent level in init_group for adopt, update optim tests, adopt failing rosenbrock 2024-11-12 20:49:01 -08:00
mrT23 e31e5d2d64 imports 2024-11-12 07:53:39 -08:00
Tal 68d5a64e45 extend existing unittests 2024-11-12 07:53:39 -08:00
Alex Cunha a38c2b4475 fix bug 2023-11-16 20:27:59 -08:00
Alex Cunha bbf0ab000c return if cuda not available 2023-11-16 20:27:59 -08:00
Alex Cunha e628ed7e67 device agnostic testing 2023-11-16 20:27:59 -08:00
Ross Wightman 68b2824e49
Update optim test to remove Variable/.data and fix _state_dict optim test for PyTorch 2.1 (#1988)
* Update optim test to remove Variable/.data and fix _state_dict optim test

* Attempt to run python 3.11 w/ 2.1

* Try factoring out testmarker to common var

* More fiddling

* Abandon attempt to reduce redunancy

* Another try
2023-10-12 11:53:10 -07:00
Ross Wightman 24720abe3b Merge branch 'master' into attn_update 2021-09-13 16:51:10 -07:00
Ross Wightman 8642401e88 Swap botnet 26/50 weights/models after realizing a mistake in arch def, now figuring out why they were so low... 2021-09-05 15:17:19 -07:00
Ross Wightman 5f12de4875 Add initial AttentionPool2d that's being trialed. Fix comment and still trying to improve reliability of sgd test. 2021-09-05 12:41:14 -07:00
Ross Wightman 54e90e82a5 Another attempt at sgd momentum test passing... 2021-09-03 20:50:26 -07:00
Ross Wightman fc894c375c Another attempt at sgd momentum test passing... 2021-08-27 10:39:31 -07:00
Ross Wightman 708d87a813 Fix ViT SAM weight compat as weights at URL changed to not use repr layer. Fix #825. Tweak optim test. 2021-08-27 09:20:13 -07:00
Ross Wightman c207e02782 MOAR optimizer changes. Woo! 2021-08-18 22:20:35 -07:00
Ross Wightman 42c1f0cf6c Fix lars tests 2021-08-18 21:05:34 -07:00
Ross Wightman a426511c95 More optimizer cleanup. Change all to no longer use .data. Improve (b)float16 use with adabelief. Add XLA compatible Lars. 2021-08-18 17:21:56 -07:00
Ross Wightman a6af48be64 add madgradw optimizer 2021-08-17 22:19:27 -07:00
Ross Wightman 55fb5eedf6 Remove experiment from lamb impl 2021-08-17 21:48:26 -07:00
Ross Wightman 959eaff121 Add optimizer tests and update testing to pytorch 1.9 2021-08-17 17:59:15 -07:00