Ross Wightman
|
24720abe3b
|
Merge branch 'master' into attn_update
|
2021-09-13 16:51:10 -07:00 |
|
Ross Wightman
|
8642401e88
|
Swap botnet 26/50 weights/models after realizing a mistake in arch def, now figuring out why they were so low...
|
2021-09-05 15:17:19 -07:00 |
|
Ross Wightman
|
5f12de4875
|
Add initial AttentionPool2d that's being trialed. Fix comment and still trying to improve reliability of sgd test.
|
2021-09-05 12:41:14 -07:00 |
|
Ross Wightman
|
54e90e82a5
|
Another attempt at sgd momentum test passing...
|
2021-09-03 20:50:26 -07:00 |
|
Ross Wightman
|
fc894c375c
|
Another attempt at sgd momentum test passing...
|
2021-08-27 10:39:31 -07:00 |
|
Ross Wightman
|
708d87a813
|
Fix ViT SAM weight compat as weights at URL changed to not use repr layer. Fix #825. Tweak optim test.
|
2021-08-27 09:20:13 -07:00 |
|
Ross Wightman
|
c207e02782
|
MOAR optimizer changes. Woo!
|
2021-08-18 22:20:35 -07:00 |
|
Ross Wightman
|
42c1f0cf6c
|
Fix lars tests
|
2021-08-18 21:05:34 -07:00 |
|
Ross Wightman
|
a426511c95
|
More optimizer cleanup. Change all to no longer use .data. Improve (b)float16 use with adabelief. Add XLA compatible Lars.
|
2021-08-18 17:21:56 -07:00 |
|
Ross Wightman
|
a6af48be64
|
add madgradw optimizer
|
2021-08-17 22:19:27 -07:00 |
|
Ross Wightman
|
55fb5eedf6
|
Remove experiment from lamb impl
|
2021-08-17 21:48:26 -07:00 |
|
Ross Wightman
|
959eaff121
|
Add optimizer tests and update testing to pytorch 1.9
|
2021-08-17 17:59:15 -07:00 |