7 Commits

Author	SHA1	Message	Date
Ross Wightman	c207e02782	MOAR optimizer changes. Woo!	2021-08-18 22:20:35 -07:00
Ross Wightman	a426511c95	More optimizer cleanup. Change all to no longer use .data. Improve (b)float16 use with adabelief. Add XLA compatible Lars.	2021-08-18 17:21:56 -07:00
Ross Wightman	9541f4963b	One more scalar -> tensor fix for lamb optimizer	2021-08-18 11:20:25 -07:00
Ross Wightman	8f68193c91	Update lamp.py comment	2021-08-18 09:27:40 -07:00
Ross Wightman	55fb5eedf6	Remove experiment from lamb impl	2021-08-17 21:48:26 -07:00
Ross Wightman	ac469b50da	Optimizer improvements, additions, cleanup ... * Add MADGRAD code * Fix Lamb (non-fused variant) to work w/ PyTorch XLA * Tweak optimizer factory args (lr/learning_rate and opt/optimizer_name), may break compat * Use newer fn signatures for all add,addcdiv, addcmul in optimizers * Use upcoming PyTorch native Nadam if it's available * Cleanup lookahead opt * Add optimizer tests * Remove novograd.py impl as it was messy, keep nvnovograd * Make AdamP/SGDP work in channels_last layout * Add rectified adablief mode (radabelief) * Support a few more PyTorch optim, adamax, adagrad	2021-08-17 17:51:20 -07:00
Ross Wightman	1042b8a146	Add non fused LAMB optimizer option	2021-08-09 13:13:43 -07:00