9 Commits

Author	SHA1	Message	Date
Ross Wightman	ac469b50da	Optimizer improvements, additions, cleanup ... * Add MADGRAD code * Fix Lamb (non-fused variant) to work w/ PyTorch XLA * Tweak optimizer factory args (lr/learning_rate and opt/optimizer_name), may break compat * Use newer fn signatures for all add,addcdiv, addcmul in optimizers * Use upcoming PyTorch native Nadam if it's available * Cleanup lookahead opt * Add optimizer tests * Remove novograd.py impl as it was messy, keep nvnovograd * Make AdamP/SGDP work in channels_last layout * Add rectified adablief mode (radabelief) * Support a few more PyTorch optim, adamax, adagrad	2021-08-17 17:51:20 -07:00
Ross Wightman	137a374930	Merge pull request #555 from MichaelMonashev/patch-1 ... benchmark.py argument description fixed	2021-05-04 11:44:01 -07:00
Ross Wightman	e15e68d881	Fix #566, summary.csv writing to pwd on local_rank != 0. Tweak benchmark mem handling to see if it reduces likelihood of 'bad' exceptions on OOM.	2021-04-15 23:03:56 -07:00
Michael Monashev	0be1fa4793	Argument description fixed	2021-04-11 18:08:43 +03:00
Ross Wightman	37c71a5609	Some further create_optimizer_v2 tweaks, remove some redudnant code, add back safe model str. Benchmark step times per batch.	2021-04-01 22:34:55 -07:00
Ross Wightman	288682796f	Update benchmark script to add precision arg. Fix some downstream (DeiT) compat issues with latest changes. Bump version to 0.4.7	2021-04-01 16:40:12 -07:00
Ross Wightman	4445eaa470	Add img_size to benchmark output	2021-03-05 16:48:31 -08:00
Ross Wightman	0706d05d52	Benchmark models listed in txt file. Add more hybrid vit variants for testing	2021-02-28 16:00:33 -08:00
Ross Wightman	0e16d4e9fb	Add benchmark.py script, and update optimizer factory to be more friendly to use outside of argparse interface.	2021-02-23 15:38:12 -08:00