683fba7686
Add drop args to benchmark.py
2021-12-14 13:51:00 -08:00
aaff2d82d0
Add new 50ts attn models to benchmark/meta csv files
2021-10-28 14:32:47 -07:00
1e17863b7b
Fixed botne*t26 model results, add some 50ts self-attn variants
2021-10-28 13:55:24 -07:00
71f00bfe9e
Don't run profile if model is torchscripted
2021-10-24 22:41:20 -07:00
5882e62ada
Add activation count to fvcore based profiling in benchmark.py
2021-10-24 15:30:38 -07:00
f7325c7b71
Support either deepspeed or fvcore for flop profiling
2021-10-20 15:17:30 -07:00
66253790d4
Add `--bench profile` mode for benchmark.py to just run deepspeed detailed profile on model
2021-10-19 16:06:38 -07:00
13a8bf7972
Add train size override and deepspeed GMACs counter (if deepspeed installed) to benchmark.py
2021-10-19 15:15:01 -07:00
ac469b50da
Optimizer improvements, additions, cleanup
...
* Add MADGRAD code
* Fix Lamb (non-fused variant) to work w/ PyTorch XLA
* Tweak optimizer factory args (lr/learning_rate and opt/optimizer_name), may break compat
* Use newer fn signatures for all add,addcdiv, addcmul in optimizers
* Use upcoming PyTorch native Nadam if it's available
* Cleanup lookahead opt
* Add optimizer tests
* Remove novograd.py impl as it was messy, keep nvnovograd
* Make AdamP/SGDP work in channels_last layout
* Add rectified adablief mode (radabelief)
* Support a few more PyTorch optim, adamax, adagrad
2021-08-17 17:51:20 -07:00
137a374930
Merge pull request #555 from MichaelMonashev/patch-1
...
benchmark.py argument description fixed
2021-05-04 11:44:01 -07:00
e15e68d881
Fix #566 , summary.csv writing to pwd on local_rank != 0. Tweak benchmark mem handling to see if it reduces likelihood of 'bad' exceptions on OOM.
2021-04-15 23:03:56 -07:00
0be1fa4793
Argument description fixed
2021-04-11 18:08:43 +03:00
37c71a5609
Some further create_optimizer_v2 tweaks, remove some redudnant code, add back safe model str. Benchmark step times per batch.
2021-04-01 22:34:55 -07:00
288682796f
Update benchmark script to add precision arg. Fix some downstream (DeiT) compat issues with latest changes. Bump version to 0.4.7
2021-04-01 16:40:12 -07:00
4445eaa470
Add img_size to benchmark output
2021-03-05 16:48:31 -08:00
0706d05d52
Benchmark models listed in txt file. Add more hybrid vit variants for testing
2021-02-28 16:00:33 -08:00
0e16d4e9fb
Add benchmark.py script, and update optimizer factory to be more friendly to use outside of argparse interface.
2021-02-23 15:38:12 -08:00
Ross Wightman
Ross Wightman
Michael Monashev