14 Commits

Author SHA1 Message Date
Ross Wightman
48e1df8b37 Add norm/norm_act header comments 2022-08-25 16:29:34 -07:00
Ross Wightman
8c9696c9df More model and test fixes 2022-08-22 17:40:31 -07:00
Ross Wightman
43aa84e861 Add 'fast' layer norm that doesn't cast to float32, support APEX LN impl for slight speed gain, update norm and act factories, tweak SE for ability to disable bias (needed by GCVit) 2022-08-17 14:32:58 -07:00
Ross Wightman
e27c16b8a0 Remove unecessary code for synbn guard 2022-06-24 14:57:42 -07:00
Ross Wightman
879df47c0a Support BatchNormAct2d for sync-bn use. Fix #1254 2022-06-24 14:51:26 -07:00
Ross Wightman
f670d98cb8 Make a few more layers symbolically traceable (remove from FX leaf modules)
* remove dtype kwarg from .to() calls in EvoNorm as it messed up script + trace combo
* BatchNormAct2d always uses custom forward (cut & paste from original) instead of super().forward. Fixes #1176
* BlurPool groups==channels, no need to use input.dim[1]
2022-03-24 21:43:56 -07:00
Ross Wightman
ab49d275de Significant norm update
* ConvBnAct layer renamed -> ConvNormAct and ConvNormActAa for anti-aliased
* Significant update to EfficientNet and MobileNetV3 arch to support NormAct layers and grouped conv (as alternative to depthwise)
* Update RegNet to add Z variant
* Add Pre variant of XceptionAligned that works with NormAct layers
* EvoNorm matches bits_and_tpu branch for merge
2021-12-14 13:48:30 -08:00
Ross Wightman
af607b75cc Prep a set of ResNetV2 models with GroupNorm, EvoNormB0, EvoNormS0 for BN free model experiments on TPU and IPU 2021-11-19 17:37:00 -08:00
Ross Wightman
9811e229f7 Fix regression in models with 1001 class pretrained weights. Improve batchnorm arg and BatchNormAct layer handling in several models. 2021-02-09 16:22:52 -08:00
Ross Wightman
231d04e91a ResNetV2 pre-act and non-preact model, w/ BiT pretrained weights and support for ViT R50 model. Tweaks for in21k num_classes passing. More to do... tests failing. 2020-12-28 16:59:15 -08:00
Ross Wightman
3aebc2f06c Switch DPN to use BnAct layer, train a new DPN 68b model with RA to 79.21 2020-07-12 11:13:06 -07:00
Ross Wightman
f28170df3f Fix an an untested change, remove a debug print 2020-06-01 17:26:42 -07:00
Ross Wightman
eb7653614f Monster commit, activation refactor, VoVNet, norm_act improvements, more
* refactor activations into basic PyTorch, jit scripted, and memory efficient custom auto
* implement hard-mish, better grad for hard-swish
* add initial VovNet V1/V2 impl, fix #151
* VovNet and DenseNet first models to use NormAct layers (support BatchNormAct2d, EvoNorm, InplaceIABN)
* Wrap IABN for any models that use it
* make more models torchscript compatible (DPN, PNasNet, Res2Net, SelecSLS) and add tests
2020-06-01 17:16:52 -07:00
Ross Wightman
14edacdf9a DenseNet converted to support ABN (norm + act) modules. Experimenting with EvoNorm, IABN 2020-05-09 18:26:41 -07:00