91 Commits

Author SHA1 Message Date
Ross Wightman
cb3f9c23bb
Metaformer baselines for vision (final PR with cleanup) (#1793)
* update

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* merge with poolformer, initial version

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Revert "Update metaformers.py"

This reverts commit 2916f37f8d862cd4fcbc86c858b23894daec3339.

* Revert "Update metaformers.py"

This reverts commit 1d882eb494caf5b5a3a1e9c7442d883e1707998e.

* Revert "Update metaformers.py"

This reverts commit 2209d0830eed9b87149df4a1f25f65da46d47f4b.

* Revert "Update metaformers.py"

This reverts commit 32bede4e279ee1027a63d5b55591153e53c209ec.

* Revert "Update metaformers.py"

This reverts commit 4ed934e00068be576881a87d5218b355ca01a6be.

* Revert "Update metaformers.py"

This reverts commit 3f0b07536764c47af42290a218a15097b5cec1a0.

* Revert "Update metaformers.py"

This reverts commit 2fef9006d72824789b733dd8dbc20b21b79add0b.

* Update metaformers.py

* Update metaformers.py

* rename model

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Stem/Downsample rework

* Update metaformers.py

* try NHWC

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Squashed commit of the following:

commit b7696a30a772dbbb2e00d81e7096c24dac97df73
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Fri Feb 10 01:46:44 2023 -0800

    Update metaformers.py

commit 41fe5c36263b40a6cd7caddb85b10c5d82d48023
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Fri Feb 10 01:03:47 2023 -0800

    Update metaformers.py

commit a3aee37c35985c01ca07902d860f809b648c612c
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Fri Feb 10 00:32:04 2023 -0800

    Update metaformers.py

commit f938beb81b4f46851d6d6f04ae7a9a74871ee40d
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Fri Feb 10 00:24:58 2023 -0800

    Update metaformers.py

commit 10bde717e51c95cdf20135c8bba77a7a1b00d78c
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Sun Feb 5 02:11:28 2023 -0800

    Update metaformers.py

commit 39274bd45e78b8ead0509367f800121f0d7c25f4
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Sun Feb 5 02:06:58 2023 -0800

    Update metaformers.py

commit a2329ab8ec00d0ebc00979690293c3887cc44a4c
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Sun Feb 5 02:03:34 2023 -0800

    Update metaformers.py

commit 53b8ce5b8a6b6d828de61788bcc2e6043ebb3081
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Sun Feb 5 02:02:37 2023 -0800

    Update metaformers.py

commit ab6225b9414f534815958036f6d5a392038d7ab2
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Sun Feb 5 01:04:55 2023 -0800

    try NHWC

commit 02fcc30eaa67a3c92cae56f3062b2542c32c9283
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Sat Feb 4 23:47:06 2023 -0800

    Update metaformers.py

commit 366aae93047934bd3d7d37a077e713d424fa429c
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Sat Feb 4 23:37:30 2023 -0800

    Stem/Downsample rework

commit 26a8e481a5cb2a32004a796bc25c6800cd2fb7b7
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Wed Feb 1 07:42:07 2023 -0800

    Update metaformers.py

commit a913f5d4384aa4b2f62fdab46254ae3772df00ee
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Wed Feb 1 07:41:24 2023 -0800

    Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* channels first for whole network

* Channels first

* Update metaformers.py

* Use buffer for randformer

* Update metaformers.py

* Remove einsum

* don't test randformer for feature extraction

* arbitrary input sizes for randformer

* Squashed commit of the following:

commit 6c089ca4325ab10942fe56e0999dcc1a11e1d2f0
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Mon Mar 6 02:11:17 2023 -0800

    Update metaformers.py

commit 521528a900e49ef8f462f5ccd795efb3a5d14214
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Mon Mar 6 02:06:08 2023 -0800

    Update metaformers.py

commit 3827eec7963698ff727fbb13ace53594ceb374d5
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Mon Mar 6 02:03:08 2023 -0800

    Update metaformers.py

commit ac1c6fea8adcd846e031ea0f5fa81ffe63d3c4bb
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Mon Mar 6 02:01:04 2023 -0800

    Update metaformers.py

commit 26f3d343cdc46183543f83482187f669f3181ddf
Merge: d577129 f736730
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Mon Mar 6 01:57:29 2023 -0800

    Merge branch 'metaformer_workspace' of https://github.com/fffffgggg54/pytorch-image-models into metaformer_workspace

commit d577129aaa23fb348a8bb93bcd17cf1d5a4e8ff8
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Mon Mar 6 01:57:20 2023 -0800

    Update metaformers.py

commit f7367304e8f3b7a9a7f16e0a032bb72546afcc2a
Author: Fredo Guan <fredo.guan@hotmail.com>
Date:   Mon Mar 6 01:56:11 2023 -0800

    Metaformer baselines for vision (#12)

* formatting, cleanup, fix dropout

* fix regression, pass kwargs

* fix poolformerv1 weights, formatting

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* Update metaformers.py

* some cleanup

* SDPA from ViT, fix imports

* Update metaformers.py

* fix head reset

* fast norm bias patch for metaformers

* Metaformer refactor, remove rand/ident models, fix issues, remove old poolformer

* Switch to hub weights

---------

Co-authored-by: Fredo Guan <fredo.guan@hotmail.com>
2023-05-05 11:18:26 -07:00
Ross Wightman
3386af8c86 Final push to get remaining models using multi-weight pretrained configs, almost all weights on HF hub 2023-04-26 15:52:13 -07:00
Ross Wightman
356309959c ResNet models on HF hub, multi-weight support, add torchvision v2 weights, new 12k pretrained and fine-tuned timm anti-aliased weights 2023-04-05 14:19:42 -07:00
Ross Wightman
3863d63516 Adding EVA02 weights and model defs, move beit based eva_giant to same eva.py file. Cleanup rotary pos, add lang oriented freq bands to be compat with eva design choice. Fix #1738 2023-03-27 17:16:07 -07:00
Ross Wightman
572f05096a Swin and FocalNet weights on HF hub. Add model deprecation functionality w/ some registry tweaks. 2023-03-18 14:55:09 -07:00
Ross Wightman
acfd85ad68 All swin models support spatial output, add output_fmt to v1/v2 and use ClassifierHead.
* update ClassifierHead to allow different input format
* add output format support to patch embed
* fix some flatten issues for a few conv head models
* add Format enum and helpers for tensor format (layout) choices
2023-03-15 23:21:51 -07:00
Ross Wightman
7266c5c716 Merge branch 'main' into focalnet_and_swin_refactor 2023-02-17 09:20:14 -08:00
Ross Wightman
086bd55a94 Add EfficientFormer-V2, refactor EfficientFormer and Levit for more uniformity across the 3 related arch. Add features_out support to levit conv models and efficientformer_v2. All weights on hub. 2023-02-03 14:12:29 -08:00
Ross Wightman
01fdf44438 Initial focalnet import, more refactoring needed for timm. 2023-01-09 16:18:19 -08:00
Fredo Guan
84178fca60
Merge branch 'rwightman:main' into main 2022-12-12 23:13:58 -08:00
Fredo Guan
3bd96609c8
Davit (#1)
Implement the davit model from https://arxiv.org/abs/2204.03645 and https://github.com/dingmyu/davit
2022-12-06 17:19:25 -08:00
Ross Wightman
927f031293 Major module / path restructure, timm.models.layers -> timm.layers, add _ prefix to all non model modules in timm.models 2022-12-06 15:00:06 -08:00
Ross Wightman
755570e2d6 Rename _pretrained.py -> pretrained.py, not feasible to change the other files to same scheme without breaking uses 2022-12-05 10:21:34 -08:00
Ross Wightman
72cfa57761 Add ported Tensorflow MaxVit weights. Add a few more CLIP ViT fine-tunes. Tweak some model tag names. Improve model tag name sorting. Update HF hub push config layout. 2022-12-05 10:21:34 -08:00
Ross Wightman
0dadb4a6e9 Initial multi-weight support, handled so old pretraing config handling co-exists with new tags. 2022-12-05 10:21:34 -08:00
Ross Wightman
ff6a919cf5 Add --fast-norm arg to benchmark.py, train.py, validate.py 2022-08-25 17:20:46 -07:00
Ross Wightman
ffaf97f813 MaxxVit! A very configurable MaxVit and CoAtNet impl with lots of goodies.. 2022-08-22 17:42:10 -07:00
Ross Wightman
6e559e9b5f Add MViT (Multi-Scale) V2 2022-08-17 15:12:31 -07:00
Ross Wightman
c486aa71f8 Add GCViT 2022-08-17 14:29:18 -07:00
Ross Wightman
fba6ecd39b Add EfficientFormer 2022-08-17 14:08:53 -07:00
Ross Wightman
ff4a38e2c3 Add PyramidVisionTransformerV2 2022-08-17 12:06:05 -07:00
Ross Wightman
6064d16a2d Add initial EdgeNeXt import. Significant cleanup / reorg (like ConvNeXt). Fix #1320
* edgenext refactored for torchscript compat, stage base organization
* slight refactor of ConvNeXt to match some EdgeNeXt additions
* remove use of funky LayerNorm layer in ConvNeXt and just use nn.LayerNorm and LayerNorm2d (permute)
2022-07-01 15:18:42 -07:00
Ross Wightman
879df47c0a Support BatchNormAct2d for sync-bn use. Fix #1254 2022-06-24 14:51:26 -07:00
Ross Wightman
9a86b900fa Official SwinV2 models 2022-05-12 15:05:10 -07:00
Ross Wightman
d07d015173
Merge pull request #1249 from okojoalg/sequencer
Add Sequencer
2022-05-09 20:42:43 -07:00
okojoalg
578d52e752 Add Sequencer 2022-05-06 00:36:01 +09:00
Ross Wightman
41dc49a337 Vision Transformer refactoring and Rel Pos impl 2022-05-02 15:37:39 -07:00
Ross Wightman
b049a5c5c6 Merge remote-tracking branch 'origin/master' into norm_norm_norm 2022-03-21 13:41:43 -07:00
Ross Wightman
372ad5fa0d Significant model refactor and additions:
* All models updated with revised foward_features / forward_head interface
* Vision transformer and MLP based models consistently output sequence from forward_features (pooling or token selection considered part of 'head')
* WIP param grouping interface to allow consistent grouping of parameters for layer-wise decay across all model types
* Add gradient checkpointing support to a significant % of models, especially popular architectures
* Formatting and interface consistency improvements across models
* layer-wise LR decay impl part of optimizer factory w/ scale support in scheduler
* Poolformer and Volo architectures added
2022-02-28 13:56:23 -08:00
Ross Wightman
c6e4b7895a Swin V2 CR impl refactor.
* reformat and change some naming so closer to existing timm vision transformers
* remove typing that wasn't adding clarity (or causing torchscript issues)
* support non-square windows
* auto window size adjust from image size
* post-norm + main-branch no
2022-02-23 17:28:52 -08:00
Ross Wightman
58ba49c8ef Add MobileViT models (w/ ByobNet base). Close #1038. 2022-01-31 15:39:34 -08:00
Ross Wightman
5f81d4de23 Move DeiT to own file, vit getting crowded. Working towards fixing #1029, make pooling interface for transformers and mlp closer to convnets. Still working through some details... 2022-01-26 22:53:57 -08:00
Ross Wightman
95cfc9b3e8 Merge remote-tracking branch 'origin/master' into norm_norm_norm 2022-01-25 22:20:45 -08:00
Ross Wightman
abc9ba2544 Transitioning default_cfg -> pretrained_cfg. Improving handling of pretrained_cfg source (HF-Hub, files, timm config, etc). Checkpoint handling tweaks. 2022-01-25 21:54:13 -08:00
Ross Wightman
18934debc5 Add initial ConvNeXt impl (mods of official code) 2022-01-14 16:28:27 -08:00
ICLR Author
44d6d51668 Add ConvMixer 2021-10-09 21:09:51 -04:00
Ross Wightman
1c9284c640 Add BeiT 'finetuned' 1k weights and pretrained 22k weights, pretraining specific (masked) model excluded for now 2021-09-13 16:38:23 -07:00
Ross Wightman
f1808e0970 Post crossvit merge cleanup, change model names to reflect input size, cleanup img size vs scale handling, fix tests 2021-09-13 11:49:54 -07:00
Richard Chen
7ab9d4555c add crossvit 2021-09-01 17:13:12 -04:00
Alexander Soare
623e8b8eb8 wip xcit 2021-07-11 09:39:38 +01:00
Alexander Soare
23bb72ce5e nested_transformer wip 2021-07-02 20:12:29 +01:00
Ross Wightman
742c2d5247 Add Gather-Excite and Global Context attn modules. Refactor existing SE-like attn for consistency and refactor byob/byoanet for less redundancy. 2021-05-27 18:03:29 -07:00
Ross Wightman
bfc72f75d3 Expand scope of testing for non-std vision transformer / mlp models. Some related cleanup and create fn cleanup for all vision transformer and mlp models. More CoaT weights. 2021-05-24 21:13:26 -07:00
Ross Wightman
f45de37690 Merge branch 'master' into levit_visformer_rednet 2021-05-22 16:34:31 -07:00
Ross Wightman
306c86b668 Merge branch 'convit' of https://github.com/amaarora/pytorch-image-models into amaarora-convit 2021-05-21 16:27:10 -07:00
Aman Arora
40c506ba1e Add ConViT 2021-05-20 23:17:28 +00:00
李鑫杰
00548b8427 Add Twins 2021-05-18 19:21:53 +08:00
Ross Wightman
ecc7552c5c Add levit, levit_c, and visformer model defs. Largely untested and not finished cleanup. 2021-05-14 17:16:34 -07:00
Ross Wightman
5fcddb96a8 Merge branch 'master' into cait 2021-05-05 17:29:38 -07:00
Ross Wightman
12efffa6b1 Initial MLP-Mixer attempt... 2021-05-05 00:59:45 -07:00