angiecao
608e319eb6
[Feature] Support Side Adapter Network ( #3232 )
...
## Motivation
Support SAN for Open-Vocabulary Semantic Segmentation
Paper: [Side Adapter Network for Open-Vocabulary Semantic
Segmentation](https://arxiv.org/abs/2302.12242 )
official Code: [SAN](https://github.com/MendelXu/SAN )
## Modification
- Added the parameters of backbone vit for implementing the image
encoder of CLIP.
- Added text encoder code.
- Added segmentor multimodel encoder-decoder code for open-vocabulary
semantic segmentation.
- Added SideAdapterNetwork decode head code.
- Added config files for train and inference.
- Added tools for converting pretrained models.
- Added loss implementation for mask classification model, such as SAN,
Maskformer and remove dependency on mmdetection.
- Added test units for text encoder, multimodel encoder-decoder, san
decode head and hungarian_assigner.
## Use cases
### Convert Models
**pretrained SAN model**
The official pretrained model can be downloaded from
[san_clip_vit_b_16.pth](https://huggingface.co/Mendel192/san/blob/main/san_vit_b_16.pth )
and
[san_clip_vit_large_14.pth](https://huggingface.co/Mendel192/san/blob/main/san_vit_large_14.pth ).
Use tools/model_converters/san2mmseg.py to convert offcial model into
mmseg style.
`python tools/model_converters/san2mmseg.py <MODEL_PATH> <OUTPUT_PATH>`
**pretrained CLIP model**
Use the CLIP model provided by openai to train SAN. The CLIP model can
be download from
[ViT-B-16.pt](https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt )
and
[ViT-L-14-336px.pt](https://openaipublic.azureedge.net/clip/models/3035c92b350959924f9f00213499208652fc7ea050643e8b385c2dac08641f02/ViT-L-14-336px.pt ).
Use tools/model_converters/clip2mmseg.py to convert model into mmseg
style.
`python tools/model_converters/clip2mmseg.py <MODEL_PATH> <OUTPUT_PATH>`
### Inference
test san_vit-base-16 model on coco-stuff164k dataset
`python tools/test.py
./configs/san/san-vit-b16_coco-stuff164k-640x640.py
<TRAINED_MODEL_PATH>`
### Train
test san_vit-base-16 model on coco-stuff164k dataset
`python tools/train.py
./configs/san/san-vit-b16_coco-stuff164k-640x640.py --cfg-options
model.pretrained=<PRETRAINED_MODEL_PATH>`
## Comparision Results
### Train on COCO-Stuff164k
| | | mIoU | mAcc | pAcc |
| --------------- | ----- | ----- | ----- | ----- |
| san-vit-base16 | official | 41.93 | 56.73 | 67.69 |
| | mmseg | 41.93 | 56.84 | 67.84 |
| san-vit-large14 | official | 45.57 | 59.52 | 69.76 |
| | mmseg | 45.78 | 59.61 | 69.21 |
### Evaluate on Pascal Context
| | | mIoU | mAcc | pAcc |
| --------------- | ----- | ----- | ----- | ----- |
| san-vit-base16 | official | 54.05 | 72.96 | 77.77 |
| | mmseg | 54.04 | 73.74 | 77.71 |
| san-vit-large14 | official | 57.53 | 77.56 | 78.89 |
| | mmseg | 56.89 | 76.96 | 78.74 |
### Evaluate on Voc12Aug
| | | mIoU | mAcc | pAcc |
| --------------- | ----- | ----- | ----- | ----- |
| san-vit-base16 | official | 93.86 | 96.61 | 97.11 |
| | mmseg | 94.58 | 97.01 | 97.38 |
| san-vit-large14 | official | 95.17 | 97.61 | 97.63 |
| | mmseg | 95.58 | 97.75 | 97.79 |
---------
Co-authored-by: CastleDream <35064479+CastleDream@users.noreply.github.com>
Co-authored-by: yeedrag <46050186+yeedrag@users.noreply.github.com>
Co-authored-by: Yang-ChangHui <71805205+Yang-Changhui@users.noreply.github.com>
Co-authored-by: Xu CAO <49406546+SheffieldCao@users.noreply.github.com>
Co-authored-by: xiexinch <xiexinch@outlook.com>
Co-authored-by: 小飞猪 <106524776+ooooo-create@users.noreply.github.com>
2023-09-20 21:20:26 +08:00
谢昕辰
da4125587e
[Refactor] Support TTA ( #2184 )
...
* tta init
* use mmcv transform
* test city
* add multiscale
* fix merge
* add softmax to post process
* add ut
* add tta pipeline to other datasets
* remove softmax
* add encoder_decoder_tta ut
* add encoder_decoder_tta ut
* rename
* rename file
* rename config
* rm aug_test
* move flip to post process
* fix channel
2022-12-30 22:52:07 +08:00
谢昕辰
b3b7629d38
[Fix] Fix binary segmentation ( #2101 )
...
* add out_channels
* fix forward
* add decode_head ut
* add segmentor ut
* refine postprocess
* fix
2022-09-27 16:49:38 +08:00
zhengmiao
35c695bba7
[Refactory] Clean UTs
2022-05-09 10:29:38 +00:00
Julius Zhang
96b369bd55
[Fix] Fix loss parse in val_step ( #906 )
...
* [Fix] Fix loss parse in val_step
* Add val_step unittest
* Add train_step unittest
2021-09-25 18:17:40 -07:00
Junjun2016
2fe0bddf5e
[Dcos] Add header for files ( #796 )
...
* Add header for files
* Delete header in config files
2021-08-16 23:16:55 -07:00
Jerry Jiarui XU
3150dd0ce4
refactor test organization ( #440 )
...
* refactor test organization
* fixed se layer
* update mmcv uper bound
2021-03-30 17:55:09 -07:00
