angiecao
|
608e319eb6
|
[Feature] Support Side Adapter Network (#3232)
## Motivation
Support SAN for Open-Vocabulary Semantic Segmentation
Paper: [Side Adapter Network for Open-Vocabulary Semantic
Segmentation](https://arxiv.org/abs/2302.12242)
official Code: [SAN](https://github.com/MendelXu/SAN)
## Modification
- Added the parameters of backbone vit for implementing the image
encoder of CLIP.
- Added text encoder code.
- Added segmentor multimodel encoder-decoder code for open-vocabulary
semantic segmentation.
- Added SideAdapterNetwork decode head code.
- Added config files for train and inference.
- Added tools for converting pretrained models.
- Added loss implementation for mask classification model, such as SAN,
Maskformer and remove dependency on mmdetection.
- Added test units for text encoder, multimodel encoder-decoder, san
decode head and hungarian_assigner.
## Use cases
### Convert Models
**pretrained SAN model**
The official pretrained model can be downloaded from
[san_clip_vit_b_16.pth](https://huggingface.co/Mendel192/san/blob/main/san_vit_b_16.pth)
and
[san_clip_vit_large_14.pth](https://huggingface.co/Mendel192/san/blob/main/san_vit_large_14.pth).
Use tools/model_converters/san2mmseg.py to convert offcial model into
mmseg style.
`python tools/model_converters/san2mmseg.py <MODEL_PATH> <OUTPUT_PATH>`
**pretrained CLIP model**
Use the CLIP model provided by openai to train SAN. The CLIP model can
be download from
[ViT-B-16.pt](https://openaipublic.azureedge.net/clip/models/5806e77cd80f8b59890b7e101eabd078d9fb84e6937f9e85e4ecb61988df416f/ViT-B-16.pt)
and
[ViT-L-14-336px.pt](https://openaipublic.azureedge.net/clip/models/3035c92b350959924f9f00213499208652fc7ea050643e8b385c2dac08641f02/ViT-L-14-336px.pt).
Use tools/model_converters/clip2mmseg.py to convert model into mmseg
style.
`python tools/model_converters/clip2mmseg.py <MODEL_PATH> <OUTPUT_PATH>`
### Inference
test san_vit-base-16 model on coco-stuff164k dataset
`python tools/test.py
./configs/san/san-vit-b16_coco-stuff164k-640x640.py
<TRAINED_MODEL_PATH>`
### Train
test san_vit-base-16 model on coco-stuff164k dataset
`python tools/train.py
./configs/san/san-vit-b16_coco-stuff164k-640x640.py --cfg-options
model.pretrained=<PRETRAINED_MODEL_PATH>`
## Comparision Results
### Train on COCO-Stuff164k
| | | mIoU | mAcc | pAcc |
| --------------- | ----- | ----- | ----- | ----- |
| san-vit-base16 | official | 41.93 | 56.73 | 67.69 |
| | mmseg | 41.93 | 56.84 | 67.84 |
| san-vit-large14 | official | 45.57 | 59.52 | 69.76 |
| | mmseg | 45.78 | 59.61 | 69.21 |
### Evaluate on Pascal Context
| | | mIoU | mAcc | pAcc |
| --------------- | ----- | ----- | ----- | ----- |
| san-vit-base16 | official | 54.05 | 72.96 | 77.77 |
| | mmseg | 54.04 | 73.74 | 77.71 |
| san-vit-large14 | official | 57.53 | 77.56 | 78.89 |
| | mmseg | 56.89 | 76.96 | 78.74 |
### Evaluate on Voc12Aug
| | | mIoU | mAcc | pAcc |
| --------------- | ----- | ----- | ----- | ----- |
| san-vit-base16 | official | 93.86 | 96.61 | 97.11 |
| | mmseg | 94.58 | 97.01 | 97.38 |
| san-vit-large14 | official | 95.17 | 97.61 | 97.63 |
| | mmseg | 95.58 | 97.75 | 97.79 |
---------
Co-authored-by: CastleDream <35064479+CastleDream@users.noreply.github.com>
Co-authored-by: yeedrag <46050186+yeedrag@users.noreply.github.com>
Co-authored-by: Yang-ChangHui <71805205+Yang-Changhui@users.noreply.github.com>
Co-authored-by: Xu CAO <49406546+SheffieldCao@users.noreply.github.com>
Co-authored-by: xiexinch <xiexinch@outlook.com>
Co-authored-by: 小飞猪 <106524776+ooooo-create@users.noreply.github.com>
|
2023-09-20 21:20:26 +08:00 |