## Motivation
The DETR-related modules have been refactored in
open-mmlab/mmdetection#8763, which causes breakings of MaskFormer and
Mask2Former in both MMDetection (has been fixed in
open-mmlab/mmdetection#9515) and MMSegmentation. This pr fix the bugs in
MMSegmentation.
### TO-DO List
- [x] update configs
- [x] check and modify data flow
- [x] fix unit test
- [x] aligning inference
- [x] write a ckpt converter
- [x] write ckpt update script
- [x] update model zoo
- [x] update model link in readme
- [x] update
[faq.md](https://github.com/open-mmlab/mmsegmentation/blob/dev-1.x/docs/en/notes/faq.md#installation)
## Tips of Fixing other implementations based on MaskXFormer of mmseg
1. The Transformer modules should be built directly. The original
building with register manner has been refactored.
2. The config requires to be modified. Delete `type` and modify several
keys, according to the modifications in this pr.
3. The `batch_first` is set `True` uniformly in the new implementations.
Hence the data flow requires to be transposed and config of
`batch_first` needs to be modified.
4. The checkpoint trained on the old implementation should be converted
to be used in the new one.
### Convert script
```Python
import argparse
from copy import deepcopy
from collections import OrderedDict
import torch
from mmengine.config import Config
from mmseg.models import build_segmentor
from mmseg.utils import register_all_modules
register_all_modules(init_default_scope=True)
def parse_args():
parser = argparse.ArgumentParser(
description='MMSeg convert MaskXFormer model, by Li-Qingyun')
parser.add_argument('Mask_what_former', type=int,
help='Mask what former, can be a `1` or `2`',
choices=[1, 2])
parser.add_argument('CFG_FILE', help='config file path')
parser.add_argument('OLD_CKPT_FILEPATH', help='old ckpt file path')
parser.add_argument('NEW_CKPT_FILEPATH', help='new ckpt file path')
args = parser.parse_args()
return args
args = parse_args()
def get_new_name(old_name: str):
new_name = old_name
if 'encoder.layers' in new_name:
new_name = new_name.replace('attentions.0', 'self_attn')
new_name = new_name.replace('ffns.0', 'ffn')
if 'decoder.layers' in new_name:
if args.Mask_what_former == 2:
# for Mask2Former
new_name = new_name.replace('attentions.0', 'cross_attn')
new_name = new_name.replace('attentions.1', 'self_attn')
else:
# for Mask2Former
new_name = new_name.replace('attentions.0', 'self_attn')
new_name = new_name.replace('attentions.1', 'cross_attn')
return new_name
def cvt_sd(old_sd: OrderedDict):
new_sd = OrderedDict()
for name, param in old_sd.items():
new_name = get_new_name(name)
assert new_name not in new_sd
new_sd[new_name] = param
assert len(new_sd) == len(old_sd)
return new_sd
if __name__ == '__main__':
cfg = Config.fromfile(args.CFG_FILE)
model_cfg = cfg.model
segmentor = build_segmentor(model_cfg)
refer_sd = segmentor.state_dict()
old_ckpt = torch.load(args.OLD_CKPT_FILEPATH)
old_sd = old_ckpt['state_dict']
new_sd = cvt_sd(old_sd)
print(segmentor.load_state_dict(new_sd))
new_ckpt = deepcopy(old_ckpt)
new_ckpt['state_dict'] = new_sd
torch.save(new_ckpt, args.NEW_CKPT_FILEPATH)
print(f'{args.NEW_CKPT_FILEPATH} has been saved!')
```
Usage:
```bash
# for example
python ckpt4pr2532.py 1 configs/maskformer/maskformer_r50-d32_8xb2-160k_ade20k-512x512.py original_ckpts/maskformer_r50-d32_8xb2-160k_ade20k-512x512_20221030_182724-cbd39cc1.pth cvt_outputs/maskformer_r50-d32_8xb2-160k_ade20k-512x512_20221030_182724.pth
python ckpt4pr2532.py 2 configs/mask2former/mask2former_r50_8xb2-160k_ade20k-512x512.py original_ckpts/mask2former_r50_8xb2-160k_ade20k-512x512_20221204_000055-4c62652d.pth cvt_outputs/mask2former_r50_8xb2-160k_ade20k-512x512_20221204_000055.pth
```
---------
Co-authored-by: MeowZheng <meowzheng@outlook.com>
* tta init
* use mmcv transform
* test city
* add multiscale
* fix merge
* add softmax to post process
* add ut
* add tta pipeline to other datasets
* remove softmax
* add encoder_decoder_tta ut
* add encoder_decoder_tta ut
* rename
* rename file
* rename config
* rm aug_test
* move flip to post process
* fix channel
* [WIP] Refactor data flow
* model return
* [WIP] Refactor data flow
* support data_samples is optional
* fix benchmark
* fix base
* minors
* rebase
* fix api
* ut
* fix api inference
* comments
* docstring
* docstring
* docstring
* fix bug of slide inference
* add assert c > 1
* [Fix] Add avg_non_ignore in cross entropy loss
* [Fix] Add avg_non_ignore in cross entropy loss
* add docstring
* fix ut
* fix docstring and comments
* fix
* fix bce
* fix avg_factor in BCE and add more ut
* add avg_non_ignore
* add more ut
* fix part of ut
* fix part of ut
* test avg_non_ignore would not affect ce/bce when reduction none/sum
* test avg_non_ignore would not affect ce/bce when reduction none/sum/mean
* re-organize ut
* re-organize ut
* re-organize ut
* re-organize hardcode case
* fix parts of comments
* fix another parts of comments
* fix
* knet first commit
* fix import error in knet
* remove kernel update head from decoder head
* [Feature] Add kenerl updation for some decoder heads.
* [Feature] Add kenerl updation for some decoder heads.
* directly use forward_feature && modify other 3 decoder heads
* remover kernel_update attr
* delete unnecessary variables in forward function
* delete kernel update function
* delete kernel update function
* delete kernel_generate_head
* add unit test & comments in knet.py
* add copyright to fix lint error
* modify config names of knet
* rename swin-l 640
* upload models&logs and refactor knet_head.py
* modify docstrings and add some ut
* add url, modify docstring and add loss ut
* modify docstrings
* [Fix] Fix the bug that when all pixels in an image is ignored, the accuracy calculation raises ZeroDivisionError
* use eps
* all close
* add ignore test
* add eps
* assert original HardSwish when PyTorch > 1.6 in unit test
* assert original HardSwish when PyTorch > 1.6 in unit test
* assert original HardSwish when PyTorch > 1.6 in unit test
* assert original HardSwish when PyTorch > 1.6 in unit test
* assert original HardSwish when PyTorch > 1.6 in unit test
* assert original HardSwish when PyTorch > 1.6 in unit test
* [Feature] add focal loss
* fix the bug of 'non' reduction type
* refine the implementation
* add class_weight and ignore_index; support different alpha values for different classes
* fixed some bugs
* fix bugs
* add comments
* modify test
* Update mmseg/models/losses/focal_loss.py
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
* update test_focal_loss.py
* modified the implementation
* Update mmseg/models/losses/focal_loss.py
Co-authored-by: Jerry Jiarui XU <xvjiarui0826@gmail.com>
* update focal_loss.py
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
Co-authored-by: Jerry Jiarui XU <xvjiarui0826@gmail.com>
* [Fix] Fix the bug that vit cannot load pretrain properly when using init_cfg to specify the pretrain scheme
* [Fix] fix the coverage problem
* Update mmseg/models/backbones/vit.py
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
* [Fix] make the predicate more concise and clearer
* [Fix] Modified the judgement logic
* Update tests/test_models/test_backbones/test_vit.py
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
* add comments
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
* add TIMMBackbone and unittests
* add timm to tests requirements
* deprecate pt1.3.1
* reduce the unittests input of timm backbone
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* fix ci
* remove unittests of large models of timm backbone
* generate coverage report for all unittests env
* reduce the unittests input of timm backbone
* reduce the unittests input of timm backbone
* fix single loss type
* fix error in ohem & point_head
* fix coverage miss
* fix uncoverage error of PointHead loss
* fix coverage miss
* fix uncoverage error of PointHead loss
* nn.modules.container.ModuleList to nn.ModuleList
* more simple format
* merge unittest def
