History

MengzhangLI 2e28db0dcb [Feature] Support K-Net (#1289 ) * knet first commit * fix import error in knet * remove kernel update head from decoder head * [Feature] Add kenerl updation for some decoder heads. * [Feature] Add kenerl updation for some decoder heads. * directly use forward_feature && modify other 3 decoder heads * remover kernel_update attr * delete unnecessary variables in forward function * delete kernel update function * delete kernel update function * delete kernel_generate_head * add unit test & comments in knet.py * add copyright to fix lint error * modify config names of knet * rename swin-l 640 * upload models&logs and refactor knet_head.py * modify docstrings and add some ut * add url, modify docstring and add loss ut * modify docstrings		2022-03-10 20:05:37 +08:00
..
README.md	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00
knet.yml	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00
knet_s3_deeplabv3_r50-d8_8x2_512x512_adamw_80k_ade20k.py	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00
knet_s3_fcn_r50-d8_8x2_512x512_adamw_80k_ade20k.py	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00
knet_s3_pspnet_r50-d8_8x2_512x512_adamw_80k_ade20k.py	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00
knet_s3_upernet_r50-d8_8x2_512x512_adamw_80k_ade20k.py	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00
knet_s3_upernet_swin-l_8x2_512x512_adamw_80k_ade20k.py	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00
knet_s3_upernet_swin-l_8x2_640x640_adamw_80k_ade20k.py	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00
knet_s3_upernet_swin-t_8x2_512x512_adamw_80k_ade20k.py	[Feature] Support K-Net (#1289 )	2022-03-10 20:05:37 +08:00

README.md

K-Net

K-Net: Towards Unified Image Segmentation

Introduction

Official Repo

Code Snippet

Abstract

Semantic, instance, and panoptic segmentations have been addressed using different and specialized frameworks despite their underlying connections. This paper presents a unified, simple, and effective framework for these essentially similar tasks. The framework, named K-Net, segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class. To remedy the difficulties of distinguishing various instances, we propose a kernel update strategy that enables each kernel dynamic and conditional on its meaningful group in the input image. K-Net can be trained in an end-to-end manner with bipartite matching, and its training and inference are naturally NMS-free and box-free. Without bells and whistles, K-Net surpasses all previous published state-of-the-art single-model results of panoptic segmentation on MS COCO test-dev split and semantic segmentation on ADE20K val split with 55.2% PQ and 54.3% mIoU, respectively. Its instance segmentation performance is also on par with Cascade Mask R-CNN on MS COCO with 60%-90% faster inference speeds. Code and models will be released at this https URL.

@inproceedings{zhang2021knet,
    title={{K-Net: Towards} Unified Image Segmentation},
    author={Wenwei Zhang and Jiangmiao Pang and Kai Chen and Chen Change Loy},
    year={2021},
    booktitle={NeurIPS},
}

Results and models

ADE20K

Method	Backbone	Crop Size	Lr schd	Mem (GB)	Inf time (fps)	mIoU	mIoU(ms+flip)	config	download
KNet + FCN	R-50-D8	512x512	80000	7.01	19.24	43.60	45.12	config	model \| log
KNet + PSPNet	R-50-D8	512x512	80000	6.98	20.04	44.18	45.58	config	model \| log
KNet + DeepLabV3	R-50-D8	512x512	80000	7.42	12.10	45.06	46.11	config	model \| log
KNet + UperNet	R-50-D8	512x512	80000	7.34	17.11	43.45	44.07	config	model \| log
KNet + UperNet	Swin-T	512x512	80000	7.57	15.56	45.84	46.27	config	model \| log
KNet + UperNet	Swin-L	512x512	80000	13.5	8.29	52.05	53.24	config	model \| log
KNet + UperNet	Swin-L	640x640	80000	13.54	8.29	52.21	53.34	config	model \| log

Note:

All experiments of K-Net are implemented with 8 V100 (32G) GPUs with 2 samplers per GPU.