mmselfsup/docs/MODEL_ZOO.md

Model Zoo

OpenSelfSup needs your contribution! Since we don't have sufficient GPUs to run these large-scale experiments, your contributions, including parameter studies, reproducing of results, implementing new methods, etc, are essential to make OpenSelfSup better. Your contribution will be recorded in the below table, top contributors will be included in the author list of OpenSelfSup!

Pre-trained model download links and speed test.

Note

• If not specifically indicated, the testing GPUs are NVIDIA Tesla V100.
• Experiments with the same batch size are directly comparable in speed.
• The table records the implementors who implemented the methods (either by themselves or refactoring from other repos), and the experimenters who performed experiments and reproduced the results. The experimenters should be responsible for the evaluation results on all the benchmarks, and the implementors should be responsible for the implementation as well as the results; If the experimenter is not indicated, an implementator is the experimenter by default.

Method (Implementator)	Config (Experimenter)	Remarks	Download link	Batch size	Epochs	Time per epoch
ImageNet	-	torchvision	imagenet_r50-21352794.pth	-	-	-
Random	-	kaiming	random_r50-5d0fa71b.pth	-	-	-
Relative-Loc (@Jiahao000)	selfsup/relative_loc/r50.py	default	relative_loc_r50-342c9097.pth	512	70	21min17s
Rotation-Pred (@XiaohangZhan)	selfsup/rotation_pred/r50.py	default	rotation_r50-cfab8ebb.pth	128	70	49min58s
DeepCluster (@XiaohangZhan)	selfsup/deepcluster/r50.py	default	deepcluster_r50-bb8681e2.pth	512	200	41min57s
NPID (@XiaohangZhan)	selfsup/npid/r50.py	default	npid_r50-dec3df0c.pth	256	200	20min5s
	selfsup/npid/r50_ensure_neg.py	ensure_neg=True	npid_r50_ensure_neg-ce09b7ae.pth
ODC (@XiaohangZhan)	selfsup/odc/r50_v1.py (@Jiahao000)	default	odc_r50_v1-5af5dd0c.pth	512	440	28min22s
MoCo (@XiaohangZhan)	selfsup/moco/r50_v1.py	default	moco_r50_v1-4ad89b5c.pth	256	200	22min58s
MoCo v2 (@XiaohangZhan)	selfsup/moco/r50_v2.py	default	moco_r50_v2-e3b0c442.pth	256	200	25min7s
SimCLR (@XiaohangZhan)	selfsup/simclr/r50_bs256_ep200.py	default	simclr_r50_bs256_ep200-4577e9a6.pth	256	200	32min13s
	selfsup/simclr/r50_bs256_ep200_mocov2_neck.py	-> MoCo v2 neck	simclr_r50_bs256_ep200_mocov2_neck-0d6e5ff2.pth
BYOL (@XiaohangZhan)	selfsup/byol/r50_bs4096_ep200.py (@xieenze)	default	byol_r50-e3b0c442.pth	4096 (256 for speed test)	200	30min57s
BYOL(@scnuhealthy)	selfsup/byol/r50_bs256_accumulate16_ep300.py	default	byol_r50_bs256_accmulate16_ep300-5df46722.pth	256	300

Benchmarks

VOC07 SVM & SVM Low-shot

Method	Config	Remarks	Best layer	VOC07 SVM	VOC07 SVM Low-shot
					1	2	4	8	16	32	64	96
ImageNet	-	torchvision	feat5	87.17	52.99	63.55	73.7	78.79	81.76	83.75	85.18	85.97
Random	-	kaiming	feat2	30.54	9.15	9.39	11.09	12.3	14.3	17.41	21.32	23.77
Relative-Loc	selfsup/relative_loc/r50.py	default	feat4	64.78	18.17	22.08	29.37	35.58	41.8	48.73	55.55	58.33
Rotation-Pred	selfsup/rotation_pred/r50.py	default	feat4	67.38	18.91	23.33	30.57	38.22	45.83	52.23	58.08	61.11
DeepCluster	selfsup/deepcluster/r50.py	default	feat5	74.26	29.73	37.66	45.85	55.57	62.48	66.15	70.0	71.37
NPID	selfsup/npid/r50.py	default	feat5	74.50	24.19	31.24	39.69	50.99	59.03	64.4	68.69	70.84
	selfsup/npid/r50_ensure_neg.py	ensure_neg=True	feat5	75.70
ODC	selfsup/odc/r50_v1.py	default	feat5	78.42	32.42	40.27	49.95	59.96	65.71	69.99	73.64	75.13
MoCo	selfsup/moco/r50_v1.py	default	feat5	79.18	30.03	37.73	47.64	58.78	66.0	70.6	74.6	76.07
MoCo v2	selfsup/moco/r50_v2.py	default	feat5	84.26	43.0	52.48	63.43	71.74	76.35	78.9	81.31	82.45
SimCLR	selfsup/simclr/r50_bs256_ep200.py	default	feat5	78.95	32.45	40.76	50.4	59.01	65.45	70.13	73.58	75.35
	selfsup/simclr/r50_bs256_ep200_mocov2_neck.py	-> MoCo v2 neck	feat5	77.65
BYOL	selfsup/byol/r50_bs4096_ep200.py	default	feat5	85.10	44.48	52.09	62.88	70.87	76.18	79.45	81.88	83.08

ImageNet Linear Classification

Note

• Config: configs/benchmarks/linear_classification/imagenet/r50_multihead.py for ImageNet (Multi) and configs/benchmarks/linear_classification/imagenet/r50_last.py for ImageNet (Last).
• For DeepCluster, use the corresponding one with _sobel.
• ImageNet (Multi) evaluates features in around 9k dimensions from different layers. Top-1 result of the last epoch is reported.
• ImageNet (Last) evaluates the last feature after global average pooling, e.g., 2048 dimensions for resnet50. The best top-1 result among all epochs is reported.
• Usually, we report the best result from ImageNet (Multi) and ImageNet (Last) to ensure fairness, since different methods achieve their best performance on different layers.

Method	Config	Remarks	ImageNet (Multi)					ImageNet (Last)
			feat1	feat2	feat3	feat4	feat5	avgpool
ImageNet	-	torchvision	15.18	33.96	47.86	67.56	76.17	74.12
Random	-	kaiming	11.37	16.21	13.47	9.07	6.54	4.35
Relative-Loc	selfsup/relative_loc/r50.py	default	14.76	31.29	45.77	49.31	40.20	38.83
Rotation-Pred	selfsup/rotation_pred/r50.py	default	12.89	34.30	44.91	54.99	49.09	47.01
DeepCluster	selfsup/deepcluster/r50.py	default	12.78	30.81	43.88	57.71	51.68	46.92
NPID	selfsup/npid/r50.py	default	14.28	31.20	40.68	54.46	56.61	56.60
ODC	selfsup/odc/r50_v1.py	default	14.76	31.82	42.44	55.76	57.70	53.42
MoCo	selfsup/moco/r50_v1.py	default	15.32	33.08	44.68	57.27	60.60	61.02
MoCo v2	selfsup/moco/r50_v2.py	default	14.74	32.81	44.95	61.61	66.73	67.69
SimCLR	selfsup/simclr/r50_bs256_ep200.py	default	17.09	31.37	41.38	54.35	61.57	60.06
	selfsup/simclr/r50_bs256_ep200_mocov2_neck.py	-> MoCo v2 neck	16.97	31.88	41.73	54.33	59.94	58.00
BYOL	selfsup/byol/r50_bs4096_ep200.py	default	16.70	34.22	46.61	60.78	69.14	67.10
BYOL	selfsup/byol/r50_bs256_accumulate16_ep300.py	default	14.07	34.44	47.22	63.08	72.35

Places205 Linear Classification

Note

• Config: configs/benchmarks/linear_classification/places205/r50_multihead.py.
• For DeepCluster, use the corresponding one with _sobel.
• Places205 evaluates features in around 9k dimensions from different layers. Top-1 result of the last epoch is reported.

Method	Config	Remarks	Places205
			feat1	feat2	feat3	feat4	feat5
ImageNet	-	torchvision	21.27	36.10	43.03	51.38	53.05
Random	-	kaiming	17.19	21.70	19.23	14.59	11.73
Relative-Loc	selfsup/relative_loc/r50.py	default	21.07	34.86	42.84	45.71	41.45
Rotation-Pred	selfsup/rotation_pred/r50.py	default	18.65	35.71	42.28	45.98	43.72
DeepCluster	selfsup/deepcluster/r50.py	default	18.80	33.93	41.44	47.22	42.61
NPID	selfsup/npid/r50.py	default	20.53	34.03	40.48	47.13	47.73
ODC	selfsup/odc/r50_v1.py	default	20.94	34.78	41.19	47.45	49.18
MoCo	selfsup/moco/r50_v1.py	default	21.13	35.19	42.40	48.78	50.70
MoCo v2	selfsup/moco/r50_v2.py	default	21.88	35.75	43.65	49.99	52.57
SimCLR	selfsup/simclr/r50_bs256_ep200.py	default	22.55	34.14	40.35	47.15	51.64
	selfsup/simclr/r50_bs256_ep200_mocov2_neck.py	-> MoCo v2 neck
BYOL	selfsup/byol/r50_bs4096_ep200.py	default	22.28	35.95	43.03	49.79	52.75

ImageNet Semi-Supervised Classification

Note

• In this benchmark, the necks or heads are removed and only the backbone CNN is evaluated by appending a linear classification head. All parameters are fine-tuned.
• Config: under configs/benchmarks/semi_classification/imagenet_1percent/ for 1% data, and configs/benchmarks/semi_classification/imagenet_10percent/ for 10% data.
• When training with 1% ImageNet, we find hyper-parameters especially the learning rate greatly influence the performance. Hence, we prepare a list of settings with the base learning rate from {0.001, 0.01, 0.1} and the learning rate multiplier for the head from {1, 10, 100}. We choose the best performing setting for each method.
• Please use --deterministic in this benchmark.

Method	Config	Remarks	Optimal setting for ImageNet 1%	ImageNet 1%
				top-1	top-5
ImageNet	-	torchvision	r50_lr0_001_head100.py	68.68	88.87
Random	-	kaiming	r50_lr0_01_head1.py	1.56	4.99
Relative-Loc	selfsup/relative_loc/r50.py	default	r50_lr0_01_head100.py	16.48	40.37
Rotation-Pred	selfsup/rotation_pred/r50.py	default	r50_lr0_01_head100.py	18.98	44.05
DeepCluster	selfsup/deepcluster/r50.py	default	r50_lr0_01_head1_sobel.py	33.44	58.62
NPID	selfsup/npid/r50.py	default	r50_lr0_01_head100.py	27.95	54.37
ODC	selfsup/odc/r50_v1.py	default	r50_lr0_1_head100.py	32.39	61.02
MoCo	selfsup/moco/r50_v1.py	default	r50_lr0_01_head100.py	33.15	61.30
MoCo v2	selfsup/moco/r50_v2.py	default	r50_lr0_01_head100.py	39.07	68.31
SimCLR	selfsup/simclr/r50_bs256_ep200.py	default	r50_lr0_01_head100.py	36.09	64.50
	selfsup/simclr/r50_bs256_ep200_mocov2_neck.py	-> MoCo v2 neck	r50_lr0_01_head100.py	36.31	64.68
BYOL	selfsup/byol/r50_bs4096_ep200.py	default	r50_lr0_01_head10.py	49.37	76.75

Method	Config	Remarks	Optimal setting for ImageNet 10%	ImageNet 10%
				top-1	top-5
ImageNet	-	torchvision	r50_lr0_001_head10.py	74.53	92.19
Random	-	kaiming	r50_lr0_01_head1.py	21.78	44.24
Relative-Loc	selfsup/relative_loc/r50.py	default	r50_lr0_01_head100.py	53.86	79.62
Rotation-Pred	selfsup/rotation_pred/r50.py	default	r50_lr0_01_head100.py	54.75	80.21
DeepCluster	selfsup/deepcluster/r50.py	default	r50_lr0_01_head1_sobel.py	52.94	77.96
NPID	selfsup/npid/r50.py	default	r50_lr0_01_head100.py	57.22	81.39
ODC	selfsup/odc/r50_v1.py	default	r50_lr0_1_head10.py	58.15	82.55
MoCo	selfsup/moco/r50_v1.py	default	r50_lr0_01_head100.py	60.08	84.02
MoCo v2	selfsup/moco/r50_v2.py	default	r50_lr0_01_head100.py	61.80	85.11
SimCLR	selfsup/simclr/r50_bs256_ep200.py	default	r50_lr0_01_head100.py	58.46	82.60
	selfsup/simclr/r50_bs256_ep200_mocov2_neck.py	-> MoCo v2 neck	r50_lr0_01_head100.py	58.38	82.53
BYOL	selfsup/byol/r50_bs4096_ep200.py	default	r50_lr0_01_head100.py	65.94	87.81

PASCAL VOC07+12 Object Detection

Note

• This benchmark follows the evluation protocols set up by MoCo.
• Config: benchmarks/detection/configs/pascal_voc_R_50_C4_24k_moco.yaml.
• Please follow here to run the evaluation.

Method	Config	Remarks	VOC07+12
			AP50	AP	AP75
ImageNet	-	torchvision	81.58	54.19	59.80
Random	-	kaiming	59.02	32.83	31.60
Relative-Loc	selfsup/relative_loc/r50.py	default	80.36	55.13	61.18
Rotation-Pred	selfsup/rotation_pred/r50.py	default	80.91	55.52	61.39
NPID	selfsup/npid/r50.py	default	80.03	54.11	59.50
MoCo	selfsup/moco/r50_v1.py	default	81.38	55.95	62.23
MoCo v2	selfsup/moco/r50_v2.py	default	82.24	56.97	63.43
SimCLR	selfsup/simclr/r50_bs256_ep200.py	default	79.41	51.54	55.63
BYOL	selfsup/byol/r50_bs4096_ep200.py	default	80.95	51.87	56.53

COCO2017 Object Detection

Note

• This benchmark follows the evluation protocols set up by MoCo.
• Config: benchmarks/detection/configs/coco_R_50_C4_2x_moco.yaml.
• Please follow here to run the evaluation.

Method	Config	Remarks	COCO2017
			AP50(Box)	AP(Box)	AP75(Box)	AP50(Mask)	AP(Mask)	AP75(Mask)
ImageNet	-	torchvision	59.9	40.0	43.1	56.5	34.7	36.9
Random	-	kaiming	54.6	35.6	38.2	51.5	31.4	33.5
Relative-Loc	selfsup/relative_loc/r50.py	default	59.6	40.0	43.5	56.5	35.0	37.3
Rotation-Pred	selfsup/rotation_pred/r50.py	default	59.3	40.0	43.6	56.0	34.9	37.4
NPID	selfsup/npid/r50.py	default	59.0	39.4	42.8	55.9	34.5	36.6
MoCo	selfsup/moco/r50_v1.py	default	60.5	40.9	44.2	57.1	35.5	37.7
MoCo v2	selfsup/moco/r50_v2.py	default	60.1	40.6	44.0	56.9	35.3	38.0
SimCLR	selfsup/simclr/r50_bs256_ep200.py	default	59.1	39.6	42.9	55.9	34.6	37.1
BYOL	selfsup/byol/r50_bs4096_ep200.py	default	60.5	40.3	43.9	56.8	35.1	37.3