From f6d6cae4d70f5a7eed2ed360f0fb644dc242967e Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Wed, 1 Jul 2020 13:14:52 +0800 Subject: [PATCH 1/9] Update MODEL_ZOO.md --- docs/MODEL_ZOO.md | 101 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 98 insertions(+), 3 deletions(-) diff --git a/docs/MODEL_ZOO.md b/docs/MODEL_ZOO.md index 53f03290..76dc6d37 100644 --- a/docs/MODEL_ZOO.md +++ b/docs/MODEL_ZOO.md @@ -2,17 +2,112 @@ ## Pre-trained model download links -
MethodConfigRemarksDownload link
ImageNet-torchvisionimagenet_r50-21352794.pth
Random-kaimingrandom_r50-5d0fa71b.pth
Relative-Locselfsup/relative_loc/r50.pydefault
Rotation-Predselfsup/rotation_pred/r50.pydefaultrotation_r50-cfab8ebb.pth
DeepClusterselfsup/deepcluster/r50.pydefaultdeepcluster_r50-bb8681e2.pth
NPIDselfsup/npid/r50.pydefaultnpid_r50-dec3df0c.pth
ODCselfsup/odc/r50_v1.pydefaultodc_r50_v1-5af5dd0c.pth
MoCoselfsup/moco/r50_v1.pydefaultmoco_r50_v1-4ad89b5c.pth
MoCo v2selfsup/moco/r50_v2.pydefaultmoco_r50_v2-58f10cfe.pth
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
moco_r50_v2_simclr_neck-70379356.pth
SimCLRselfsup/simclr/r50_bs256_ep200.pydefaultsimclr_r50_bs256_ep200-4577e9a6.pth
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 necksimclr_r50_bs256_ep200_mocov2_neck-0d6e5ff2.pth
BYOLselfsup/byol/r50.pydefault
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MethodConfigRemarksDownload link
ImageNet-torchvisionimagenet_r50-21352794.pth
Random-kaimingrandom_r50-5d0fa71b.pth
Relative-Locselfsup/relative_loc/r50.pydefault
Rotation-Predselfsup/rotation_pred/r50.pydefaultrotation_r50-cfab8ebb.pth
DeepClusterselfsup/deepcluster/r50.pydefaultdeepcluster_r50-bb8681e2.pth
NPIDselfsup/npid/r50.pydefaultnpid_r50-dec3df0c.pth
selfsup/npid/r50_ensure_neg.pydefaultnpid_r50_ensure_neg-ce09b7ae.pth
ODCselfsup/odc/r50_v1.pydefaultodc_r50_v1-5af5dd0c.pth
MoCoselfsup/moco/r50_v1.pydefaultmoco_r50_v1-4ad89b5c.pth
MoCo v2selfsup/moco/r50_v2.pydefaultmoco_r50_v2-58f10cfe.pth
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
moco_r50_v2_simclr_neck-70379356.pth
SimCLRselfsup/simclr/r50_bs256_ep200.pydefaultsimclr_r50_bs256_ep200-4577e9a6.pth
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 necksimclr_r50_bs256_ep200_mocov2_neck-0d6e5ff2.pth
BYOLselfsup/byol/r50.pydefault
## Benchmarks ### VOC07 SVM & SVM Low-shot -
MethodConfigRemarksBest layerVOC07 SVMVOC07 SVM Low-shot
124816326496
ImageNet-torchvisionfeat587.1752.9963.5573.778.7981.7683.7585.1885.97
Random-kaimingfeat230.22
Relative-Locfeat5
Rotation-Predselfsup/rotation_pred/r50.pydefaultfeat467.38
DeepClusterselfsup/deepcluster/r50.pydefaultfeat574.26
NPIDselfsup/npid/r50.pydefaultfeat574.50
ODCselfsup/odc/r50_v1.pydefaultfeat578.42
MoCoselfsup/moco/r50_v1.pydefaultfeat579.18
MoCo v2selfsup/moco/r50_v2.pydefaultfeat584.05
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
feat584.00
SimCLRselfsup/simclr/r50_bs256_ep200.pydefaultfeat578.95
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 neckfeat577.65
BYOLselfsup/byol/r50.pydefault
+
MethodConfigRemarksBest layerVOC07 SVMVOC07 SVM Low-shot
124816326496
ImageNet-torchvisionfeat587.1752.9963.5573.778.7981.7683.7585.1885.97
Random-kaimingfeat230.22
Relative-Locfeat5
Rotation-Predselfsup/rotation_pred/r50.pydefaultfeat467.38
DeepClusterselfsup/deepcluster/r50.pydefaultfeat574.26
NPIDselfsup/npid/r50.pydefaultfeat574.50
selfsup/npid/r50_ensure_neg.pyensure_neg=Truefeat575.70
ODCselfsup/odc/r50_v1.pydefaultfeat578.42
MoCoselfsup/moco/r50_v1.pydefaultfeat579.18
MoCo v2selfsup/moco/r50_v2.pydefaultfeat584.05
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
feat584.00
SimCLRselfsup/simclr/r50_bs256_ep200.pydefaultfeat578.95
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 neckfeat577.65
BYOLselfsup/byol/r50.pydefault
### ImageNet Linear Classification -
MethodConfigRemarksImageNet (Multi)ImageNet (Last)
feat1feat2feat3feat4feat5avgpool
ImageNet-torchvision15.1833.9647.8667.5676.1774.12
Random-kaiming4.35
Relative-Locselfsup/relative_loc/r50.pydefault
Rotation-Predselfsup/rotation_pred/r50.pydefault12.8934.3044.9154.9949.09
DeepClusterselfsup/deepcluster/r50.pydefault46.92
NPIDselfsup/npid/r50.pydefault14.2831.2040.6854.4656.61
ODCselfsup/odc/r50_v1.pydefault14.7531.5542.4955.7257.57
MoCoselfsup/moco/r50_v1.pydefault15.3233.0844.6857.2760.6061.02
MoCo v2selfsup/moco/r50_v2.pydefault15.3534.5745.8160.9666.7267.02
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
SimCLRselfsup/simclr/r50_bs256_ep200.pydefault
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 neck
BYOLselfsup/byol/r50.pydefault
+
MethodConfigRemarksImageNet (Multi)ImageNet (Last)
feat1feat2feat3feat4feat5avgpool
ImageNet-torchvision15.1833.9647.8667.5676.1774.12
Random-kaiming4.35
Relative-Locselfsup/relative_loc/r50.pydefault
Rotation-Predselfsup/rotation_pred/r50.pydefault12.8934.3044.9154.9949.0947.01
DeepClusterselfsup/deepcluster/r50.pydefault46.92
NPIDselfsup/npid/r50.pydefault14.2831.2040.6854.4656.6156.60
ODCselfsup/odc/r50_v1.pydefault14.7631.8242.4455.7657.7053.42
MoCoselfsup/moco/r50_v1.pydefault15.3233.0844.6857.2760.6061.02
MoCo v2selfsup/moco/r50_v2.pydefault15.3534.5745.8160.9666.7267.02
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
SimCLRselfsup/simclr/r50_bs256_ep200.pydefault17.0931.3741.3854.3561.5760.06
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 neck16.9731.8841.7354.3359.9458.00
BYOLselfsup/byol/r50.pydefault
### Place Linear Classification From 897991486fc42c672f987d1041b098cd8838beb2 Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Wed, 1 Jul 2020 17:32:24 +0800 Subject: [PATCH 2/9] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index a5811bc0..e1760abb 100644 --- a/README.md +++ b/README.md @@ -53,6 +53,7 @@ For existing methods in this repo, you only need to modify config files to adjus Please refer to [CHANGELOG.md](docs/CHANGELOG.md) for details and release history. +[2020-06-26] `OpenSelfSup` v0.2.0 is released. [2020-06-16] `OpenSelfSup` v0.1.0 is released. ## Installation From e036b44a1f3f2011bf0063b3c3c1f783255038d1 Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Wed, 1 Jul 2020 20:01:03 +0800 Subject: [PATCH 3/9] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e1760abb..0f802b65 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,8 @@ For existing methods in this repo, you only need to modify config files to adjus Please refer to [CHANGELOG.md](docs/CHANGELOG.md) for details and release history. -[2020-06-26] `OpenSelfSup` v0.2.0 is released. +[2020-06-26] `OpenSelfSup` v0.2.0 is released with benchmark results and support of new features. + [2020-06-16] `OpenSelfSup` v0.1.0 is released. ## Installation From 1e075fbc283f4014697b56ff1bdd5c3e20c3672f Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Sat, 4 Jul 2020 15:13:57 +0800 Subject: [PATCH 4/9] Update GETTING_STARTED.md --- docs/GETTING_STARTED.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index 0248e97e..1e50fdf6 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -1,4 +1,4 @@ -# Getting Started +se# Getting Started This page provides basic tutorials about the usage of OpenSelfSup. For installation instructions, please see [INSTALL.md](INSTALL.md). @@ -118,7 +118,7 @@ Where are the checkpoints and logs? E.g., if you use `configs/benchmarks/linear_ ```shell # train -bash benchmarks/dist_train_linear.sh ${CONFIG_FILE} ${WEIGHT_FILE} [optional arguments] +bash benchmarks/dist_train_semi.sh ${CONFIG_FILE} ${WEIGHT_FILE} [optional arguments] # test (unnecessary if have validation in training) bash tools/dist_test.sh ${CONFIG_FILE} ${GPUS} ${CHECKPOINT} ``` From c1db3416390f8ed839048501aeb44a76882f785e Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Sat, 4 Jul 2020 15:17:14 +0800 Subject: [PATCH 5/9] Update GETTING_STARTED.md --- docs/GETTING_STARTED.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index 1e50fdf6..998f12e1 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -85,7 +85,7 @@ Augments: Working directories: The features, logs and intermediate files generated are saved in `$SVM_WORK_DIR/` as follows: - `dist_test_svm_epoch.sh`: `SVM_WORK_DIR=$WORK_DIR/` (The same as that mentioned in `Train with single/multiple GPUs` above.) Hence, the files will be overridden to save space when evaluating with a new `$EPOCH`. -- `dist_test_svm_pretrain.sh`: `SVM_WORK_DIR=$WORK_DIR/$PRETRAIN_NAME/`, e.g., if `PRETRAIN=pretrains/odc_v1.pth`, then `PRETRAIN_NAME=odc_v1.pth`; if `PRETRAIN=random`, then `PRETRAIN_NAME=random`. +- `dist_test_svm_pretrain.sh`: `SVM_WORK_DIR=$WORK_DIR/$PRETRAIN_NAME/`, e.g., if `PRETRAIN=pretrains/odc_r50_v1-5af5dd0c.pth`, then `PRETRAIN_NAME=odc_r50_v1-5af5dd0c.pth`; if `PRETRAIN=random`, then `PRETRAIN_NAME=random`. The evaluation records are saved in `$SVM_WORK_DIR/logs/eval_svm.log` ### ImageNet / Places205 Linear Classification @@ -96,7 +96,7 @@ python tools/extract_backbone_weights.py ${CHECKPOINT} ${WEIGHT_FILE} ``` Arguments: - `CHECKPOINTS`: the checkpoint file of a selfsup method named as `epoch_*.pth`. -- `WEIGHT_FILE`: the output backbone weights file, e.g., `pretrains/moco_v1_epoch200.pth`. +- `WEIGHT_FILE`: the output backbone weights file, e.g., `pretrains/moco_r50_v1-4ad89b5c.pth`. **Next**, train and test linear classification: ```shell @@ -112,7 +112,7 @@ Augments: - `--deterministic`: Switch on "deterministic" mode which slows down training but the results are reproducible. Working directories: -Where are the checkpoints and logs? E.g., if you use `configs/benchmarks/linear_classification/imagenet/r50_multihead.py` to evaluate `pretrains/moco_v1_epoch200.pth`, then the working directories for this evalution is `work_dirs/benchmarks/linear_classification/imagenet/r50_multihead/moco_v1_epoch200.pth/`. +Where are the checkpoints and logs? E.g., if you use `configs/benchmarks/linear_classification/imagenet/r50_multihead.py` to evaluate `pretrains/moco_r50_v1-4ad89b5c.pth`, then the working directories for this evalution is `work_dirs/benchmarks/linear_classification/imagenet/r50_multihead/moco_r50_v1-4ad89b5c.pth/`. ### ImageNet Semi-Supervised Classification From 21a4fb38369da3df3dd5b8299783947029c6f090 Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Sat, 4 Jul 2020 15:20:10 +0800 Subject: [PATCH 6/9] Update MODEL_ZOO.md --- docs/MODEL_ZOO.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/MODEL_ZOO.md b/docs/MODEL_ZOO.md index 76dc6d37..79ebd26a 100644 --- a/docs/MODEL_ZOO.md +++ b/docs/MODEL_ZOO.md @@ -107,6 +107,10 @@ ### ImageNet Linear Classification +**NOte** +* ImageNet (Multi) evaluates features in around 9k dimensions from different layers. Top-1 result of the last epoch is reported. +* ImageNet (Last) evaluates the last feature after global average pooling, e.g., 2048 dimensions for resnet50. The best top-1 result among all epochs is reported. +
MethodConfigRemarksImageNet (Multi)ImageNet (Last)
feat1feat2feat3feat4feat5avgpool
ImageNet-torchvision15.1833.9647.8667.5676.1774.12
Random-kaiming4.35
Relative-Locselfsup/relative_loc/r50.pydefault
Rotation-Predselfsup/rotation_pred/r50.pydefault12.8934.3044.9154.9949.0947.01
DeepClusterselfsup/deepcluster/r50.pydefault46.92
NPIDselfsup/npid/r50.pydefault14.2831.2040.6854.4656.6156.60
ODCselfsup/odc/r50_v1.pydefault14.7631.8242.4455.7657.7053.42
MoCoselfsup/moco/r50_v1.pydefault15.3233.0844.6857.2760.6061.02
MoCo v2selfsup/moco/r50_v2.pydefault15.3534.5745.8160.9666.7267.02
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
SimCLRselfsup/simclr/r50_bs256_ep200.pydefault17.0931.3741.3854.3561.5760.06
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 neck16.9731.8841.7354.3359.9458.00
BYOLselfsup/byol/r50.pydefault
### Place Linear Classification From 8e2a540892069570278e3ab16b7a33042013fc4b Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Sat, 4 Jul 2020 15:24:27 +0800 Subject: [PATCH 7/9] Update MODEL_ZOO.md --- docs/MODEL_ZOO.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/MODEL_ZOO.md b/docs/MODEL_ZOO.md index 79ebd26a..c07ec3c3 100644 --- a/docs/MODEL_ZOO.md +++ b/docs/MODEL_ZOO.md @@ -108,6 +108,8 @@ ### ImageNet Linear Classification **NOte** +* Config: `configs/benchmarks/linear_classification/imagenet/r50_multihead.py` for ImageNet (Multi) and `configs/benchmarks/linear_classification/imagenet/r50_moco.py` for ImageNet (Last). +* For DeepCluster, use the corresponding one with `_sobel`. * ImageNet (Multi) evaluates features in around 9k dimensions from different layers. Top-1 result of the last epoch is reported. * ImageNet (Last) evaluates the last feature after global average pooling, e.g., 2048 dimensions for resnet50. The best top-1 result among all epochs is reported. @@ -121,6 +123,7 @@ Coming soon. **Note** * In this benchmark, the necks or heads are removed and only the backbone CNN is evaluated by appending a linear classification head. All parameters are fine-tuned. +* Config: under `configs/benchmarks/semi_classification/imagenet_1percent/` for 1% data, and `configs/benchmarks/semi_classification/imagenet_10percent/` for 10% data. * When training with 1% ImageNet, we find hyper-parameters especially the learning rate greatly influence the performance. Hence, we prepare a list of settings with the base learning rate from \{0.01, 0.1\} and the learning rate multiplier for the head from \{1, 10, 100\}. We choose the best performing setting for each method. * Please use `--deterministic` in this benchmark.
MethodConfigRemarksOptimal setting for ImageNet 1%ImageNet 1%
top-1top-5
ImageNet-torchvisionr50_lr0_01_head1.py63.1085.73
Random-kaimingr50_lr0_01_head1.py1.564.99
Relative-Locselfsup/relative_loc/r50.pydefault
Rotation-Predselfsup/rotation_pred/r50.pydefaultr50_lr0_01_head100.py18.9844.05
DeepClusterselfsup/deepcluster/r50.pydefaultr50_lr0_01_head1_sobel.py33.4458.62
NPIDselfsup/npid/r50.pydefaultr50_lr0_01_head100.py27.9554.37
ODCselfsup/odc/r50_v1.pydefaultr50_lr0_1_head100.py32.3961.02
MoCoselfsup/moco/r50_v1.pydefaultr50_lr0_01_head100.py33.1561.30
MoCo v2selfsup/moco/r50_v2.pydefaultr50_lr0_01_head100.py38.7167.90
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
SimCLRselfsup/simclr/r50_bs256_ep200.pydefaultr50_lr0_01_head100.py36.0964.50
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 neck
BYOLselfsup/byol/r50.pydefault
From 14622792def4fccf51bdddc3242f03991ad1e1a3 Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Mon, 6 Jul 2020 21:49:17 +0800 Subject: [PATCH 8/9] Update MODEL_ZOO.md --- docs/MODEL_ZOO.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/MODEL_ZOO.md b/docs/MODEL_ZOO.md index c07ec3c3..889116ef 100644 --- a/docs/MODEL_ZOO.md +++ b/docs/MODEL_ZOO.md @@ -124,8 +124,11 @@ Coming soon. **Note** * In this benchmark, the necks or heads are removed and only the backbone CNN is evaluated by appending a linear classification head. All parameters are fine-tuned. * Config: under `configs/benchmarks/semi_classification/imagenet_1percent/` for 1% data, and `configs/benchmarks/semi_classification/imagenet_10percent/` for 10% data. -* When training with 1% ImageNet, we find hyper-parameters especially the learning rate greatly influence the performance. Hence, we prepare a list of settings with the base learning rate from \{0.01, 0.1\} and the learning rate multiplier for the head from \{1, 10, 100\}. We choose the best performing setting for each method. +* When training with 1% ImageNet, we find hyper-parameters especially the learning rate greatly influence the performance. Hence, we prepare a list of settings with the base learning rate from \{0.001, 0.01, 0.1\} and the learning rate multiplier for the head from \{1, 10, 100\}. We choose the best performing setting for each method. * Please use `--deterministic` in this benchmark. + +
MethodConfigRemarksOptimal setting for ImageNet 1%ImageNet 1%
top-1top-5
ImageNet-torchvisionr50_lr0_001_head100.py68.6888.87
Random-kaimingr50_lr0_01_head1.py1.564.99
Relative-Locselfsup/relative_loc/r50.pydefault
Rotation-Predselfsup/rotation_pred/r50.pydefaultr50_lr0_01_head100.py18.9844.05
DeepClusterselfsup/deepcluster/r50.pydefaultr50_lr0_01_head1_sobel.py33.4458.62
NPIDselfsup/npid/r50.pydefaultr50_lr0_01_head100.py27.9554.37
ODCselfsup/odc/r50_v1.pydefaultr50_lr0_1_head100.py32.3961.02
MoCoselfsup/moco/r50_v1.pydefaultr50_lr0_01_head100.py33.1561.30
MoCo v2selfsup/moco/r50_v2.pydefaultr50_lr0_01_head100.py38.7167.90
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
SimCLRselfsup/simclr/r50_bs256_ep200.pydefaultr50_lr0_01_head100.py36.0964.50
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 neck
BYOLselfsup/byol/r50.pydefault
+
MethodConfigRemarksOptimal setting for ImageNet 1%ImageNet 1%
top-1top-5
ImageNet-torchvisionr50_lr0_01_head1.py63.1085.73
Random-kaimingr50_lr0_01_head1.py1.564.99
Relative-Locselfsup/relative_loc/r50.pydefault
Rotation-Predselfsup/rotation_pred/r50.pydefaultr50_lr0_01_head100.py18.9844.05
DeepClusterselfsup/deepcluster/r50.pydefaultr50_lr0_01_head1_sobel.py33.4458.62
NPIDselfsup/npid/r50.pydefaultr50_lr0_01_head100.py27.9554.37
ODCselfsup/odc/r50_v1.pydefaultr50_lr0_1_head100.py32.3961.02
MoCoselfsup/moco/r50_v1.pydefaultr50_lr0_01_head100.py33.1561.30
MoCo v2selfsup/moco/r50_v2.pydefaultr50_lr0_01_head100.py38.7167.90
selfsup/moco/r50_v2_simclr_neck.py-> SimCLR neck
SimCLRselfsup/simclr/r50_bs256_ep200.pydefaultr50_lr0_01_head100.py36.0964.50
selfsup/simclr/r50_bs256_ep200_mocov2_neck.py-> MoCo v2 neck
BYOLselfsup/byol/r50.pydefault
### PASCAL VOC07+12 Object Detection From 526aad6e1a6a21955d291c8d08717ebaf28be11c Mon Sep 17 00:00:00 2001 From: Xiaohang Zhan Date: Mon, 6 Jul 2020 21:54:27 +0800 Subject: [PATCH 9/9] Update README.md --- README.md | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 0f802b65..a00538a7 100644 --- a/README.md +++ b/README.md @@ -20,15 +20,14 @@ Below is the relations among Unsupervised Learning, Self-Supervised Learning and - **All methods in one repository** -* For comprehensive comparison in all benchmarks, refer to [MODEL_ZOO.md](docs/MODEL_ZOO.md). - -
MethodVOC07 SVM (best layer)ImageNet (best layer)
ImageNet87.1776.17
Random30.2213.70
Relative-Loc
Rotation-Pred67.3854.99
DeepCluster74.26
NPID74.5056.61
ODC78.4257.6
MoCo79.1860.60
MoCo v284.0566.72
SimCLR78.95
BYOL
+ For comprehensive comparison in all benchmarks, refer to [MODEL_ZOO.md](docs/MODEL_ZOO.md). +
MethodVOC07 SVM (best layer)ImageNet (best layer)
ImageNet87.1776.17
Random30.2213.70
Relative-Loc
Rotation-Pred67.3854.99
DeepCluster74.26
NPID74.5056.61
ODC78.4257.6
MoCo79.1860.60
MoCo v284.0566.72
SimCLR78.95
BYOL
- **Flexibility & Extensibility** -`OpenSelfSup` follows a similar code architecture of MMDetection while is even more flexible than MMDetection, since OpenSelfSup integrates various self-supervised tasks including classification, joint clustering and feature learning, contrastive learning, tasks with a memory bank, etc. + `OpenSelfSup` follows a similar code architecture of MMDetection while is even more flexible than MMDetection, since OpenSelfSup integrates various self-supervised tasks including classification, joint clustering and feature learning, contrastive learning, tasks with a memory bank, etc. -For existing methods in this repo, you only need to modify config files to adjust hyper-parameters. It is also simple to design your own methods, please refer to [GETTING_STARTED.md](docs/GETTING_STARTED.md). + For existing methods in this repo, you only need to modify config files to adjust hyper-parameters. It is also simple to design your own methods, please refer to [GETTING_STARTED.md](docs/GETTING_STARTED.md). - **Efficiency** @@ -38,16 +37,16 @@ For existing methods in this repo, you only need to modify config files to adjus We standardize the benchmarks including logistic regression, SVM / Low-shot SVM from linearly probed features, semi-supervised classification, and object detection. Below are the setting of these benchmarks. -| Benchmarks | Setting | Remarks | -|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| -| ImageNet Linear Classification (Multi) | [goyal2019scaling](http://openaccess.thecvf.com/content_ICCV_2019/papers/Goyal_Scaling_and_Benchmarking_Self-Supervised_Visual_Representation_Learning_ICCV_2019_paper.pdf) | Evaluate different layers. | -| ImageNet Linear Classification (Last) | [MoCo](http://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf) | Evaluate the last layer after global pooling. | -| Places205 Linear Classification | [goyal2019scaling](http://openaccess.thecvf.com/content_ICCV_2019/papers/Goyal_Scaling_and_Benchmarking_Self-Supervised_Visual_Representation_Learning_ICCV_2019_paper.pdf) | Evaluate different layers. | -| ImageNet Semi-Sup Classification | -| PASCAL VOC07 SVM | [goyal2019scaling](http://openaccess.thecvf.com/content_ICCV_2019/papers/Goyal_Scaling_and_Benchmarking_Self-Supervised_Visual_Representation_Learning_ICCV_2019_paper.pdf) | Costs="1.0,10.0,100.0" to save evaluation time w/o change of results. | -| PASCAL VOC07 Low-shot SVM | [goyal2019scaling](http://openaccess.thecvf.com/content_ICCV_2019/papers/Goyal_Scaling_and_Benchmarking_Self-Supervised_Visual_Representation_Learning_ICCV_2019_paper.pdf) | Costs="1.0,10.0,100.0" to save evaluation time w/o change of results. | -| PASCAL VOC07+12 Object Detection | [MoCo](http://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf) | | -| COCO17 Object Detection | [MoCo](http://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf) | | + | Benchmarks | Setting | Remarks | + |----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| + | ImageNet Linear Classification (Multi) | [goyal2019scaling](http://openaccess.thecvf.com/content_ICCV_2019/papers/Goyal_Scaling_and_Benchmarking_Self-Supervised_Visual_Representation_Learning_ICCV_2019_paper.pdf) | Evaluate different layers. | + | ImageNet Linear Classification (Last) | [MoCo](http://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf) | Evaluate the last layer after global pooling. | + | Places205 Linear Classification | [goyal2019scaling](http://openaccess.thecvf.com/content_ICCV_2019/papers/Goyal_Scaling_and_Benchmarking_Self-Supervised_Visual_Representation_Learning_ICCV_2019_paper.pdf) | Evaluate different layers. | + | ImageNet Semi-Sup Classification | + | PASCAL VOC07 SVM | [goyal2019scaling](http://openaccess.thecvf.com/content_ICCV_2019/papers/Goyal_Scaling_and_Benchmarking_Self-Supervised_Visual_Representation_Learning_ICCV_2019_paper.pdf) | Costs="1.0,10.0,100.0" to save evaluation time w/o change of results. | + | PASCAL VOC07 Low-shot SVM | [goyal2019scaling](http://openaccess.thecvf.com/content_ICCV_2019/papers/Goyal_Scaling_and_Benchmarking_Self-Supervised_Visual_Representation_Learning_ICCV_2019_paper.pdf) | Costs="1.0,10.0,100.0" to save evaluation time w/o change of results. | + | PASCAL VOC07+12 Object Detection | [MoCo](http://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf) | | + | COCO17 Object Detection | [MoCo](http://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf) | | ## Change Log @@ -96,4 +95,4 @@ fair_self_supervision_benchmark](https://github.com/facebookresearch/fair_self_s ## Contact -This repo is currently maintained by Xiaohang Zhan ([@XiaohangZhan](http://github.com/XiaohangZhan)). +This repo is currently maintained by Xiaohang Zhan ([@XiaohangZhan](http://github.com/XiaohangZhan)) and Jiahao Xie ([@Jiahao000](https://github.com/Jiahao000)).