mmpretrain/projects/maskfeat_video

MaskFeat Pre-training with Video

• MaskFeat Pre-training with Video
  • Description
  • Usage
    • Setup Environment
    • Data Preparation
    • Pre-training Commands
      • On Local Single GPU
      • On Multiple GPUs
      • On Multiple GPUs with Slurm
    • Downstream Tasks Commands
      • On Multiple GPUs
      • On Multiple GPUs with Slurm
  • Results
  • Citation
  • Checklist

Description

Author: @fangyixiao18

This is the implementation of MaskFeat with video dataset, like Kinetics400.

Usage

Setup Environment

Requirements:

• MMPretrain >= 1.0.0rc0
• MMAction2 >= 1.0.0rc3

Please refer to Get Started documentation of MMPretrain to finish installation.

Besides, to process the video data, we apply transforms in MMAction2. The instruction to install MMAction2 can be found in Get Started documentation.

Data Preparation

You can refer to the documentation in MMAction2.

Pre-training Commands

At first, you need to add the current folder to PYTHONPATH, so that Python can find your model files. In projects/maskfeat_video/ root directory, please run command below to add it.

export PYTHONPATH=`pwd`:$PYTHONPATH

Then run the following commands to train the model:

On Local Single GPU

# train with mim
mim train mmpretrain ${CONFIG} --work-dir ${WORK_DIR}

# a specific command example
mim train mmpretrain configs/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400.py \
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400/

# train with scripts
python tools/train.py configs/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400.py \
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400/

On Multiple GPUs

# train with mim
# a specific command examples, 8 GPUs here
mim train mmpretrain configs/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400.py \
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400/ \
    --launcher pytorch --gpus 8

# train with scripts
bash tools/dist_train.sh configs/maskfeat_mvit-small_8xb32-amp-coslr-300e_k400.py 8

Note:

• CONFIG: the config files under the directory configs/
• WORK_DIR: the working directory to save configs, logs, and checkpoints

On Multiple GPUs with Slurm

# train with mim
mim train mmpretrain configs/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400.py \
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/ \
    --launcher slurm --gpus 16 --gpus-per-node 8 \
    --partition ${PARTITION}

# train with scripts
GPUS_PER_NODE=8 GPUS=16 bash tools/slurm_train.sh ${PARTITION} maskfeat-video \
    configs/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400.py \
    --work-dir work_dirs/selfsup/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/

Note:

• CONFIG: the config files under the directory configs/
• WORK_DIR: the working directory to save configs, logs, and checkpoints
• PARTITION: the slurm partition you are using

Downstream Tasks Commands

To evaluate the MaskFeat MViT pretrained with MMPretrain, we recommend to run MMAction2:

On Multiple GPUs

# command example for train
mim train mmaction2 ${CONFIG} \
    --work-dir ${WORK_DIR} \
    --launcher pytorch -gpus 8 \
    --cfg-options model.backbone.init_cfg.type=Pretrained \
    model.backbone.init_cfg.checkpoint=${CHECKPOINT} \
    model.backbone.init_cfg.prefix="backbone." \
    ${PY_ARGS}
    [optional args]

mim train mmaction2 configs/mvit-small_ft-8xb8-coslr-100e_k400.py \
    --work-dir work_dirs/benchmarks/maskfeat/training_maskfeat-mvit-k400/ \
    --launcher pytorch -gpus 8 \
    --cfg-options model.backbone.init_cfg.type=Pretrained \
    model.backbone.init_cfg.checkpoint=https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400_20230131-87d60b6f.pth \
    model.backbone.init_cfg.prefix="backbone." \
    $PY_ARGS

# command example for test
mim test mmaction2 configs/mvit-small_ft-8xb16-coslr-100e_k400.py \
  --checkpoint https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/mvit-small_ft-8xb16-coslr-100e_k400/mvit-small_ft-8xb16-coslr-100e_k400_20230131-5e8303f5.pth \
  --work-dir work_dirs/benchmarks/maskfeat/maskfeat-mvit-k400/test/ \
  --launcher pytorch --gpus 8

On Multiple GPUs with Slurm

mim train mmaction2 ${CONFIG} \
    --work-dir ${WORK_DIR} \
    --launcher slurm --gpus 8 --gpus-per-node 8 \
    --partition ${PARTITION} \
    --cfg-options model.backbone.init_cfg.type=Pretrained \
    model.backbone.init_cfg.checkpoint=$CHECKPOINT \
    model.backbone.init_cfg.prefix="backbone." \
    $PY_ARGS

mim test mmaction2 ${CONFIG} \
    --checkpoint https://download.openmmlab.com/mmselfsup/1.x/maskfeat/maskfeat_mvit-small_16xb32-amp-coslr-300e_k400/mvit-small_ft-8xb16-coslr-100e_k400/mvit-small_ft-8xb16-coslr-100e_k400_20230131-5e8303f5.pth
    --work-dir ${WORK_DIR} \
    --launcher slurm --gpus 8 --gpus-per-node 8 \
    --partition ${PARTITION} \
    $PY_ARGS

Note:

• CONFIG: the config files under the directory configs/
• WORK_DIR: the working directory to save configs, logs, and checkpoints
• PARTITION: the slurm partition you are using
• CHECKPOINT: the pretrained checkpoint of MMPretrain saved in working directory, like $WORK_DIR/epoch_300.pth
• PY_ARGS: other optional args

Results

The Fine-tuning results are based on Kinetics400(K400) dataset.

Due to the version of K400 dataset, our pretraining, fine-tuning and the final test results are based on MMAction2 version, which is a little different from PySlowFast version.

Algorithm	Backbone	Epoch	Batch Size	Fine-tuning	Pretrain Links	Fine-tuning Links
MaskFeat	MViT-small	300	512	81.8	config | model | log	config | model | log

Remarks:

• We converted the pretrained model from PySlowFast and run fine-tuning with MMAction2, based on MMAction2 version of K400, we got 81.5 test accuracy. The pretrained model from MMPretrain got 81.8, as provided above.
• We also tested our model on other version of K400, we got 82.1 test accuracy.
• Some other details can be found in MMAction2 MViT page.

Citation

@InProceedings{wei2022masked,
    author    = {Wei, Chen and Fan, Haoqi and Xie, Saining and Wu, Chao-Yuan and Yuille, Alan and Feichtenhofer, Christoph},
    title     = {Masked Feature Prediction for Self-Supervised Visual Pre-Training},
    booktitle = {CVPR},
    year      = {2022},
}

Checklist

Here is a checklist illustrating a usual development workflow of a successful project, and also serves as an overview of this project's progress.

• Milestone 1: PR-ready, and acceptable to be one of the projects/.

  • Finish the code

  • Basic docstrings & proper citation

  • Inference correctness

  • A full README

• Milestone 2: Indicates a successful model implementation.

  • Training-time correctness

• Milestone 3: Good to be a part of our core package!

  • Type hints and docstrings

  • Unit tests

  • Code polishing

  • metafile.yml and README.md

• Refactor and Move your modules into the core package following the codebase's file hierarchy structure.