113 lines
5.2 KiB
Markdown
113 lines
5.2 KiB
Markdown
# Preparing Dataset
|
|
|
|
:bangbang: The dataset preparation contains many details, welcome community contribution to fix any bug, Thanks!
|
|
|
|
Our dataloader follows [Detectron2](https://github.com/facebookresearch/detectron2) that contains: <br/>
|
|
(1) [A dataset registrator](datasets/registration) <br/>
|
|
(2) [A dataset mapper](datasets/dataset_mappers) <br/>
|
|
We modify the dataset registration and mapper for custom datasets.
|
|
|
|
## Training Dataset
|
|
We assume all the datasets are stored under:
|
|
```
|
|
.xdecoder_data
|
|
```
|
|
|
|
### COCO (SEEM & X-Decoder)
|
|
|
|
```sh
|
|
# Prepare panoptic_train2017, panoptic_semseg_train2017 exactly the same as [Mask2Fomer](https://github.com/facebookresearch/Mask2Former/tree/main/datasets)
|
|
|
|
# (SEEM & X-Decoder) Download additional logistic and custom annotation files to .xdecoder_data/coco/annotations
|
|
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/caption_class_similarity.pth
|
|
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/captions_train2017_filtrefgumdval_filtvlp.json
|
|
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/grounding_train2017_filtrefgumdval_filtvlp.json
|
|
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/panoptic_train2017_filtrefgumdval_filtvlp.json
|
|
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/refcocog_umd_val.json
|
|
wget https://github.com/peteanderson80/coco-caption/blob/master/annotations/captions_val2014.json
|
|
|
|
# (SEEM) Download LVIS annotations for mask preparation
|
|
wget https://huggingface.co/xdecoder/SEEM/resolve/main/coco_train2017_filtrefgumdval_lvis.json
|
|
```
|
|
|
|
After dataset preparation, the dataset structure would be:
|
|
```
|
|
.xdecoder_data
|
|
└── coco/
|
|
├── train2017/
|
|
├── val2017/
|
|
├── panoptic_train2017/
|
|
├── panoptic_semseg_train2017/
|
|
├── panoptic_val2017/
|
|
├── panoptic_semseg_val2017/
|
|
└── annotations/
|
|
├── refcocog_umd_val.json
|
|
├── captions_val2014.json
|
|
├── panoptic_val2017.json
|
|
├── caption_class_similarity.pth
|
|
├── panoptic_train2017_filtrefgumdval_filtvlp.json
|
|
└── grounding_train2017_filtrefgumdval_filtvlp.json
|
|
└── lvis/
|
|
└── coco_train2017_filtrefgumdval_lvis.json
|
|
```
|
|
|
|
#### 4M Image Text Pairs (X-Decoder)
|
|
We follow the exact data preparation for the image text pairs data with [ViLT](https://github.com/dandelin/ViLT/blob/master/DATA.md).
|
|
```
|
|
# The pretrained arrow file are put under .xdecoder_data/pretrain_arrows_code224 with the following list of files.
|
|
["filtcoco2017val_caption_karpathy_train.arrow", "filtcoco2017val_caption_karpathy_val.arrow", "filtcoco2017val_caption_karpathy_restval.arrow"] + ["code224_vg.arrow"] + [f"code224_sbu_{i}.arrow" for i in range(9)] + [f"code224_conceptual_caption_train_{i}.arrow" for i in range(31)]
|
|
# ["filtcoco2017val_caption_karpathy_train.arrow", "filtcoco2017val_caption_karpathy_val.arrow", "filtcoco2017val_caption_karpathy_restval.arrow"] are originated from ["filtcoco2017val_caption_karpathy_train.arrow", "filtcoco2017val_caption_karpathy_val.arrow", "filtcoco2017val_caption_karpathy_restval.arrow"] with deletion of coco val2017 overlapped images to avoid information leakage.
|
|
```
|
|
|
|
To get quick started:
|
|
```sh
|
|
# Download coco karparthy test set (we hack the training data to be coco_caption_karpathy_test.arrow only for quick start in the codebase)
|
|
wget https://huggingface.co/xdecoder/X-Decoder/resolve/main/coco_caption_karpathy_test.arrow
|
|
```
|
|
|
|
After dataset preparation, the dataset structure would be:
|
|
```
|
|
.xdecoder_data
|
|
└── pretrain_arrows_code224/
|
|
├── coco_caption_karpathy_test.arrow
|
|
├── *filtcoco2017val_caption_karpathy_train.arrow
|
|
├── ...
|
|
├── *code224_vg.arrow
|
|
├── *code224_sbu_0.arrow
|
|
├── ...
|
|
├── *code224_conceptual_caption_train_0.arrow
|
|
└── ...
|
|
* Those datasets are optional for debugging the pipeline. ! NEED to add back when you are training the model.
|
|
```
|
|
|
|
***NOTE:***
|
|
|
|
<img src="https://user-images.githubusercontent.com/11957155/226159078-7f817452-76f8-44f4-af7a-9f13f3e02554.png" width="500">
|
|
There are overlap between COCO2017, COCO-Karpathy and REF-COCO dataset, and ref-coco is all overlapped with the COCO2017 training data, we have exclude the refcocog-umd validation, coco-karpathy test split during training.
|
|
|
|
## Evaluation Dataset
|
|
|
|
### RefCOCO (SEEM & X-Decoder)
|
|
Please refer to COCO Preparation on [line](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once/blob/v1.0/assets/readmes/DATASET.md#coco-seem--x-decoder).
|
|
|
|
### ADE20K, Cityscapes (X-Decoder)
|
|
Please Refer to [Mask2Former](https://github.com/facebookresearch/Mask2Former/tree/main/datasets).
|
|
|
|
### BDD100K (X-Decoder)
|
|
Please download the 10k split of BDD100k at https://doc.bdd100k.com/download.html#id1
|
|
|
|
### PascalVOC and all other interactive evaluation datasets (SEEM)
|
|
Please follow the instruction on [RITM](https://github.com/SamsungLabs/ritm_interactive_segmentation)
|
|
|
|
After dataset preparation, the dataset structure would be:
|
|
```
|
|
.xdecoder_data
|
|
└── PascalVOC/
|
|
├── Annotations/
|
|
├── ImageSets
|
|
├── JPEGImages/
|
|
├── SegmentationClass/
|
|
└── SegmentationObject/
|
|
```
|
|
|