mirror of https://github.com/alibaba/EasyCV.git
72 lines
2.3 KiB
Markdown
72 lines
2.3 KiB
Markdown
# Prepare Datasets
|
||
|
||
- [Prepare Cifar](#Prepare Cifar)
|
||
- [Prepare Imagenet](#Prepare Imagenet)
|
||
- [Prepare Imagenet-TFrecords](#Prepare Imagenet-TFrecords)
|
||
- [Prepare COCO](#Prepare COCO)
|
||
- [Prepare PAI-Itag detection](#Prepare PAI-Itag detection)
|
||
|
||
## Prepare Cifar
|
||
|
||
Download dataset [cifar10](http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/data/cifar10/cifar-10-python.tar.gz) and uncompress files to `data/cifar`, directory structure is as follows:
|
||
|
||
```text
|
||
data/cifar
|
||
└── cifar-10-batches-py
|
||
├── batches.meta
|
||
├── data_batch_1
|
||
├── data_batch_2
|
||
├── data_batch_3
|
||
├── data_batch_4
|
||
├── data_batch_5
|
||
├── readme.html
|
||
├── read.py
|
||
└── test_batch
|
||
```
|
||
|
||
## Prepare Imagenet
|
||
|
||
1. Go to the [download-url](http://www.image-net.org/download-images), Register an account and log in .
|
||
2. Download the following files：
|
||
|
||
- Training images (Task 1 & 2). 138GB.
|
||
|
||
- Validation images (all tasks). 6.3GB.
|
||
3. Unzip the downloaded file.
|
||
4. Using this [scrip](https://github.com/BVLC/caffe/blob/master/data/ilsvrc12/get_ilsvrc_aux.sh) to get data meta.
|
||
|
||
## Prepare Imagenet-TFrecords
|
||
|
||
1. Go to the [download-url](https://www.kaggle.com/hmendonca/imagenet-1k-tfrecords-ilsvrc2012-part-0), Register an account and log in .
|
||
2. The dataset is divided into two parts, [part0](https://www.kaggle.com/hmendonca/imagenet-1k-tfrecords-ilsvrc2012-part-0) (79GB) and [part1](https://www.kaggle.com/hmendonca/imagenet-1k-tfrecords-ilsvrc2012-part-1) (75GB), you need download all of them.
|
||
|
||
## Prepare COCO
|
||
|
||
Download [COCO2017](https://cocodataset.org/#download) dataset to `data/coco`, directory structure is as follows
|
||
|
||
```text
|
||
data/coco
|
||
├── annotations
|
||
├── train2017
|
||
└── val2017
|
||
```
|
||
|
||
## Prepare PAI-Itag detection
|
||
|
||
Download [SmallCOCO](http://pai-vision-data-hz.oss-cn-zhangjiakou.aliyuncs.com/unittest/data/detection/small_coco_itag/small_coco_itag.tar.gz) dataset to `data/coco`,
|
||
directory structure is as follows:
|
||
|
||
```text
|
||
data/coco/
|
||
├── train2017
|
||
├── train2017_20_local.manifest
|
||
├── val2017
|
||
└── val2017_20_local.manifest
|
||
```
|
||
|
||
replace train_data and val_data path in config file
|
||
```shell
|
||
sed -i 's#train2017.manifest#train2017_20_local.manifest#g' configs/detection/yolox_coco_pai.py
|
||
sed -i 's#val2017.manifest#val2017_20_local.manifest#g' configs/detection/yolox_coco_pai.py
|
||
```
|