mirrors
/
mmpretrain
mirror of https://github.com/open-mmlab/mmpretrain.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[Docs] update visualization doc. (#1160) 

* [Docs] update visualization doc

* update doc

* update folder

* update analysis

* Update print config tool

Co-authored-by: mzr1996 <mzr1996@163.com>
pull/1207/head
Songyang Zhang 2022-11-18 12:44:55 +08:00 committed by GitHub
parent e51ecfb129
commit f458bf5a64
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 470 additions and 448 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/en/index.rst
View File

@ -20,9 +20,18 @@ You can switch between Chinese and English documentation in the lower-left corne
user_guides/train_test.md
user_guides/config.md
user_guides/finetune.md
user_guides/analysis.md
user_guides/visualization.md
user_guides/useful_tools.md
.. toctree::
:maxdepth: 1
:caption: Useful Tools
useful_tools/dataset_visualization.md
useful_tools/scheduler_visualization.md
useful_tools/cam_visualization.md
useful_tools/print_config.md
useful_tools/verify_dataset.md
useful_tools/log_results_analysis.md
useful_tools/complexity_analysis.md
.. toctree::
:maxdepth: 1

143
docs/en/user_guides/visualization.md → docs/en/useful_tools/cam_visualization.md
View File

@ -1,140 +1,15 @@
# Visualization Tools
# Class Activation Map(CAM) Visualization
<!-- TOC -->
- [Browse Dataset](#browse-dataset)
- [Parameter Schedule Visualization](#parameter-schedule-visualization)
- [Class Activation Map Visualization](#class-activation-map-visualization)
- [FAQs](#faqs)
- [Introduction of the CAM visualization tool](#introduction-of-the-cam-visualization-tool)
- [How to visualize the CAM of CNN(ResNet-50)](#how-to-visualize-the-cam-of-cnnresnet-50)
- [How to visualize the CAM of vision transformer](#how-to-visualize-the-cam-of-vision-transformer)
<!-- TOC -->
## Browse Dataset
```bash
python tools/visualizations/browse_dataset.py \
${CONFIG_FILE} \
[-o, --output-dir ${OUTPUT_DIR}] \
[-p, --phase ${DATASET_PHASE}] \
[-n, --show-number ${NUMBER_IMAGES_DISPLAY}] \
[-i, --show-interval ${SHOW_INTERRVAL}] \
[-m, --mode ${DISPLAY_MODE}] \
[-r, --rescale-factor ${RESCALE_FACTOR}] \
[-c, --channel-order ${CHANNEL_ORDER}] \
[--cfg-options ${CFG_OPTIONS}]
```
**Description of all arguments**：
- `config` : The path of a model config file.
- `-o, --output-dir`: The output path for visualized images. If not specified, it will be set to `''`, which means not to save.
- **`-p, --phase`**: Phase of visualizing dataset，must be one of `['train', 'val', 'test']`. If not specified, it will be set to `'train'`.
- **`-n, --show-number`**: The number of samples to visualized. If not specified, display all images in the dataset.
- `--show-interval`: The interval of show (s).
- **`-m, --mode`**: The display mode, can be one of `['original', 'transformed', 'concat', 'pipeline']`. If not specified, it will be set to `'transformed'`.
- **`-r, --rescale-factor`**: The image rescale factor, which is useful if the output is too large or too small.
- `-c, --channel-order`: The channel of the showing images, could be "BGR" or "RGB", If not specified, it will be set to 'BGR'.
- `--cfg-options` : Modifications to the configuration file, refer to [Learn about Configs](./config.md).
```{note}
1. The `-m, --mode` is about display mode, display original pictures or transformed pictures or comparison pictures:
- "original" means show images load from disk;
- "transformed" means to show images after transformed;
- "concat" means show images stitched by "original" and "transformed" images;
- "pipeline" means show all the intermediate images throghout the pipeline.
2. The `-r, --rescale-factor` option is set when the label information is too large or too small relative to the picture. For example, when visualizing the CIFAR dataset, since the resolution of the image is very small, `--rescale-factor` can be set to 10.
```
**Examples**：
1. In **'original'** mode:
```shell
python ./tools/visualizations/browse_dataset.py ./configs/resnet/resnet101_8xb16_cifar10.py --phase val --output-dir tmp --mode original --show-number 100 --rescale-factor 10 --channel-order RGB
```
- `--phase val`: Visual validation set, can be simplified to `-p val`;
- `--output-dir tmp`: The visualization results are saved in the "tmp" folder, can be simplified to `-o tmp`;
- `--mode original`: Visualize the original image, can be simplified to `-m original`;
- `--show-number 100`: visualize 100 images, can be simplified to `-n 100`;
- `--rescale-factor`: the image is enlarged by 10 times, can be simplified to `-r 10`;
- `--channel-order RGB`: The channel order of the visualized image is "RGB", can be simplified to `-c RGB`.
<div align=center><img src="https://user-images.githubusercontent.com/18586273/190993839-216a7a1e-590e-47b9-92ae-08f87a7d58df.jpg" style=" width: auto; height: 40%; "></div>
2. In **'transformed'** mode:
```shell
python ./tools/visualizations/browse_dataset.py ./configs/resnet/resnet50_8xb32_in1k.py -n 100 -r 2
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/190994696-737b09d9-d0fb-4593-94a2-4487121e0286.JPEG" style=" width: auto; height: 40%; "></div>
3. In **'concat'** mode:
```shell
python ./tools/visualizations/browse_dataset.py configs/swin_transformer/swin-small_16xb64_in1k.py -n 10 -m concat
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/190995078-3872feb2-d4e2-4727-a21b-7062d52f7d3e.JPEG" style=" width: auto; height: 40%; "></div>
4. In **'pipeline'** mode：
```shell
python ./tools/visualizations/browse_dataset.py configs/swin_transformer/swin-small_16xb64_in1k.py -m pipeline
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/190995525-fac0220f-6630-4013-b94a-bc6de4fdff7a.JPEG" style=" width: auto; height: 40%; "></div>
## Parameter Schedule Visualization
```bash
python tools/visualizations/vis_scheduler.py \
${CONFIG_FILE} \
[-p, --parameter ${PARAMETER_NAME}] \
[-d, --dataset-size ${DATASET_SIZE}] \
[-n, --ngpus ${NUM_GPUs}] \
[-s, --save-path ${SAVE_PATH}] \
[--title ${TITLE}] \
[--style ${STYLE}] \
[--window-size ${WINDOW_SIZE}] \
[--cfg-options]
```
**Description of all arguments**：
- `config`: The path of a model config file.
- **`-p, --parameter`**: The param to visualize its change curve, choose from "lr" and "momentum". Default to use "lr".
- **`-d, --dataset-size`**: The size of the datasets. If set，`build_dataset` will be skipped and `${DATASET_SIZE}` will be used as the size. Default to use the function `build_dataset`.
- **`-n, --ngpus`**: The number of GPUs used in training, default to be 1.
- **`-s, --save-path`**: The learning rate curve plot save path, default not to save.
- `--title`: Title of figure. If not set, default to be config file name.
- `--style`: Style of plt. If not set, default to be `whitegrid`.
- `--window-size`: The shape of the display window. If not specified, it will be set to `12*7`. If used, it must be in the format `'W*H'`.
- `--cfg-options`: Modifications to the configuration file, refer to [Learn about Configs](./config.md).
```{note}
Loading annotations maybe consume much time, you can directly specify the size of the dataset with `-d, dataset-size` to save time.
```
**Examples**：
```bash
python tools/visualizations/vis_scheduler.py configs/resnet/resnet50_b16x8_cifar100.py
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/191006713-023f065d-d366-4165-a52e-36176367506e.png" style=" width: auto; height: 40%; "></div>
When using ImageNet, directly specify the size of ImageNet, as below:
```bash
python tools/visualizations/vis_scheduler.py configs/repvgg/repvgg-B3g4_4xb64-autoaug-lbs-mixup-coslr-200e_in1k.py --dataset-size 1281167 --ngpus 4 --save-path ./repvgg-B3g4_4xb64-lr.jpg
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/191006721-0f680e07-355e-4cd6-889c-86c0cad9acb7.png" style=" width: auto; height: 40%; "></div>
## Class Activation Map Visualization
## Introduction of the CAM visualization tool
MMClassification provides `tools\visualizations\vis_cam.py` tool to visualize class activation map. Please use `pip install "grad-cam>=1.3.6"` command to install [pytorch-grad-cam](https://github.com/jacobgil/pytorch-grad-cam).
@ -190,7 +65,7 @@ python tools/visualizations/vis_cam.py \
The argument `--preview-model` can view all network layers names in the given model. It will be helpful if you know nothing about the model layers when setting `--target-layers`.
```
**Examples(CNN)**：
## How to visualize the CAM of CNN(ResNet-50)
Here are some examples of `target-layers` in ResNet-50, which can be any module or layer:
@ -252,7 +127,7 @@ For example, the `backbone.layer4[-1]` is the same as `backbone.layer4.2` since
| ------------------------------------ | --------------------------------------- | ------------------------------------------- | ----------------------------------------- | ----------------------------------------- |
| <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557492-98ac5ce0-61f9-4da9-8ea7-396d0b6a20fa.jpg' height="auto" width="160"></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557541-a4cf7d86-7267-46f9-937c-6f657ea661b4.jpg' height="auto" width="145" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557547-2731b53e-e997-4dd2-a092-64739cc91959.jpg' height="auto" width="145" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557545-8189524a-eb92-4cce-bf6a-760cab4a8065.jpg' height="auto" width="145" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144557548-c1e3f3ec-3c96-43d4-874a-3b33cd3351c5.jpg' height="auto" width="145" ></div> |
**Examples(Transformer)**：
## How to visualize the CAM of vision transformer
Here are some examples:
@ -300,7 +175,3 @@ To exclude these extra tokens, we need know the number of extra tokens. Almost a
| Image | ResNet50 | ViT | Swin | T2T-ViT |
| --------------------------------------- | ------------------------------------------ | -------------------------------------- | --------------------------------------- | ------------------------------------------ |
| <div align=center><img src='https://user-images.githubusercontent.com/18586273/144429496-628d3fb3-1f6e-41ff-aa5c-1b08c60c32a9.JPEG' height="auto" width="165" ></div> | <div align=center><img src=https://user-images.githubusercontent.com/18586273/144431491-a2e19fe3-5c12-4404-b2af-a9552f5a95d9.jpg height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144436218-245a11de-6234-4852-9c08-ff5069f6a739.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144436168-01b0e565-442c-4e1e-910c-17c62cff7cd3.jpg' height="auto" width="150" ></div> | <div align=center><img src='https://user-images.githubusercontent.com/18586273/144436198-51dbfbda-c48d-48cc-ae06-1a923d19b6f6.jpg' height="auto" width="150" ></div> |
## FAQs
- None

68
docs/en/useful_tools/complexity_analysis.md 100644
View File

@ -0,0 +1,68 @@
# Model Complexity Analysis
## Get the FLOPs and params (experimental)
We provide a script adapted from [fvcore](https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/flop_count.py) to compute the FLOPs and params of a given model.
```shell
python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}]
```
Description of all arguments:
- `config` : The path of the model config file.
- `--shape`: Input size, support single value or double value parameter, such as `--shape 256` or `--shape 224 256`. If not set, default to be `224 224`.
You will get the final result like this.
```text
==============================
Input shape: (3, 224, 224)
Flops: 17.582G
Params: 91.234M
Activation: 23.895M
==============================
```
Also, you will get the detailed complexity information of each layer like this:
```text
| module | #parameters or shape | #flops | #activations |
|:------------------------------------------|:-----------------------|:----------|:---------------|
| model | 91.234M | 17.582G | 23.895M |
| backbone | 85.799M | 17.582G | 23.895M |
| backbone.cls_token | (1, 1, 768) | | |
| backbone.pos_embed | (1, 197, 768) | | |
| backbone.patch_embed.projection | 0.591M | 0.116G | 0.151M |
| backbone.patch_embed.projection.weight | (768, 3, 16, 16) | | |
| backbone.patch_embed.projection.bias | (768,) | | |
| backbone.layers | 85.054M | 17.466G | 23.744M |
| backbone.layers.0 | 7.088M | 1.455G | 1.979M |
| backbone.layers.1 | 7.088M | 1.455G | 1.979M |
| backbone.layers.2 | 7.088M | 1.455G | 1.979M |
| backbone.layers.3 | 7.088M | 1.455G | 1.979M |
| backbone.layers.4 | 7.088M | 1.455G | 1.979M |
| backbone.layers.5 | 7.088M | 1.455G | 1.979M |
| backbone.layers.6 | 7.088M | 1.455G | 1.979M |
| backbone.layers.7 | 7.088M | 1.455G | 1.979M |
| backbone.layers.8 | 7.088M | 1.455G | 1.979M |
| backbone.layers.9 | 7.088M | 1.455G | 1.979M |
| backbone.layers.10 | 7.088M | 1.455G | 1.979M |
| backbone.layers.11 | 7.088M | 1.455G | 1.979M |
| backbone.ln1 | 1.536K | 0.756M | 0 |
| backbone.ln1.weight | (768,) | | |
| backbone.ln1.bias | (768,) | | |
| head.layers | 5.435M | | |
| head.layers.pre_logits | 2.362M | | |
| head.layers.pre_logits.weight | (3072, 768) | | |
| head.layers.pre_logits.bias | (3072,) | | |
| head.layers.head | 3.073M | | |
| head.layers.head.weight | (1000, 3072) | | |
| head.layers.head.bias | (1000,) | | |
```
```{warning}
This tool is still experimental and we do not guarantee that the number is correct. You may well use the result for simple comparisons, but double-check it before you adopt it in technical reports or papers.
- FLOPs are related to the input shape while parameters are not. The default input shape is (1, 3, 224, 224).
- Some operators are not counted into FLOPs like custom operators. Refer to [`fvcore.nn.flop_count._DEFAULT_SUPPORTED_OPS`](https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/flop_count.py) for details.
```

92
docs/en/useful_tools/dataset_visualization.md 100644
View File

@ -0,0 +1,92 @@
# Dataset Visualization
<!-- TOC -->
- [Introduce the dataset visualization tool](#introduce-the-dataset-visualization-tool)
- [How to visualize the original image](#how-to-visualize-the-original-image)
- [How to visualize the transformed images](#how-to-visualize-the-transformed-images)
- [How to visualize the transformed images and original images together](#how-to-visualize-the-transformed-images-and-original-images-together)
<!-- TOC -->
## Introduce the dataset visualization tool
```bash
python tools/visualizations/browse_dataset.py \
${CONFIG_FILE} \
[-o, --output-dir ${OUTPUT_DIR}] \
[-p, --phase ${DATASET_PHASE}] \
[-n, --show-number ${NUMBER_IMAGES_DISPLAY}] \
[-i, --show-interval ${SHOW_INTERRVAL}] \
[-m, --mode ${DISPLAY_MODE}] \
[-r, --rescale-factor ${RESCALE_FACTOR}] \
[-c, --channel-order ${CHANNEL_ORDER}] \
[--cfg-options ${CFG_OPTIONS}]
```
**Description of all arguments**：
- `config` : The path of a model config file.
- `-o, --output-dir`: The output path for visualized images. If not specified, it will be set to `''`, which means not to save.
- **`-p, --phase`**: Phase of visualizing dataset，must be one of `['train', 'val', 'test']`. If not specified, it will be set to `'train'`.
- **`-n, --show-number`**: The number of samples to visualized. If not specified, display all images in the dataset.
- `--show-interval`: The interval of show (s).
- **`-m, --mode`**: The display mode, can be one of `['original', 'transformed', 'concat', 'pipeline']`. If not specified, it will be set to `'transformed'`.
- **`-r, --rescale-factor`**: The image rescale factor, which is useful if the output is too large or too small.
- `-c, --channel-order`: The channel of the showing images, could be "BGR" or "RGB", If not specified, it will be set to 'BGR'.
- `--cfg-options` : Modifications to the configuration file, refer to [Learn about Configs](./config.md).
```{note}
1. The `-m, --mode` is about display mode, display original pictures or transformed pictures or comparison pictures:
- "original" means show images load from disk;
- "transformed" means to show images after transformed;
- "concat" means show images stitched by "original" and "transformed" images;
- "pipeline" means show all the intermediate images throghout the pipeline.
2. The `-r, --rescale-factor` option is set when the label information is too large or too small relative to the picture. For example, when visualizing the CIFAR dataset, since the resolution of the image is very small, `--rescale-factor` can be set to 10.
```
## How to visualize the original image
In **'original'** mode:
```shell
python ./tools/visualizations/browse_dataset.py ./configs/resnet/resnet101_8xb16_cifar10.py --phase val --output-dir tmp --mode original --show-number 100 --rescale-factor 10 --channel-order RGB
```
- `--phase val`: Visual validation set, can be simplified to `-p val`;
- `--output-dir tmp`: The visualization results are saved in the "tmp" folder, can be simplified to `-o tmp`;
- `--mode original`: Visualize the original image, can be simplified to `-m original`;
- `--show-number 100`: visualize 100 images, can be simplified to `-n 100`;
- `--rescale-factor`: the image is enlarged by 10 times, can be simplified to `-r 10`;
- `--channel-order RGB`: The channel order of the visualized image is "RGB", can be simplified to `-c RGB`.
<div align=center><img src="https://user-images.githubusercontent.com/18586273/190993839-216a7a1e-590e-47b9-92ae-08f87a7d58df.jpg" style=" width: auto; height: 40%; "></div>
## How to visualize the transformed images
In **'transformed'** mode:
```shell
python ./tools/visualizations/browse_dataset.py ./configs/resnet/resnet50_8xb32_in1k.py -n 100 -r 2
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/190994696-737b09d9-d0fb-4593-94a2-4487121e0286.JPEG" style=" width: auto; height: 40%; "></div>
## How to visualize the transformed images and original images together
In **'concat'** mode:
```shell
python ./tools/visualizations/browse_dataset.py configs/swin_transformer/swin-small_16xb64_in1k.py -n 10 -m concat
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/190995078-3872feb2-d4e2-4727-a21b-7062d52f7d3e.JPEG" style=" width: auto; height: 40%; "></div>
4. In **'pipeline'** mode：
```shell
python ./tools/visualizations/browse_dataset.py configs/swin_transformer/swin-small_16xb64_in1k.py -m pipeline
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/190995525-fac0220f-6630-4013-b94a-bc6de4fdff7a.JPEG" style=" width: auto; height: 40%; "></div>

177
docs/en/useful_tools/log_result_analysis.md 100644
View File

@ -0,0 +1,177 @@
# Log and Results Analysis
<!-- TOC -->
- [Log Analysis](#log-analysis)
- [Introduction of log analysis tool](#introduction-of-log-analysis-tool)
- [How to plot the loss/accuracy curve](#how-to-plot-the-lossaccuracy-curve)
- [How to calculate training time](#how-to-calculate-training-time)
- [Result Analysis](#result-analysis)
- [Evaluate Results](#evaluate-results)
- [View Typical Results](#view-typical-results)
<!-- TOC -->
## Log Analysis
### Introduction of log analysis tool
`tools/analysis_tools/analyze_logs.py` plots curves of given keys according to the log files.
<div align=center><img src="../_static/image/tools/analysis/analyze_log.jpg" style=" width: 75%; height: 30%; "></div>
```shell
python tools/analysis_tools/analyze_logs.py plot_curve \
${JSON_LOGS} \
[--keys ${KEYS}] \
[--title ${TITLE}] \
[--legend ${LEGEND}] \
[--backend ${BACKEND}] \
[--style ${STYLE}] \
[--out ${OUT_FILE}] \
[--window-size ${WINDOW_SIZE}]
```
**Description of all arguments**：
- `json_logs` : The paths of the log files, separate multiple files by spaces.
- `--keys` : The fields of the logs to analyze, separate multiple keys by spaces. Defaults to 'loss'.
- `--title` : The title of the figure. Defaults to use the filename.
- `--legend` : The names of legend, the number of which must be equal to `len(${JSON_LOGS}) * len(${KEYS})`. Defaults to use `"${JSON_LOG}-${KEYS}"`.
- `--backend` : The backend of matplotlib. Defaults to auto selected by matplotlib.
- `--style` : The style of the figure. Default to `whitegrid`.
- `--out` : The path of the output picture. If not set, the figure won't be saved.
- `--window-size`: The shape of the display window. The format should be `'W*H'`. Defaults to `'12*7'`.
```{note}
The `--style` option depends on `seaborn` package, please install it before setting it.
```
### How to plot the loss/accuracy curve
We present some examples here to show how to plot the loss curve of accuracy curve by using the `tools/analysis_tools/analyze_logs.py`
#### Plot the loss curve in training.
```shell
python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys loss --legend loss
```
#### Plot the top-1 accuracy and top-5 accuracy curves, and save the figure to results.jpg.
```shell
python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys accuracy_top-1 accuracy_top-5 --legend top1 top5 --out results.jpg
```
#### Compare the top-1 accuracy of two log files in the same figure.
```shell
python tools/analysis_tools/analyze_logs.py plot_curve log1.json log2.json --keys accuracy_top-1 --legend exp1 exp2
```
```{note}
The tool will automatically select to find keys in training logs or validation logs according to the keys.
Therefore, if you add a custom evaluation metric, please also add the key to `TEST_METRICS` in this tool.
```
### How to calculate training time
`tools/analysis_tools/analyze_logs.py` can also calculate the training time according to the log files.
```shell
python tools/analysis_tools/analyze_logs.py cal_train_time \
${JSON_LOGS}
[--include-outliers]
```
**Description of all arguments**:
- `json_logs` : The paths of the log files, separate multiple files by spaces.
- `--include-outliers` : If set, include the first iteration in each epoch (Sometimes the time of first iterations is longer).
Example:
```shell
python tools/analysis_tools/analyze_logs.py cal_train_time work_dirs/some_exp/20200422_153324.log.json
```
The output is expected to be like the below.
```text
-----Analyze train time of work_dirs/some_exp/20200422_153324.log.json-----
slowest epoch 68, average time is 0.3818
fastest epoch 1, average time is 0.3694
time std over epochs is 0.0020
average iter time: 0.3777 s/iter
```
## Result Analysis
With the `--out` argument in `tools/test.py`, we can save the inference results of all samples as a file.
And with this result file, we can do further analysis.
### How to conduct offline metric evaluation
We provide `tools/analysis_tools/eval_metric.py` to enable the user evaluate the model from the prediction files.
```shell
python tools/analysis_tools/eval_metric.py \
${CONFIG} \
${RESULT} \
[--metrics ${METRICS}] \
[--cfg-options ${CFG_OPTIONS}] \
[--metric-options ${METRIC_OPTIONS}]
```
Description of all arguments:
- `config` : The path of the model config file.
- `result`: The Output result file in json/pickle format from `tools/test.py`.
- `--metrics` : Evaluation metrics, the acceptable values depend on the dataset.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](./config.md)
- `--metric-options`: If specified, the key-value pair arguments will be passed to the `metric_options` argument of dataset's `evaluate` function.
```{note}
In `tools/test.py`, we support using `--out-items` option to select which kind of results will be saved. Please ensure the result file includes "class_scores" to use this tool.
```
**Examples**:
```shell
python tools/analysis_tools/eval_metric.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py your_result.pkl --metrics accuracy --metric-options "topk=(1,5)"
```
### How to visualize the prediction results
We can also use this tool `tools/analysis_tools/analyze_results.py` to save the images with the highest scores in successful or failed prediction.
```shell
python tools/analysis_tools/analyze_results.py \
${CONFIG} \
${RESULT} \
[--out-dir ${OUT_DIR}] \
[--topk ${TOPK}] \
[--cfg-options ${CFG_OPTIONS}]
```
**Description of all arguments**:
- `config` : The path of the model config file.
- `result`: Output result file in json/pickle format from `tools/test.py`.
- `--out_dir`: Directory to store output files.
- `--topk`: The number of images in successful or failed prediction with the highest `topk` scores to save. If not specified, it will be set to 20.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](./config.md)
```{note}
In `tools/test.py`, we support using `--out-items` option to select which kind of results will be saved. Please ensure the result file includes "pred_score", "pred_label" and "pred_class" to use this tool.
```
**Examples**:
```shell
python tools/analysis_tools/analyze_results.py \
configs/resnet/resnet50_b32x8_imagenet.py \
result.pkl \
--out_dir results \
--topk 50
```

25
docs/en/useful_tools/print_config.md 100644
View File

@ -0,0 +1,25 @@
# How to Get the Complete Config
We also provide the `print_config.py` tools to print the complete configuration of the given experiment.
You can check each item of the config before the training by using the following command.
## Description
`tools/misc/print_config.py` prints the whole config verbatim, expanding all its imports.
```shell
python tools/misc/print_config.py ${CONFIG} [--cfg-options ${CFG_OPTIONS}]
```
Description of all arguments:
- `config` : The path of the model config file.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](./config.md)
## Examples
Print the complete config of `configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py`
```shell
python tools/misc/print_config.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py
```

59
docs/en/useful_tools/scheduler_visualization.md 100644
View File

@ -0,0 +1,59 @@
# Hyper-parameter Scheduler Visualization
<!-- TOC -->
- [Parameter Schedule Visualization](#parameter-schedule-visualization)
- [How to plot the learning rate curve without training](#how-to-plot-the-learning-rate-curve-without-training)
<!-- TOC -->
This tool aims to help the user to check the hyper-parameter scheduler of the optimizer(without training), which support the "learning rate" or "momentum"
## Introduce the scheduler visualization tool
```bash
python tools/visualizations/vis_scheduler.py \
${CONFIG_FILE} \
[-p, --parameter ${PARAMETER_NAME}] \
[-d, --dataset-size ${DATASET_SIZE}] \
[-n, --ngpus ${NUM_GPUs}] \
[-s, --save-path ${SAVE_PATH}] \
[--title ${TITLE}] \
[--style ${STYLE}] \
[--window-size ${WINDOW_SIZE}] \
[--cfg-options]
```
**Description of all arguments**：
- `config`: The path of a model config file.
- **`-p, --parameter`**: The param to visualize its change curve, choose from "lr" and "momentum". Default to use "lr".
- **`-d, --dataset-size`**: The size of the datasets. If set，`build_dataset` will be skipped and `${DATASET_SIZE}` will be used as the size. Default to use the function `build_dataset`.
- **`-n, --ngpus`**: The number of GPUs used in training, default to be 1.
- **`-s, --save-path`**: The learning rate curve plot save path, default not to save.
- `--title`: Title of figure. If not set, default to be config file name.
- `--style`: Style of plt. If not set, default to be `whitegrid`.
- `--window-size`: The shape of the display window. If not specified, it will be set to `12*7`. If used, it must be in the format `'W*H'`.
- `--cfg-options`: Modifications to the configuration file, refer to [Learn about Configs](./config.md).
```{note}
Loading annotations maybe consume much time, you can directly specify the size of the dataset with `-d, dataset-size` to save time.
```
## How to plot the learning rate curve without training
You can use the following command to plot the step learning rate schedule used in the config `configs/resnet/resnet50_b16x8_cifar100.py`:
```bash
python tools/visualizations/vis_scheduler.py configs/resnet/resnet50_b16x8_cifar100.py
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/191006713-023f065d-d366-4165-a52e-36176367506e.png" style=" width: auto; height: 40%; "></div>
When using ImageNet, directly specify the size of ImageNet, as below:
```bash
python tools/visualizations/vis_scheduler.py configs/repvgg/repvgg-B3g4_4xb64-autoaug-lbs-mixup-coslr-200e_in1k.py --dataset-size 1281167 --ngpus 4 --save-path ./repvgg-B3g4_4xb64-lr.jpg
```
<div align=center><img src="https://user-images.githubusercontent.com/18586273/191006721-0f680e07-355e-4cd6-889c-86c0cad9acb7.png" style=" width: auto; height: 40%; "></div>

28
docs/en/useful_tools/verify_dataset.md 100644
View File

@ -0,0 +1,28 @@
# Verify Dataset
In MMClassification, we also provide a tool `tools/misc/verify_dataset.py` to check whether there exists **broken pictures** in the given dataset.
## Introduce the tool
```shell
python tools/print_config.py \
${CONFIG} \
[--out-path ${OUT-PATH}] \
[--phase ${PHASE}] \
[--num-process ${NUM-PROCESS}]
[--cfg-options ${CFG_OPTIONS}]
```
**Description of all arguments**:
- `config` : The path of the model config file.
- `--out-path` : The path to save the verification result, if not set, defaults to 'brokenfiles.log'.
- `--phase` : Phase of dataset to verify, accept "train" "test" and "val", if not set, defaults to "train".
- `--num-process` : number of process to use, if not set, defaults to 1.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](./config.md)
## Example
```shell
python tools/misc/verify_dataset.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py --out-path broken_imgs.log --phase val --num-process 8
```

249
docs/en/user_guides/analysis.md
View File

@ -1,249 +0,0 @@
# Analysis Tools (TODO)
<!-- TOC -->
- [Log Analysis](#log-analysis)
- [Plot Curves](#plot-curves)
- [Calculate Training Time](#calculate-training-time)
- [Result Analysis](#result-analysis)
- [Evaluate Results](#evaluate-results)
- [View Typical Results](#view-typical-results)
- [Model Complexity](#model-complexity)
- [FAQs](#faqs)
<!-- TOC -->
## Log Analysis
### Plot Curves
`tools/analysis_tools/analyze_logs.py` plots curves of given keys according to the log files.
<div align=center><img src="../_static/image/tools/analysis/analyze_log.jpg" style=" width: 75%; height: 30%; "></div>
```shell
python tools/analysis_tools/analyze_logs.py plot_curve \
${JSON_LOGS} \
[--keys ${KEYS}] \
[--title ${TITLE}] \
[--legend ${LEGEND}] \
[--backend ${BACKEND}] \
[--style ${STYLE}] \
[--out ${OUT_FILE}] \
[--window-size ${WINDOW_SIZE}]
```
**Description of all arguments**：
- `json_logs` : The paths of the log files, separate multiple files by spaces.
- `--keys` : The fields of the logs to analyze, separate multiple keys by spaces. Defaults to 'loss'.
- `--title` : The title of the figure. Defaults to use the filename.
- `--legend` : The names of legend, the number of which must be equal to `len(${JSON_LOGS}) * len(${KEYS})`. Defaults to use `"${JSON_LOG}-${KEYS}"`.
- `--backend` : The backend of matplotlib. Defaults to auto selected by matplotlib.
- `--style` : The style of the figure. Default to `whitegrid`.
- `--out` : The path of the output picture. If not set, the figure won't be saved.
- `--window-size`: The shape of the display window. The format should be `'W*H'`. Defaults to `'12*7'`.
```{note}
The `--style` option depends on `seaborn` package, please install it before setting it.
```
Examples:
- Plot the loss curve in training.
```shell
python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys loss --legend loss
```
- Plot the top-1 accuracy and top-5 accuracy curves, and save the figure to results.jpg.
```shell
python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys accuracy_top-1 accuracy_top-5 --legend top1 top5 --out results.jpg
```
- Compare the top-1 accuracy of two log files in the same figure.
```shell
python tools/analysis_tools/analyze_logs.py plot_curve log1.json log2.json --keys accuracy_top-1 --legend exp1 exp2
```
```{note}
The tool will automatically select to find keys in training logs or validation logs according to the keys.
Therefore, if you add a custom evaluation metric, please also add the key to `TEST_METRICS` in this tool.
```
### Calculate Training Time
`tools/analysis_tools/analyze_logs.py` can also calculate the training time according to the log files.
```shell
python tools/analysis_tools/analyze_logs.py cal_train_time \
${JSON_LOGS}
[--include-outliers]
```
**Description of all arguments**:
- `json_logs` : The paths of the log files, separate multiple files by spaces.
- `--include-outliers` : If set, include the first iteration in each epoch (Sometimes the time of first iterations is longer).
Example:
```shell
python tools/analysis_tools/analyze_logs.py cal_train_time work_dirs/some_exp/20200422_153324.log.json
```
The output is expected to be like the below.
```text
-----Analyze train time of work_dirs/some_exp/20200422_153324.log.json-----
slowest epoch 68, average time is 0.3818
fastest epoch 1, average time is 0.3694
time std over epochs is 0.0020
average iter time: 0.3777 s/iter
```
## Result Analysis
With the `--out` argument in `tools/test.py`, we can save the inference results of all samples as a file.
And with this result file, we can do further analysis.
### Evaluate Results
`tools/analysis_tools/eval_metric.py` can evaluate metrics again.
```shell
python tools/analysis_tools/eval_metric.py \
${CONFIG} \
${RESULT} \
[--metrics ${METRICS}] \
[--cfg-options ${CFG_OPTIONS}] \
[--metric-options ${METRIC_OPTIONS}]
```
Description of all arguments:
- `config` : The path of the model config file.
- `result`: The Output result file in json/pickle format from `tools/test.py`.
- `--metrics` : Evaluation metrics, the acceptable values depend on the dataset.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](./config.md)
- `--metric-options`: If specified, the key-value pair arguments will be passed to the `metric_options` argument of dataset's `evaluate` function.
```{note}
In `tools/test.py`, we support using `--out-items` option to select which kind of results will be saved. Please ensure the result file includes "class_scores" to use this tool.
```
**Examples**:
```shell
python tools/analysis_tools/eval_metric.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py your_result.pkl --metrics accuracy --metric-options "topk=(1,5)"
```
### View Typical Results
`tools/analysis_tools/analyze_results.py` can save the images with the highest scores in successful or failed prediction.
```shell
python tools/analysis_tools/analyze_results.py \
${CONFIG} \
${RESULT} \
[--out-dir ${OUT_DIR}] \
[--topk ${TOPK}] \
[--cfg-options ${CFG_OPTIONS}]
```
**Description of all arguments**:
- `config` : The path of the model config file.
- `result`: Output result file in json/pickle format from `tools/test.py`.
- `--out_dir`: Directory to store output files.
- `--topk`: The number of images in successful or failed prediction with the highest `topk` scores to save. If not specified, it will be set to 20.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](./config.md)
```{note}
In `tools/test.py`, we support using `--out-items` option to select which kind of results will be saved. Please ensure the result file includes "pred_score", "pred_label" and "pred_class" to use this tool.
```
**Examples**:
```shell
python tools/analysis_tools/analyze_results.py \
configs/resnet/resnet50_b32x8_imagenet.py \
result.pkl \
--out_dir results \
--topk 50
```
## Model Complexity
### Get the FLOPs and params (experimental)
We provide a script adapted from [fvcore](https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/flop_count.py) to compute the FLOPs and params of a given model.
```shell
python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}]
```
Description of all arguments:
- `config` : The path of the model config file.
- `--shape`: Input size, support single value or double value parameter, such as `--shape 256` or `--shape 224 256`. If not set, default to be `224 224`.
You will get the final result like this.
```text
==============================
Input shape: (3, 224, 224)
Flops: 17.582G
Params: 91.234M
Activation: 23.895M
==============================
```
Also, you will get the detailed complexity information of each layer like this:
```text
| module | #parameters or shape | #flops | #activations |
|:------------------------------------------|:-----------------------|:----------|:---------------|
| model | 91.234M | 17.582G | 23.895M |
| backbone | 85.799M | 17.582G | 23.895M |
| backbone.cls_token | (1, 1, 768) | | |
| backbone.pos_embed | (1, 197, 768) | | |
| backbone.patch_embed.projection | 0.591M | 0.116G | 0.151M |
| backbone.patch_embed.projection.weight | (768, 3, 16, 16) | | |
| backbone.patch_embed.projection.bias | (768,) | | |
| backbone.layers | 85.054M | 17.466G | 23.744M |
| backbone.layers.0 | 7.088M | 1.455G | 1.979M |
| backbone.layers.1 | 7.088M | 1.455G | 1.979M |
| backbone.layers.2 | 7.088M | 1.455G | 1.979M |
| backbone.layers.3 | 7.088M | 1.455G | 1.979M |
| backbone.layers.4 | 7.088M | 1.455G | 1.979M |
| backbone.layers.5 | 7.088M | 1.455G | 1.979M |
| backbone.layers.6 | 7.088M | 1.455G | 1.979M |
| backbone.layers.7 | 7.088M | 1.455G | 1.979M |
| backbone.layers.8 | 7.088M | 1.455G | 1.979M |
| backbone.layers.9 | 7.088M | 1.455G | 1.979M |
| backbone.layers.10 | 7.088M | 1.455G | 1.979M |
| backbone.layers.11 | 7.088M | 1.455G | 1.979M |
| backbone.ln1 | 1.536K | 0.756M | 0 |
| backbone.ln1.weight | (768,) | | |
| backbone.ln1.bias | (768,) | | |
| head.layers | 5.435M | | |
| head.layers.pre_logits | 2.362M | | |
| head.layers.pre_logits.weight | (3072, 768) | | |
| head.layers.pre_logits.bias | (3072,) | | |
| head.layers.head | 3.073M | | |
| head.layers.head.weight | (1000, 3072) | | |
| head.layers.head.bias | (1000,) | | |
```
```{warning}
This tool is still experimental and we do not guarantee that the number is correct. You may well use the result for simple comparisons, but double-check it before you adopt it in technical reports or papers.
- FLOPs are related to the input shape while parameters are not. The default input shape is (1, 3, 224, 224).
- Some operators are not counted into FLOPs like custom operators. Refer to [`fvcore.nn.flop_count._DEFAULT_SUPPORTED_OPS`](https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/flop_count.py) for details.
```
## FAQs
- None

59
docs/en/user_guides/useful_tools.md
View File

@ -1,59 +0,0 @@
# Useful Tools (TODO)
<!-- TOC -->
- [Print the entire config](#print-the-entire-config)
- [Verify Dataset](#verify-dataset)
- [FAQs](#faqs)
<!-- TOC -->
## Print the entire config
`tools/misc/print_config.py` prints the whole config verbatim, expanding all its imports.
```shell
python tools/misc/print_config.py ${CONFIG} [--cfg-options ${CFG_OPTIONS}]
```
Description of all arguments:
- `config` : The path of the model config file.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](./config.md)
**Examples**:
```shell
python tools/misc/print_config.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py
```
## Verify Dataset
`tools/misc/verify_dataset.py` can verify dataset, check whether there are broken pictures in the given dataset.
```shell
python tools/print_config.py \
${CONFIG} \
[--out-path ${OUT-PATH}] \
[--phase ${PHASE}] \
[--num-process ${NUM-PROCESS}]
[--cfg-options ${CFG_OPTIONS}]
```
**Description of all arguments**:
- `config` : The path of the model config file.
- `--out-path` : The path to save the verification result, if not set, defaults to 'brokenfiles.log'.
- `--phase` : Phase of dataset to verify, accept "train" "test" and "val", if not set, defaults to "train".
- `--num-process` : number of process to use, if not set, defaults to 1.
- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Learn about Configs](./config.md)
**Examples**:
```shell
python tools/misc/verify_dataset.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py --out-path broken_imgs.log --phase val --num-process 8
```
## FAQs
- None

3
tools/misc/print_config.py
View File

@ -1,6 +1,7 @@
# Copyright (c) OpenMMLab. All rights reserved.
import argparse
import rich
from mmengine import Config, DictAction
@ -28,7 +29,7 @@ def main():
cfg = Config.fromfile(args.config)
if args.cfg_options is not None:
cfg.merge_from_dict(args.cfg_options)
print(f'Config:\n{cfg.pretty_text}')
rich.print(cfg.pretty_text)
if __name__ == '__main__':