mmsegmentation/.dev/log_collector

Log Collector

Function

Automatically collect logs and write the result in a json file or markdown file.

If there are several .log.json files in one folder, Log Collector assumes that the .log.json files other than the first one are resume from the preceding .log.json file. Log Collector returns the result considering all .log.json files.

Usage:

To use log collector, you need to write a config file to configure the log collector first.

For example:

example_config.py:

# The work directory that contains folders that contains .log.json files.
work_dir = '../../work_dirs'
# The metric used to find the best evaluation.
metric = 'mIoU'

# **Don't specify the log_items and ignore_keywords at the same time.**
# Specify the log files we would like to collect in `log_items`.
# The folders specified should be the subdirectories of `work_dir`.
log_items = [
    'segformer_mit-b5_512x512_160k_ade20k_cnn_lr_with_warmup',
    'segformer_mit-b5_512x512_160k_ade20k_cnn_no_warmup_lr',
    'segformer_mit-b5_512x512_160k_ade20k_mit_trans_lr',
    'segformer_mit-b5_512x512_160k_ade20k_swin_trans_lr'
]
# Or specify `ignore_keywords`. The folders whose name contain one
# of the keywords in the `ignore_keywords` list(e.g., `'segformer'`)
# won't be collected.
# ignore_keywords = ['segformer']

# Other log items in .log.json that you want to collect.
# should not include metric.
other_info_keys = ["mAcc"]
# The output markdown file's name.
markdown_file ='markdowns/lr_in_trans.json.md'
# The output json file's name. (optional)
json_file = 'jsons/trans_in_cnn.json'

The structure of the work-dir directory should be like：

├── work-dir
│   ├── folder1
│   │   ├── time1.log.json
│   │   ├── time2.log.json
│   │   ├── time3.log.json
│   │   ├── time4.log.json
│   ├── folder2
│   │   ├── time5.log.json
│   │   ├── time6.log.json
│   │   ├── time7.log.json
│   │   ├── time8.log.json

Then , cd to the log collector folder.

Now you can run log_collector.py by using command:

python log_collector.py ./example_config.py

The output markdown file is like:

exp_num	method	mIoU best	best index	mIoU last	last index	last iter num
1	segformer_mit-b5_512x512_160k_ade20k_cnn_lr_with_warmup	0.2776	10	0.2776	10	160000
2	segformer_mit-b5_512x512_160k_ade20k_cnn_no_warmup_lr	0.2802	10	0.2802	10	160000
3	segformer_mit-b5_512x512_160k_ade20k_mit_trans_lr	0.4943	11	0.4943	11	160000
4	segformer_mit-b5_512x512_160k_ade20k_swin_trans_lr	0.4883	11	0.4883	11	160000

The output json file is like:

[
    {
        "method": "segformer_mit-b5_512x512_160k_ade20k_cnn_lr_with_warmup",
        "metric_used": "mIoU",
        "last_iter": 160000,
        "last eval": {
            "eval_index": 10,
            "mIoU": 0.2776,
            "mAcc": 0.3779
        },
        "best eval": {
            "eval_index": 10,
            "mIoU": 0.2776,
            "mAcc": 0.3779
        }
    },
    {
        "method": "segformer_mit-b5_512x512_160k_ade20k_cnn_no_warmup_lr",
        "metric_used": "mIoU",
        "last_iter": 160000,
        "last eval": {
            "eval_index": 10,
            "mIoU": 0.2802,
            "mAcc": 0.3764
        },
        "best eval": {
            "eval_index": 10,
            "mIoU": 0.2802,
            "mAcc": 0.3764
        }
    },
    {
        "method": "segformer_mit-b5_512x512_160k_ade20k_mit_trans_lr",
        "metric_used": "mIoU",
        "last_iter": 160000,
        "last eval": {
            "eval_index": 11,
            "mIoU": 0.4943,
            "mAcc": 0.6097
        },
        "best eval": {
            "eval_index": 11,
            "mIoU": 0.4943,
            "mAcc": 0.6097
        }
    },
    {
        "method": "segformer_mit-b5_512x512_160k_ade20k_swin_trans_lr",
        "metric_used": "mIoU",
        "last_iter": 160000,
        "last eval": {
            "eval_index": 11,
            "mIoU": 0.4883,
            "mAcc": 0.6061
        },
        "best eval": {
            "eval_index": 11,
            "mIoU": 0.4883,
            "mAcc": 0.6061
        }
    }
]