## Motivation
Through this PR, I (1) fix a bug, and (2) perform some associated cleanup, and (3) add a unit test. The bug occurs during evaluation when two options -- `reduce_zero_label=True`, and custom classes are used. The bug was that the `reduce_zero_label` is not properly propagated (see details below).
## Modification
1. **Bugfix**
The bug occurs [in the initialization of `CustomDataset`](5d49918b3c/mmseg/datasets/custom.py (L108-L110)) where the `reduce_zero_label` flag is not propagated to its member `self.gt_seg_map_loader_cfg`:
```python
self.gt_seg_map_loader = LoadAnnotations(
) if gt_seg_map_loader_cfg is None else LoadAnnotations(
**gt_seg_map_loader_cfg)
```
Because the `reduce_zero_label` flag was not being propagated, the zero label reduction was being [unnecessarily and explicitly duplicated during the evaluation](5d49918b3c/mmseg/core/evaluation/metrics.py (L66-L69)).
As pointed in a previous PR (#2500), `reduce_zero_label` must occur before applying the `label_map`. Due to this bug, the order gets reversed when both features are used simultaneously.
This has been fixed to:
```python
self.gt_seg_map_loader = LoadAnnotations(
reduce_zero_label=reduce_zero_label, **gt_seg_map_loader_cfg)
```
2. **Cleanup**
Due to the bug fix, since both `reduce_zero_label` and `label_map` are being applied in `get_gt_seg_map_by_idx()` (i.e. `LoadAnnotations.__call__()`), the evaluation does not need perform them anymore. However, for backwards compatibility, the evaluation keeps previous input arguments.
This was pointed out for `label_map` in a previous issue (#1415) that the `label_map` should not be applied in the evaluation. This was handled by [passing an empty dict](5d49918b3c/mmseg/datasets/custom.py (L306-L311)):
```python
# as the labels has been converted when dataset initialized
# in `get_palette_for_custom_classes ` this `label_map`
# should be `dict()`, see
# https://github.com/open-mmlab/mmsegmentation/issues/1415
# for more ditails
label_map=dict(),
reduce_zero_label=self.reduce_zero_label))
```
Similar to this, I now also set `reduce_label=False` since it is now also being handled by `get_gt_seg_map_by_idx()` (i.e. `LoadAnnotations.__call__()`).
3. **Unit test**
I've added a unit test that tests the `CustomDataset.pre_eval()` function when `reduce_zero_label=True` and custom classes are used. The test fails on the original `master` branch but passes with this fix.
## BC-breaking (Optional)
I do not anticipate this change braking any backward-compatibility.
## Checklist
- [x] Pre-commit or other linting tools are used to fix the potential lint issues.
- _I've fixed all linting/pre-commit errors._
- [x] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
- _I've added a test that passes when the fix is introduced, and fails on the original master branch._
- [x] If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
- _I don't think this change affects MMDet or MMDet3D._
- [x] The documentation has been modified accordingly, like docstring or example tutorials.
- _This change fixes an existing bug and doesn't require modifying any documentation/docstring._
* support iSAID aerial dataset
* Update and rename docs/dataset_prepare.md to 博士/dataset_prepare.md
* Update dataset_prepare.md
* fix typo
* fix typo
* fix typo
* remove imgviz
* fix wrong order in annotation name
* upload models&logs
* upload models&logs
* add load_annotations
* fix unittest coverage
* fix unittest coverage
* fix correct crop size in config
* fix iSAID unit test
* fix iSAID unit test
* fix typos
* fix wrong crop size in readme
* use smaller figure as test data
* add smaller dataset in test data
* add blank in docs
* use 0 bytes pseudo data
* add footnote and comments for crop size
* change iSAID to isaid and add default value in it
* change iSAID to isaid in _base_
Co-authored-by: MengzhangLI <mcmong@pku.edu.cn>
* update LoveDA dataset api
* revised lint errors in dataset_prepare.md
* revised lint errors in loveda.py
* revised lint errors in loveda.py
* revised lint errors in dataset_prepare.md
* revised lint errors in dataset_prepare.md
* checked with isort and yapf
* checked with isort and yapf
* checked with isort and yapf
* Revert "checked with isort and yapf"
This reverts commit 686a51d9
* Revert "checked with isort and yapf"
This reverts commit b877e121bb2935ceefc503c09675019489829feb.
* Revert "revised lint errors in dataset_prepare.md"
This reverts commit 2289e27c
* Revert "checked with isort and yapf"
This reverts commit 159db2f8
* Revert "checked with isort and yapf"
This reverts commit 159db2f8
* add configs & fix bugs
* update new branch
* upload models&logs and add format-only
* change pretraied model path of HRNet
* fix the errors in dataset_prepare.md
* fix the errors in dataset_prepare.md and configs in loveda.py
* change the description in docs_zh-CN/dataset_prepare.md
* use init_cfg
* fix test converage
* adding pseudo loveda dataset
* adding pseudo loveda dataset
* adding pseudo loveda dataset
* adding pseudo loveda dataset
* adding pseudo loveda dataset
* adding pseudo loveda dataset
* Update docs/dataset_prepare.md
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
* Update docs_zh-CN/dataset_prepare.md
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
* Update docs_zh-CN/dataset_prepare.md
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
* Delete unused lines of unittest and Add docs
* add convert .py file
* add downloading links from zenodo
* move place of LoveDA and Cityscapes in doc
* move place of LoveDA and Cityscapes in doc
Co-authored-by: MengzhangLI <mcmong@pku.edu.cn>
Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn>
* [Fix] #916 expection string type classes
* add unittests for string path classes
* fix double quote string in test_dataset.py
* move the import to the top of the file
* fix isort lint error
fix isort lint error when move the import to the top of the file
* [Feature] Add tool to show origin or augmented train data
* [Feature] Support eval concate dataset
* Add docstring and modify evaluate of concate dataset
Signed-off-by: FreyWang <wangwxyz@qq.com>
* format concat dataset in subfolder of imgfile_prefix
Signed-off-by: FreyWang <wangwxyz@qq.com>
* add unittest of concate dataset
Signed-off-by: FreyWang <wangwxyz@qq.com>
* update unittest for eval dataset with CLASSES is None
Signed-off-by: FreyWang <wangwxyz@qq.com>
* [FIX] bug of generator, which lead metric to nan when pre_eval=False
Signed-off-by: FreyWang <wangwxyz@qq.com>
* format code
Signed-off-by: FreyWang <wangwxyz@qq.com>
* add more unittest
* add more unittest
* optim concat dataset builder
* Support progressive test with fewer memory cost.
* Temp code
* Using processor to refactor evaluation workflow.
* refactor eval hook.
* Fix process bar.
* Fix middle save argument.
* Modify some variable name of dataset evaluate api.
* Modify some viriable name of eval hook.
* Fix some priority bugs of eval hook.
* Depreciated efficient_test.
* Fix training progress blocked by eval hook.
* Depreciated old test api.
* Fix test api error.
* Modify outer api.
* Build a sampler test api.
* TODO: Refactor format_results.
* Modify variable names.
* Fix num_classes bug.
* Fix sampler index bug.
* Fix grammaly bug.
* Support batch sampler.
* More readable test api.
* Remove some command arg and fix eval hook bug.
* Support format-only arg.
* Modify format_results of datasets.
* Modify tool which use test apis.
* support cityscapes eval
* fixed cityscapes
* 1. Add comments for batch_sampler;
2. Keep eval hook api same and add deprecated warning;
3. Add doc string for dataset.pre_eval;
* Add efficient_test doc string.
* Modify test tool to compat old version.
* Modify eval hook to compat with old version.
* Modify test api to compat old version api.
* Sampler explanation.
* update warning
* Modify deploy_test.py
* compatible with old output, add efficient test back
* clear logic of exclusive
* Warning about efficient_test.
* Modify format_results save folder.
* Fix bugs of format_results.
* Modify deploy_test.py.
* Update doc
* Fix deploy test bugs.
* Fix custom dataset unit tests.
* Fix dataset unit tests.
* Fix eval hook unit tests.
* Fix some imcompatible.
* Add pre_eval argument for eval hooks.
* Update eval hook doc string.
* Make pre_eval false in default.
* Add unit tests for dataset format_results.
* Fix some comments and bc-breaking bug.
* Fix pre_eval set cfg field.
* Remove redundant codes.
Co-authored-by: Jiarui XU <xvjiarui0826@gmail.com>
* add dice evaluation metric
* add dice evaluation metric
* add dice evaluation metric
* support 2 metrics
* support 2 metrics
* support 2 metrics
* support 2 metrics
* fix docstring
* use np.round once for all
* Support for custom classes
* Fix test
* Fix pre-commit
* Add pipeline logic for custom classes
* Fix minor issues, fix test
* Fix issues from PR review
* Fix tests
* Remove palette as str
* Rename old_to_new_ids to label_map
* Test for load_anns
* Remove get_palette function
* fixed temp
* Add subset of palette, remove palette as arg
* minor update
Co-authored-by: Jiarui XU <xvjiarui0826@gmail.com>
