Visualization provides an intuitive explanation of the training and testing process of the deep learning model.
MMEngine provides `Visualizer` to visualize and store the state and intermediate results of the model training and testing process, with the following features:
- It supports basic drawing interface and feature map visualization
- It enables recording training states (such as loss and lr), performance evaluation metrics, and visualization results to a specified or multiple backends, including local device, TensorBoard, and WandB.
- It can be used in any location in the code base.
## Basic Drawing APIs
`Visualizer` provides drawing APIs for common objects such as **detection bboxes, points, text, lines, circles, polygons, and binary masks**.
These APIs have the following features:
- Can be called multiple times to achieve overlay drawing requirements.
- All support multiple input types such as Tensor, Numpy array, etc.
- As the input Tensor usually includes multiple channels, `channel_reduction` can reduce them into a single channel and overlay the result to the image.
-`squeeze_mean` reduces the input channel C into a single channel using the mean function, so the output dimension becomes (1, H, W)
-`select_max` select the channel with the maximum activation, where 'activation' refers to the sum across spatial dimensions of a channel.
-`None` indicates that no reduction is needed, which allows the user to select the top k feature maps with the highest activation degree through the `topk` parameter.
-`topk` is only valid when the `channel_reduction` is `None`. It selects the top k channels according to the activation degree and then displays them overlaid with the image. The display layout can be specified using the `--arrangement` parameter.
- If `topk` is not -1, `topk` channels with the largest activation will be selected for display.
- If `topk` is -1, channel number C must be either 1 or 3 to indicate if the input is a picture. Otherwise, an error will be raised to prompt the user to reduce the channel with `channel_reduction`.
- Considering that the input feature map is usually very small, the function can upsample the feature map through `resize_shape` before the visualization.
For example, we would like to get the feature map from the layer4 output of a pre-trained ResNet18 model and visualize it.
1. Reduce the multi-channel feature map into a single channel using `select_max` and display it.
```python
import numpy as np
from torchvision.models import resnet18
from torchvision.transforms import Compose, Normalize, ToTensor
Since the output feat feature map size is 7x7, the visualization effect is not good if we directly work on it. Users can scale the feature map by overlaying the input image or the `resize_shape` parameter. If the size of the incoming image is not the same as the size of the feature map, the feature map will be forced to be resampled to the same spatial size as the input image.
Once the drawing is completed, users can choose to display the result directly or save it to different backends. The backends currently supported by MMEngine include local storage, `Tensorboard` and `WandB`. The data supported include drawn pictures, scalars, and configurations.
Any `Visualizer` can be configured with any number of storage backends. `Visualizer` will loop through all the configured backends and save the results to each one.
Note: If there are multiple backends used at the same time, the `name` field must be specified. Otherwise, it is impossible to distinguish which backend it is.
During the development, users may need to add visualization functions somewhere in their codes and save the results to different backends, which is very common for analysis and debugging. `Visualizer` in MMEngine can obtain the data from the same visualizers and then visualize them.
Users only need to instantiate the visualizer through `get_instance` during initialization. The visualizer obtained this way is unique and globally accessible. Then it can be accessed anywhere in the code through `Visualizer.get_current_instance()`.
The storage backend only provides basic functions such as saving configurations and scalars. However, users may want to utilize other powerful backend features like WandB and Tensorboard. Therefore, the storage backend provides the `experiment` attribute to facilitate users to obtain backend objects and meet various customized functions.
For example, WandB provides an API to display tables. Users can obtain the WandB objects through the `experiment` attribute and then call a specific API to save the data as a table to show.
Similarly, users can easily customize the visualizer by inheriting `Visualizer` and implementing the functions they want to override.
In most cases, users need to override `add_datasample`. The data usually includes detection bboxes and instance masks from annotations or model predictions. This interface is for drawing `datasample` data for various downstream libraries. Taking MMDetection as an example, the `datasample` data usually includes labeled bboxs, labeled masks, predicted bboxs, or predicted masks. MMDetection will inherit `Visualizer` and implement the `add_datasample` interface, drawing the data related to the detection task.
