mmengine/docs/en/design/visualization.md

Visualization

1 Overall Design

Visualization provides an intuitive explanation of the training and testing process of the deep learning model. In OpenMMLab, we expect the visualization module to meet the following requirements:

• Provides rich out-of-the-box features that can meet most computer vision visualization tasks.
• Versatile, expandable, and can be customized easily
• Able to visualize at anywhere in the training and testing process.
• Unified APIs for all OpenMMLab libraries, which is convenient for users to understand and use.

Based on the above requirements, we proposed the Visualizer and various VisBackend such as LocalVisBackend, WandbVisBackend, and TensorboardVisBackend in OpenMMLab 2.0. The visualizer could not only visualize the image data, but also things like configurations, scalars, and model structure.

• For convenience, the APIs provided by the Visualizer implement the drawing and storage functions. As an internal property of Visualizer, VisBackend will be called by Visualizer to write data to different backends.
• Considering that you may want to write data to multiple backends after drawing, Visualizer can be configured with multiple backends. When the user calls the storage API of the Visualizer, it will traverse and call all the specified APIs of VisBackend internally.

The UML diagram of the two is as follows.

2 Visualizer

The external interface of Visualizer can be divided into three categories.

1. Drawing APIs

• draw_bboxes draws a single or multiple bounding boxes
• draw_points draws a single or multiple points
• draw_texts draws a single or multiple text boxes
• draw_lines draws a single or multiple line segments
• draw_circles draws a single or multiple circles
• draw_polygons draws a single or multiple polygons
• draw_binary_masks draws single or multiple binary masks
• draw_featmap draws feature map (static method)

The above APIs can be called in a chain except for draw_featmap because the image size may change after this method is called. To avoid confusion, draw_featmap is a static method.

2. Storage APIs

• add_config writes configuration to a specific storage backend
• add_graph writes model graph to a specific storage backend
• add_image writes image to a specific storage backend
• add_scalar writes scalar to a specific storage backend
• add_scalars writes multiple scalars to a specific storage backend at once
• add_datasample the abstract interface for each repositories to draw data sample

Interfaces beginning with the add prefix represent storage APIs. [datasample] (./data_element.md)is the unified interface of each downstream repository in the OpenMMLab 2.0, and add_datasample can process the data sample directly .

3. Other APIs

• set_image sets the original image data, the default input image format is RGB
• get_image gets the image data in Numpy format after drawing, the default output format is RGB
• show for visualization
• get_backend gets a specific storage backend by name
• close closes all resources, including VisBackend

For more details, you can refer to Visualizer Tutorial.

3 VisBackend

After drawing, the drawn data can be stored in multiple visualization storage backends. To unify the interfaces, MMEngine provides an abstract class, BaseVisBackend, and some commonly used backends such as LocalVisBackend, WandbVisBackend, and TensorboardVisBackend. The main interfaces and properties of BaseVisBackend are as follows:

• add_config writes configuration to a specific storage backend
• add_graph writes model graph to a specific backend
• add_image writes image to a specific backend
• add_scalar writes scalar to a specific backend
• add_scalars writes multiple scalars to a specific backend at once
• close closes the resource that has been opened
• experiment writes backend objects, such as WandB objects and Tensorboard objects

BaseVisBackend defines five common data writing interfaces. Some writing backends are very powerful, such as WandB, which could write tables and videos. Users can directly obtain the experiment object for such needs and then call native APIs of the corresponding backend. LocalVisBackend, WandbVisBackend, and TensorboardVisBackend are all inherited from BaseVisBackend and implement corresponding storage functions according to their features. Users can also customize BaseVisBackend to extend the storage backends and implement custom storage requirements.

For more details, you can refer to Storage Backend Tutorial.