mirrors
/
mmrazor
mirror of https://github.com/open-mmlab/mmrazor.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

[Doc] add documents about pruning. (#313) 

* init

* update user guide

* update images

* update

* update How to prune your model

* update how_to_use_config_tool_of_pruning.md

* update doc

* move location

* update

* update

* update

* add mutablechannels.md

* add references

Co-authored-by: liukai <liukai@pjlab.org.cn>
Co-authored-by: jacky <jacky@xx.com>
pull/304/head^2
LKJacky 2022-10-26 13:27:48 +08:00 committed by GitHub
parent db32b32e38
commit 5eaa22574d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 1354 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

134
docs/en/advanced_guides/tutorials/how_to_prune_your_model.md 100644
View File

@ -0,0 +1,134 @@
# How to prune your model
## Overview
This section will introduce you to pruning your model. Before that, we suggest you read the document [User Guides: Pruning Framework](../../user_guides/pruning_user_guide.md) to have an overview of our pruning framework.
First, we suppose your model is defined and trained using one openmmlab repo.
Our pruning algorithms work as a wrapper of a model. To prune your model, you need to replace your model config with our algorithm config, which has a parameter 'architecture' to store your original model. The pipeline is shown below.
<p align='center'><img src="../../imgs/pruning/draw-config.png" width=400 /></p>
After this replacement, the algorithm will prune your model during your training process.
## How to Config an Algorithm
All pruning algorithms are defined in mmrazor.models.algorithms.pruning. All algorithms have some shared pruning-related arguments, some specific arguments, and some shared mmengine.BaseModel arguments.
Here we take pruning resnet34 using the l1-norm algorithm as an example. We use "mmcls::resnet/resnet34_8xb32_in1k.py" as a base config. Then we override the model config and use the original model config as the architecture of 'ItePruneAlgorithm'.
```python
_base_ = ['mmcls::resnet/resnet34_8xb32_in1k.py']
stage_ratio_1 = 0.7
stage_ratio_2 = 0.7
stage_ratio_3 = 0.7
stage_ratio_4 = 1.0
target_pruning_ratio = {
'backbone.layer1.2.conv2_(0, 64)_64': stage_ratio_1,
'backbone.layer1.0.conv1_(0, 64)_64': stage_ratio_1,
'backbone.layer1.1.conv1_(0, 64)_64': stage_ratio_1,
'backbone.layer1.2.conv1_(0, 64)_64': stage_ratio_1,
'backbone.layer2.0.conv1_(0, 128)_128': stage_ratio_2,
'backbone.layer2.3.conv2_(0, 128)_128': stage_ratio_2,
'backbone.layer2.1.conv1_(0, 128)_128': stage_ratio_2,
'backbone.layer2.2.conv1_(0, 128)_128': stage_ratio_2,
'backbone.layer2.3.conv1_(0, 128)_128': stage_ratio_2,
'backbone.layer3.0.conv1_(0, 256)_256': stage_ratio_3,
'backbone.layer3.5.conv2_(0, 256)_256': stage_ratio_3,
'backbone.layer3.1.conv1_(0, 256)_256': stage_ratio_3,
'backbone.layer3.2.conv1_(0, 256)_256': stage_ratio_3,
'backbone.layer3.3.conv1_(0, 256)_256': stage_ratio_3,
'backbone.layer3.4.conv1_(0, 256)_256': stage_ratio_3,
'backbone.layer3.5.conv1_(0, 256)_256': stage_ratio_3,
'backbone.layer4.0.conv1_(0, 512)_512': stage_ratio_4,
'backbone.layer4.2.conv2_(0, 512)_512': stage_ratio_4,
'backbone.layer4.1.conv1_(0, 512)_512': stage_ratio_4,
'backbone.layer4.2.conv1_(0, 512)_512': stage_ratio_4
}
architecture = _base_.model
model = dict(
_scope_='mmrazor',
_delete_=True,
type='ItePruneAlgorithm',
architecture=architecture,
mutator_cfg=dict(
type='BaseChannelMutator',
channel_unit_cfg=dict(
type='L1MutableChannelUnit',
default_args=dict(choice_mode='ratio'))
parse_cfg=dict(
type='BackwardTracer',
loss_calculator=dict(type='ImageClassifierPseudoLoss')),
target_pruning_ratio=target_pruning_ratio,
step_epoch=1,
prune_times=1,
data_preprocessor=None,
init_cfg=None
)
```
**Shared pruning-related arguments**: All pruning algorithms have two shared pruning-related arguments.
- Architecture
- Architecture defines the model to be pruned. Usually, you need to pass your original model config to the argument.
- mutator_cfg
- The config of a mutator to manage the structure of your model. Usually, each algorithm has a frequently-used mutator. Please refer to the next section for more detail.
**Specific arguments**:
A algorithm may have its specific arguments. You need to read their documents to know how to config. Here, we only introduce the specific arguments of ItePruneAlgorithm.
- target_pruning_ratio: target_pruning_ratio is a dict that uses the name of units as keys and the choice values as values.. It indicates how many channels remain after pruning. You can use python ./tools/get_channel_units.py --choice {config_file} to get the choice template. Please refer to [How to Use our Config Tool for Pruning](./how_to_use_config_tool_of_pruning.md).
- step_epoch: the step between two pruning operations.
- prune_times: the times to prune to reach the pruning target. Here, we prune resnet34 once, so we set it to 1.
**Shared BaseModel arguments**:
Our algorithms inherit from BaseModel, so each algorithm has shared arguments from BaseModel.
- data_preprocessor: Used for pre-processing data sampled by dataloader to the format accepted by :meth:`forward`.
- init_cfg: Initialization config dict
## How to Config A Mutator
A mutator is used to manage the structure of a model.
Mutators have two augments:
- channel_unit_cfg: config of channel units. The config should follow the template below.
```python
channel_unit_cfg = dict(
# type of used MutableChannelUnit
type ='XxxMutableChannelUnit',
# default args for MutableChananelUnit
default_args={},
units = {
# config of a unit
"xxx_unit_name": {
"init_args":{},
"channels":{},
},
...
}
),
```
MutableChannelUnit decides how to generate a channel choice. It's important to choose the right MutableChannelUnit. Here, we choose 'L1MutableChannelUnit' to apply the l1-norm algorithm.
- parse_cfg: parse_cfg defines the method to parse the model and get channel units.
There are three ways used in BaseChannelMutator to parse a model and get MutableChannelUnits.
1. Using tracer. It needs parse_cfg to be the config of a tracer.
2. Using config. When parse_cfg\['type'\]='Config'. It needs that channel_unit_cfg\['unit'\]\['xxx_unit_name\] to have a key 'channels' to indicate channel units.
3. Using the model with pre-defined DynamicOps and MutableChannels: When parse_cfg\['type'\]='Predefined', the mutator will parse the dynamic ops in the model and get channel units.
In the example above, we directly use a tracer to parse the model.
We also provide a tool for you to configure the mutator, please refer to [How to Use our Config Tool for Pruning](./how_to_use_config_tool_of_pruning.md).
Besides, please refer to [ChannelMutator](../../../../mmrazor/models/mutators/channel_mutator/channel_mutator.ipynb) for more details about ChannelMutator.
## End
After configuring the algorithm, you can rerun the config file with a pretrained checkpoint to prune your model.

239
docs/en/advanced_guides/tutorials/how_to_use_config_tool_of_pruning.md 100644
View File

@ -0,0 +1,239 @@
# How to Use our Config Tool for Pruning
## How We Get MutableChannelUnits Automatically
Our pruning framework can automatically parse a model and get MutableChannelUnits.
It makes it easy to prune new models.
The parsing process is placed in ChannelUnitMutator.prepare_from_supernet. We first trace the model and get a graph, then we parse the graph and get MutableChannelUnits.
<p align='center'><img src="../../imgs/pruning/framework-graph.png" width=400 /></p>
## How to Get ChannelUnit Config Template
To make the configuration of ChannelUnit easy, we provide an interface to get the config template: ChannelMutator.config_template(). It returns a config dict. The config\['channel_unit_cfg'\]\['units\] store all parsed MutableChannelUnits.
```python
def config_template(self,
only_mutable_units=False,
with_unit_init_args=False,
with_channels=False):
"""Config template of the mutator.
Args:
only_mutable_units (bool, optional): Whether only return config of
prunable units. It can omit unmutable MutableChannelUnits
to decrease the length of the config. Defaults to False.
with_unit_init_args (bool, optional): Whether return init_args of
units. Let it be true, when you want to change the init
args of units. Defaults to False.
with_channels (bool, optional): Whether return channel info.
The channel info can initialization the units without
tracer. When you want to prune your model without a
tracer next time, let it be true. Defaults to False.
Example:
dict(
channel_unit_cfg = dict(
# type of used MutableChannelUnit
type ='XxxMutableChannelUnit',
# default args for MutableChananelUnit
default_args={},
# config of units
units = {
# config of a unit
"xxx_unit_name": {
'init_args':{}, # if with_unit_init_args
'channels':{} # if with_channels
},
...
}
),
# config of tracer
parse_cfg={}
)
About the detail of the config of each unit, please refer to
MutableChannelUnit.config_template()
"""
```
Note the name of a unit is generated automatically according to their content, avoid to change the name in config.
Here, we give an example of getting a config template using code.
```python
from mmrazor.models.mutators import ChannelMutator
from torchvision.models import resnet34
model = resnet34()
# initialize a ChannelMutator object
mutator = ChannelMutator(
channel_unit_cfg=dict(
type='SequentialMutableChannelUnit',
default_args=dict(choice_mode='ratio'),
units={},
),
parse_cfg=dict(
type='BackwardTracer',
loss_calculator=dict(type='ImageClassifierPseudoLoss')))
# init the ChannelMutator object with a model
mutator.prepare_from_supernet(model)
config=mutator.config_template(with_unit_init_args=True)
print(config)
# {
# 'type': 'ChannelMutator',
# 'channel_unit_cfg': {
# 'type': 'SequentialMutableChannelUnit',
# 'default_args': {
# 'choice_mode': 'ratio'
# },
# 'units': {
# 'conv1_(0, 3)_3': {
# 'init_args': {
# 'num_channels': 3,
# 'choice_mode': 'ratio',
# ...
# },
# 'choice': 1.0
# },
# ...
# }
# },
# 'parse_cfg': {
# 'type': 'BackwardTracer',
# 'loss_calculator': {
# 'type': 'ImageClassifierPseudoLoss'
# }
# }
# }
```
Besides, it's also easy to initialize a new mutator using the config dict.
```python
# follow the code above
from mmrazor.registry import MODELS
mutator2=MODELS.build(config)
mutator2.prepare_from_supernet(resnet34())
```
To make your development more fluent, we provide a command tool to parse a model and return the config template.
```shell
$ python ./tools/get_channel_units.py -h
usage: get_channel_units.py [-h] [-c] [-i] [--choice] [-o OUTPUT_PATH] config
Get channel unit of a model.
positional arguments:
config config of the model
optional arguments:
-h, --help show this help message and exit
-c, --with-channel output with channel config
-i, --with-init-args output with init args
--choice output choices template. When this flag is activated, -c and -i will be ignored
-o OUTPUT_PATH, --output-path OUTPUT_PATH
the file path to store channel unit info
```
Take the algorithm Slimmable Network as an example.
```shell
python ./tools/get_channel_units.py ./configs/pruning/mmcls/autoslim/autoslim_mbv2_1.5x_slimmable_subnet_8xb256_in1k.py
# {
# "type":"SlimmableChannelMutator",
# "channel_unit_cfg":{
# "type":"SlimmableChannelUnit",
# "default_args":{},
# "units":{
# "backbone.conv1.conv_(0, 3)_3":{
# "choice":3
# },
# "backbone.conv1.conv_(0, 48)_48":{
# "choice":32
# },
...
# }
# },
# "parse_cfg":{
# "type":"BackwardTracer",
# "loss_calculator":{
# "type":"ImageClassifierPseudoLoss"
# }
# }
# }
```
The '-i' flag will return the config with the initialization arguments.
```shell
python ./tools/get_channel_units.py -i ./configs/pruning/mmcls/autoslim/autoslim_mbv2_1.5x_slimmable_subnet_8xb256_in1k.py
# {
# "type":"SlimmableChannelMutator",
# "channel_unit_cfg":{
# "type":"SlimmableChannelUnit",
# "default_args":{},
# "units":{
# "backbone.conv1.conv_(0, 3)_3":{
# "init_args":{
# "num_channels":3,
# "divisor":1,
# "min_value":1,
# "min_ratio":0.9,
# "candidate_choices":[
# 3
# ],
# "choice_mode":"number"
# },
# "choice":3
# },
# ...
# }
# },
# "parse_cfg":{
# "type":"BackwardTracer",
# "loss_calculator":{
# "type":"ImageClassifierPseudoLoss"
# }
# }
# }
```
With "--choice" flag, it will return the choice template, a dict which uses unit_name as key, and use the choice value as value.
```shell
python ./tools/get_channel_units.py -i ./configs/pruning/mmcls/autoslim/autoslim_mbv2_1.5x_slimmable_subnet_8xb256_in1k.py --choice
# {
# "backbone.conv1.conv_(0, 48)_48":32,
# "backbone.layer1.0.conv.1.conv_(0, 24)_24":16,
# "backbone.layer2.0.conv.0.conv_(0, 144)_144":144,
# "backbone.layer2.0.conv.2.conv_(0, 40)_40":24,
# "backbone.layer2.1.conv.0.conv_(0, 240)_240":176,
# "backbone.layer3.0.conv.0.conv_(0, 240)_240":192,
# "backbone.layer3.0.conv.2.conv_(0, 48)_48":48,
# "backbone.layer3.1.conv.0.conv_(0, 288)_288":240,
# "backbone.layer3.2.conv.0.conv_(0, 288)_288":144,
# "backbone.layer4.0.conv.0.conv_(0, 288)_288":264,
# "backbone.layer4.0.conv.2.conv_(0, 96)_96":88,
# "backbone.layer4.1.conv.0.conv_(0, 576)_576":288,
# "backbone.layer4.2.conv.0.conv_(0, 576)_576":336,
# "backbone.layer4.3.conv.0.conv_(0, 576)_576":432,
# "backbone.layer5.0.conv.0.conv_(0, 576)_576":576,
# "backbone.layer5.0.conv.2.conv_(0, 144)_144":144,
# "backbone.layer5.1.conv.0.conv_(0, 864)_864":576,
# "backbone.layer5.2.conv.0.conv_(0, 864)_864":648,
# "backbone.layer6.0.conv.0.conv_(0, 864)_864":864,
# "backbone.layer6.0.conv.2.conv_(0, 240)_240":240,
# "backbone.layer6.1.conv.0.conv_(0, 1440)_1440":1440,
# "backbone.layer6.2.conv.0.conv_(0, 1440)_1440":1440,
# "backbone.layer7.0.conv.0.conv_(0, 1440)_1440":1440,
# "backbone.layer7.0.conv.2.conv_(0, 480)_480":480,
# "backbone.conv2.conv_(0, 1920)_1920":1920
# }
```

BIN
docs/en/imgs/pruning/draw-config.png 100644
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 27 KiB

BIN
docs/en/imgs/pruning/framework-ChanelMutator.png 100644
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

BIN
docs/en/imgs/pruning/framework-algorithm.png 100644
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
docs/en/imgs/pruning/framework-framework.png 100644
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

BIN
docs/en/imgs/pruning/framework-graph.png 100644
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 16 KiB

BIN
docs/en/imgs/pruning/framework-op.png 100644
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 9.5 KiB

BIN
docs/en/imgs/pruning/pruning_framework.png 100644
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
docs/en/imgs/pruning/unit.png 100644
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 55 KiB

148
docs/en/user_guides/pruning_user_guide.md 100644
View File

@ -0,0 +1,148 @@
# User Guides: Pruning Framework
## Background
// TODO
## Pruning Framework
This document introduces the pruning framework in mmrazor. Our pruning framework can help you prune a model automatically, making it easy to extend new algorithms.
The pruning framework consists of five modules: Algorithm, ChanelMutator, MutableChannelUnit, MutableChannel, and DynamicOp. Their main features are detailed below:
| Module | Features |
| ------------------ | --------------------------------------------------------------------- |
| Algorithm | Controls training process. |
| ChanelMutator | Manages the pruning structure of the model. |
| MutableChannelUnit | Makes pruning decisions. |
| MutableChannel | Manage a channel mask. |
| DynamicOp | Forwards with mutable number of channels, and exports pruned modules. |
<p align="center">
<img src="../imgs/pruning/framework-framework.png" width="250"/>
</p>
## Algorithm
<p align="center">
<img src="../imgs/pruning/framework-algorithm.png" width="400">
</p>
Algorithms inherit from BaseAlgorithm. They control the training process, like deciding when to prune the model in the training/finetune process.
For example, IteAlgorithm prunes the model iteratively by certain epochs.
Here is an example of how to use PruneAlgoritm.
```python
from mmrazor.models.algorithms import IteAlgorithm
from mmengine.model import BaseModel
import torch.nn as nn
class Model(BaseModel):
def __init__(self):
super().__init__()
self.conv = nn.Conv2d(3, 8, 3, 1, 1)
def forward(self, x):
return self.conv(x)
model = Model()
algorithm = IteAlgorithm(model,
mutator_cfg=dict(
type='ChannelMutator',
channl_unit_cfg=dict(type='L1ChannelUnit')),)
print(algorithm)
# IteAlgorithm(
# (data_preprocessor): BaseDataPreprocessor()
# (architecture): Model(
# (data_preprocessor): BaseDataPreprocessor()
# (conv): DynamicConv2d(
# 3, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)
# (mutable_attrs): ModuleDict(
# (in_channels): MutableChannelContainer(name=, num_channels=3, activated_channels: 3
# (out_channels): MutableChannelContainer(name=, num_channels=8, activated_channels: 8
# )
# )
# )
# (mutator): BaseChannelMutator()
# )
```
## ChanelMutator
<p align="center"><img src="../imgs/pruning/framework-ChanelMutator.png" width="500"></p>
A ChanelMutator controls the pruning structure of a model. In other words, ChanelMutator decides how many channels each layer prunes. Usually, given a pruning target, such as a flops, latency, or pruning ratio target, the ChannelUnitMutator will output a pruning structure for the model. The pruning structure is variable. The default definition is the remaining channel ratio, and it's also easy to extend to the number of channels or channel buckets.
As some layers' channels are related, the related layers share one pruning decision. We put these associated layers into a MutableChannelUnit. Therefore, the ChanelMutator directly decides the pruning ratio of each MutableChannelUnit.
```python
from mmrazor.models.mutators import BaseChannelMutator
from mmengine.model import BaseModel
import torch.nn as nn
class Model(BaseModel):
def __init__(self):
super().__init__()
self.feature = nn.Sequential(
nn.Conv2d(3, 8, 3, 2, 1),
nn.Conv2d(8, 16, 3, 2, 1)
)
self.pool = nn.AdaptiveAvgPool2d(1)
self.head = nn.Linear(16, 1000)
def forward(self, x):
x_ = self.pool(self.feature(x))
return self.head(x_.flatten(1))
model = Model()
mutator = BaseChannelMutator()
mutator.prepare_from_supernet(model)
print(mutator.sample_choices())
# {
# 'feature.0_(0, 8)_out_1_in_1': 0.5,
# 'feature.1_(0, 16)_out_1_in_1': 0.5625
# }
```
Please refer to [ChannelMutator](../../../mmrazor/models/mutables/mutable_channel/units/mutable_channel_unit.ipynb) for more details.
## MutableChannelUnit
<p align="center"><img src="../imgs/pruning/unit.png" width="700"></p>
Because some layers' channels are related, the related layers are collected and put in a MutableChannelUnit.
Each MutableChannelUnit accepts a pruning ratio and generates a channel mask for all related layers.
All related layers are divided into two types: output_related and input_related.
- The output channels of output-related layers are in the MutableChannelUnit.
- The input channels of input-related layers are in the MutableChannelUnit.
Please refer to [MutableChannelUnit](../../../mmrazor/models/mutators/channel_mutator/channel_mutator.ipynb) for more details.
Besides, basic PyTorch modules are converted to DynamicOps, which can deal with a mutable number of channels with MutableChannels.
## DynamicOP && MutableChannel
<p align="center"><img src="../imgs/pruning/framework-op.png" width="300"></p>
**MutableChannel**: Each MutableChannel manages a channel mask for a model. They help DynamicOps to deal with mutable numbers of channels. Please refer to [MutableChannel](../../../mmrazor/models/mutables/mutable_channel/MutableChannel.md) for more details.
**DynamicOp**: DynamicOps inherit from basic torch modules, like nn.Conv2d or nn.Linear. They can forward with mutable numbers of channels and export pruned torch modules.
Compared with basic torch modules, each DynamicOp has two MutableChannel modules, which control the input and output channels.
## More Documents about Pruning
Please refer to the following documents for more details.
- Development tutorials
- [How to prune your model](../advanced_guides/tutorials/how_to_prune_your_model.md)
- [How to use config tool of pruning](../advanced_guides/tutorials/how_to_use_config_tool_of_pruning.md)
- READMEs
- [MutableChannel](../../../mmrazor/models/mutables/mutable_channel/MutableChannel.md)
- [ChannelMutator](../../../mmrazor/models/mutables/mutable_channel/units/mutable_channel_unit.ipynb)
- [MutableChannelUnit](../../../mmrazor/models/mutators/channel_mutator/channel_mutator.ipynb)

36
mmrazor/models/mutables/mutable_channel/MutableChannel.md 100644
View File

@ -0,0 +1,36 @@
# MutableChannels
MutableChannels are used to deal with mutable number of channels in DynamicOps.
```
|-----------------------------------------|
| mutable_in_channel(BaseMutableChannel) |
| --------------------------------------- |
| DynamicOp |
| --------------------------------------- |
| mutable_out_channel(BaseMutableChannel) |
| --------------------------------------- |
```
\`
All MutableChannels inherit from BaseMutableChannel. Each MutableChannel has to implement two property.
- current_choice: get and set the choice of the MutableChannel.
- current_mask: get the channel mask according to the current_choice.
## MutableChannelContainer
Here, we introduce a special MutableChannel: MutableChannelContainer. As the channels of a DynamicOp may belong to different MutableChannelUnits, we use MutableChannelContainers to store multiple MutableChannels as below.
```
-----------------------------------------------------------
| MutableChannelContainer |
-----------------------------------------------------------
|MutableChannel1| MutableChannel2 |MutableChannel3|
-----------------------------------------------------------
```
MutableChannelContainer has an method to register MutableChannels.
- register_mutable: register/store BaseMutableChannel in the
MutableChannelContainer

419
mmrazor/models/mutables/mutable_channel/units/mutable_channel_unit.ipynb 100644
View File

@ -0,0 +1,419 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# MutableChannelUnit"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Each MutableChannelUnit is a basic unit for pruning. It records all channels which are dependent on each other.\n",
"Below, we will introduce you about:\n",
"1. The data structure of MutableChannelUnit.\n",
"2. How to prune the model with a MutableChannelUnit.\n",
"3. How to get MutableChannelUnits.\n",
"4. How to develop a new MutableChannelUnit for a new pruning algorithm.\n",
"<p align=\"center\"><img src=\"../../../../../docs/en/imgs/pruning/unit.png\" alt=\"MutableChannelUnit\" width=\"800\"></p>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The Data Structure of MutableChannelUnit"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, let's parse a model and get several MutableChannelUnits."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# define a model\n",
"from mmengine.model import BaseModel\n",
"from torch import nn\n",
"import torch\n",
"from collections import OrderedDict\n",
"\n",
"class MyModel(BaseModel):\n",
"\n",
" def __init__(self):\n",
" super().__init__(None, None)\n",
" self.net = nn.Sequential(\n",
" OrderedDict([('conv0', nn.Conv2d(3, 8, 3, 1, 1)),\n",
" ('relu', nn.ReLU()),\n",
" ('conv1', nn.Conv2d(8, 16, 3, 1, 1))]))\n",
" self.pool = nn.AdaptiveAvgPool2d(1)\n",
" self.head = nn.Linear(16, 1000)\n",
"\n",
" def forward(self, x):\n",
" feature = self.net(x)\n",
" pool = self.pool(feature).flatten(1)\n",
" return self.head(pool)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"This model has 4 MutableChannelUnit(SequentialMutableChannelUnit).\n"
]
}
],
"source": [
"# There are multiple types of MutableChannelUnits. Here, We take SequentialMutableChannelUnit as the example.\n",
"from mmrazor.models.mutables.mutable_channel.units import SequentialMutableChannelUnit\n",
"from mmrazor.structures.graph import ModuleGraph\n",
"from typing import List\n",
"\n",
"model = MyModel()\n",
"graph = ModuleGraph.init_from_backward_tracer(model)\n",
"units: List[\n",
" SequentialMutableChannelUnit] = SequentialMutableChannelUnit.init_from_graph(graph) # type: ignore\n",
"print(\n",
" f'This model has {len(units)} MutableChannelUnit(SequentialMutableChannelUnit).'\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SequentialMutableChannelUnit(\n",
" name=net.conv0_(0, 8)_8\n",
" (output_related): ModuleList(\n",
" (0): Channel(net.conv0, index=(0, 8), is_output_channel=true, expand_ratio=1)\n",
" )\n",
" (input_related): ModuleList(\n",
" (0): Channel(net.conv1, index=(0, 8), is_output_channel=false, expand_ratio=1)\n",
" )\n",
" (mutable_channel): SquentialMutableChannel(num_channels=8, activated_channels=8)\n",
")\n"
]
}
],
"source": [
"unit1=units[1]\n",
"print(unit1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As shown above, each MutableChannelUnit has four important attributes: \n",
"1. name: str\n",
"2. output_related: ModuleList\n",
"3. input_related: ModuleList\n",
"4. mutable_channel: BaseMutableChannel\n",
"\n",
"\"name\" is the identifier of the MutableChannelUnit. It's automatically generated usually.\n",
"\n",
"\"output_related\" and \"input_related\" are two ModuleLists. They store all Channels with channel dependency.\n",
"The difference is that the \"output_related\" includes output channels and the \"input_related\" includes input channels.\n",
"All these channels\n",
"\n",
"\"mutable_channel\" is a BaseMutableChannel used to control the channel mask of modules. The mutable_channel is registered to the modules whose channels are stored in \"output_related\" and \"input_related\"."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to prune the model with a MutableChannelUnit."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are three steps to prune the model using a MutableChannelUnit:\n",
"1. replace modules, whose channel are stored in the \"output_related\" and \"input_related\", with dynamic ops which are able to deal with mutable number of channels.\n",
"2. register the \"mutable_channel\" to the replaced dynamic ops.\n",
"3. change the choice of the \"mutable_channel\".\n",
"\n",
"For simplicity, we run step 1 and 2 with one method \"prepare_for_pruning\"."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The current choice of unit1 is 8.\n",
"DynamicConv2d(\n",
" 3, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)\n",
" (mutable_attrs): ModuleDict(\n",
" (in_channels): MutableChannelContainer(num_channels=3, activated_channels=3)\n",
" (out_channels): MutableChannelContainer(num_channels=8, activated_channels=8)\n",
" )\n",
")\n",
"DynamicConv2d(\n",
" 8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)\n",
" (mutable_attrs): ModuleDict(\n",
" (in_channels): MutableChannelContainer(num_channels=8, activated_channels=8)\n",
" (out_channels): MutableChannelContainer(num_channels=16, activated_channels=16)\n",
" )\n",
")\n"
]
}
],
"source": [
"# We run \"prepare_for_pruning\" once before pruning to run step 1 and 2 above.\n",
"unit1.prepare_for_pruning(model)\n",
"print(f'The current choice of unit1 is {unit1.current_choice}.')\n",
"print(model.net.conv0)\n",
"print(model.net.conv1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We prune the model by changing the current_choice of the MutableChannelUnits."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"We get a sampled choice 2.\n",
"DynamicConv2d(\n",
" 3, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)\n",
" (mutable_attrs): ModuleDict(\n",
" (in_channels): MutableChannelContainer(num_channels=3, activated_channels=3)\n",
" (out_channels): MutableChannelContainer(num_channels=8, activated_channels=2)\n",
" )\n",
")\n",
"DynamicConv2d(\n",
" 8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)\n",
" (mutable_attrs): ModuleDict(\n",
" (in_channels): MutableChannelContainer(num_channels=8, activated_channels=2)\n",
" (out_channels): MutableChannelContainer(num_channels=16, activated_channels=16)\n",
" )\n",
")\n"
]
}
],
"source": [
"sampled_choice=unit1.sample_choice()\n",
"print(f'We get a sampled choice {sampled_choice}.')\n",
"unit1.current_choice=sampled_choice\n",
"print(model.net.conv0)\n",
"print(model.net.conv1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Besides, different types of MutableChannelUnit may have different types of choices. Please read documents for more details."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to get MutableChannelUnits."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are three ways to get MutableChannelUnits.\n",
"1. Using a tracer.\n",
" This way, firstly, converts a model to a graph, then converts the graph to MutableChannelUnits. It automatically returns all available MutableChannelUnits.\n",
"2. Using a config.\n",
" This way uses a config to initialize a MutableChannelUnit.\n",
"3. Using a predefined model.\n",
" This way parses a predefined model with dynamic ops. It returns all available MutableChannelUnits.\n",
"\n",
"All these three ways have corresponding documents in the README of ChannelMutator."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The model has 4 MutableChannelUnits.\n"
]
}
],
"source": [
"# 1. using tracer\n",
"def get_mutable_channel_units_using_tracer(model):\n",
" graph = ModuleGraph.init_from_backward_tracer(model)\n",
" units = SequentialMutableChannelUnit.init_from_graph(graph)\n",
" return units\n",
"\n",
"\n",
"model = MyModel()\n",
"units = get_mutable_channel_units_using_tracer(model)\n",
"print(f'The model has {len(units)} MutableChannelUnits.')"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"SequentialMutableChannelUnit(\n",
" name=net.conv0_(0, 8)_8\n",
" (output_related): ModuleList(\n",
" (0): Channel(net.conv0, index=(0, 8), is_output_channel=true, expand_ratio=1)\n",
" )\n",
" (input_related): ModuleList(\n",
" (0): Channel(net.conv1, index=(0, 8), is_output_channel=false, expand_ratio=1)\n",
" )\n",
" (mutable_channel): SquentialMutableChannel(num_channels=8, activated_channels=8)\n",
")\n"
]
}
],
"source": [
"# 2. using config\n",
"config = {\n",
" 'init_args': {\n",
" 'num_channels': 8,\n",
" },\n",
" 'channels': {\n",
" 'input_related': [{\n",
" 'name': 'net.conv1',\n",
" }],\n",
" 'output_related': [{\n",
" 'name': 'net.conv0',\n",
" }]\n",
" },\n",
" 'choice': 8\n",
"}\n",
"unit=SequentialMutableChannelUnit.init_from_cfg(model, config)\n",
"print(unit)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The model has 2 MutableChannelUnits.\n"
]
}
],
"source": [
"# 3. using predefined model\n",
"\n",
"from mmrazor.models.architectures.dynamic_ops import DynamicConv2d, DynamicLinear\n",
"from mmrazor.models.mutables import MutableChannelUnit, MutableChannelContainer,SquentialMutableChannel\n",
"from collections import OrderedDict\n",
"\n",
"class MyDynamicModel(BaseModel):\n",
"\n",
" def __init__(self):\n",
" super().__init__(None, None)\n",
" self.net = nn.Sequential(\n",
" OrderedDict([('conv0', DynamicConv2d(3, 8, 3, 1, 1)),\n",
" ('relu', nn.ReLU()),\n",
" ('conv1', DynamicConv2d(8, 16, 3, 1, 1))]))\n",
" self.pool = nn.AdaptiveAvgPool2d(1)\n",
" self.head = DynamicLinear(16, 1000)\n",
"\n",
" # register MutableChannelContainer\n",
" MutableChannelUnit._register_channel_container(\n",
" self, MutableChannelContainer)\n",
" self._register_mutables()\n",
"\n",
" def forward(self, x):\n",
" feature = self.net(x)\n",
" pool = self.pool(feature).flatten(1)\n",
" return self.head(pool)\n",
"\n",
" def _register_mutables(self):\n",
" mutable1 = SquentialMutableChannel(8)\n",
" mutable2 = SquentialMutableChannel(16)\n",
" MutableChannelContainer.register_mutable_channel_to_module(\n",
" self.net.conv0, mutable1, is_to_output_channel=True)\n",
" MutableChannelContainer.register_mutable_channel_to_module(\n",
" self.net.conv1, mutable1, is_to_output_channel=False)\n",
"\n",
" MutableChannelContainer.register_mutable_channel_to_module(\n",
" self.net.conv1, mutable2, is_to_output_channel=True)\n",
" MutableChannelContainer.register_mutable_channel_to_module(\n",
" self.head, mutable2, is_to_output_channel=False)\n",
"model=MyDynamicModel()\n",
"units=SequentialMutableChannelUnit.init_from_predefined_model(model) \n",
"print(f'The model has {len(units)} MutableChannelUnits.')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.9.12 ('mmlab')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "feec882ee78c63cb8d4b485f1b52bbb873bb9a7b094435863200c7afba202382"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}

365
mmrazor/models/mutators/channel_mutator/channel_mutator.ipynb 100644
View File

@ -0,0 +1,365 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# ChannelMutator\n",
"A channel mutator is a manager of the channel structure of a model. In other words, it manages all MutableChannelUnits of a model. \n",
"ChannelMutator is the simplest channel mutator. All other channel mutators should inherit from ChannelMutator class. We take ChannelMutator as an example."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to Construct a ChannelMutator"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Suppose we have a model archtecture defineed below"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"# define a model\n",
"from mmengine.model import BaseModel\n",
"from torch import nn\n",
"import torch\n",
"from collections import OrderedDict\n",
"\n",
"class MyModel(BaseModel):\n",
"\n",
" def __init__(self):\n",
" super().__init__(None, None)\n",
" self.net = nn.Sequential(\n",
" OrderedDict([('conv0', nn.Conv2d(3, 8, 3, 1, 1)),\n",
" ('relu', nn.ReLU()),\n",
" ('conv1', nn.Conv2d(8, 16, 3, 1, 1))]))\n",
" self.pool = nn.AdaptiveAvgPool2d(1)\n",
" self.head = nn.Linear(16, 1000)\n",
"\n",
" def forward(self, x):\n",
" feature = self.net(x)\n",
" pool = self.pool(feature).flatten(1)\n",
" return self.head(pool)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are two steps to fully constructing a ChannelMutator object as below. \n",
"1. we need to initialize a ChannelMutator object.\n",
"2. Then we need to init the ChannelMutator object with a model."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The mutator has 2 mutable channel units.\n"
]
}
],
"source": [
"from mmrazor.models.mutators import ChannelMutator\n",
"\n",
"model = MyModel()\n",
"# initialize a ChannelMutator object\n",
"mutator = ChannelMutator(\n",
" channel_unit_cfg=dict(\n",
" type='SequentialMutableChannelUnit',\n",
" default_args=dict(choice_mode='ratio'),\n",
" units={},\n",
" ),\n",
" parse_cfg=dict(\n",
" type='BackwardTracer',\n",
" loss_calculator=dict(type='ImageClassifierPseudoLoss')))\n",
"# init the ChannelMutator object with a model\n",
"mutator.prepare_from_supernet(model)\n",
"print(f'The mutator has {len(mutator.mutable_units)} mutable channel units.')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"ChannelMutator has two arguments:\n",
"1. channel_unit_cfg: config of the MutableChannelUnit to use in the ChannelMutator.\n",
"2. parse_cfg: the way to parse the model and get MutableChannelUnits."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are there ways to parse model and get MutableChannelUnits.\n",
"1. Use a tracer to get MutableChannelUnits automatically.\n",
"2. Use config dicts to indicate MutableChannelUnits.\n",
"3. Predefine MutableChannels in the model archtecture.\n",
" \n",
"The example of method 1 has been post above. We post the examples of method 2 and method 3 below."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The mutator has 2 mutable channel units.\n"
]
}
],
"source": [
"# 2. use config dicts to indicate MutableChannelUnits.\n",
"from mmrazor.models.mutators import ChannelMutator\n",
"\n",
"model = MyModel()\n",
"# initialize a ChannelMutator object\n",
"mutator = ChannelMutator(\n",
" channel_unit_cfg=dict(\n",
" type='SequentialMutableChannelUnit',\n",
" default_args=dict(choice_mode='ratio'),\n",
" units={\n",
" 'net.conv0_(0, 8)_8': {\n",
" 'init_args': {\n",
" 'num_channels': 8,\n",
" },\n",
" 'channels': {\n",
" 'input_related': [{\n",
" 'name': 'net.conv1',\n",
" }],\n",
" 'output_related': [{\n",
" 'name': 'net.conv0',\n",
" }]\n",
" },\n",
" 'choice': 1.0\n",
" },\n",
" 'net.conv1_(0, 16)_16': {\n",
" 'init_args': {\n",
" 'num_channels': 16,\n",
" },\n",
" 'channels': {\n",
" 'input_related': [{\n",
" 'name': 'head',\n",
" }],\n",
" 'output_related': [{\n",
" 'name': 'net.conv1',\n",
" }]\n",
" },\n",
" 'choice': 1.0\n",
" }\n",
" }),\n",
" parse_cfg=dict(type='Config'))\n",
"# init the ChannelMutator object with a model\n",
"mutator.prepare_from_supernet(model)\n",
"print(f'The mutator has {len(mutator.mutable_units)} mutable channel units.')"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The mutator has 2 mutable channel units.\n"
]
}
],
"source": [
"# 3. Predefine MutableChannels in the model archtecture.\n",
"\n",
"from mmrazor.models.architectures.dynamic_ops import DynamicConv2d, DynamicLinear\n",
"from mmrazor.models.mutables import MutableChannelUnit, MutableChannelContainer, SquentialMutableChannel\n",
"from collections import OrderedDict\n",
"\n",
"class MyDynamicModel(BaseModel):\n",
"\n",
" def __init__(self):\n",
" super().__init__(None, None)\n",
" self.net = nn.Sequential(\n",
" OrderedDict([('conv0', DynamicConv2d(3, 8, 3, 1, 1)),\n",
" ('relu', nn.ReLU()),\n",
" ('conv1', DynamicConv2d(8, 16, 3, 1, 1))]))\n",
" self.pool = nn.AdaptiveAvgPool2d(1)\n",
" self.head = DynamicLinear(16, 1000)\n",
"\n",
" # register MutableChannelContainer\n",
" MutableChannelUnit._register_channel_container(\n",
" self, MutableChannelContainer)\n",
" self._register_mutables()\n",
"\n",
" def forward(self, x):\n",
" feature = self.net(x)\n",
" pool = self.pool(feature).flatten(1)\n",
" return self.head(pool)\n",
"\n",
" def _register_mutables(self):\n",
" mutable1 = SquentialMutableChannel(8)\n",
" mutable2 = SquentialMutableChannel(16)\n",
" MutableChannelContainer.register_mutable_channel_to_module(\n",
" self.net.conv0, mutable1, is_to_output_channel=True)\n",
" MutableChannelContainer.register_mutable_channel_to_module(\n",
" self.net.conv1, mutable1, is_to_output_channel=False)\n",
"\n",
" MutableChannelContainer.register_mutable_channel_to_module(\n",
" self.net.conv1, mutable2, is_to_output_channel=True)\n",
" MutableChannelContainer.register_mutable_channel_to_module(\n",
" self.head, mutable2, is_to_output_channel=False)\n",
"\n",
"\n",
"model = MyDynamicModel()\n",
"# initialize a ChannelMutator object\n",
"mutator = ChannelMutator(\n",
" channel_unit_cfg=dict(\n",
" type='SequentialMutableChannelUnit',\n",
" default_args=dict(choice_mode='ratio'),\n",
" units={},\n",
" ),\n",
" parse_cfg=dict(type='Predefined'))\n",
"# init the ChannelMutator object with a model\n",
"mutator.prepare_from_supernet(model)\n",
"print(f'The mutator has {len(mutator.mutable_units)} mutable channel units.')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to Change the Structure of a Model"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The structure of a model is represented by a dict where the key is the name of a MutableChannelUnit and the value is a structure choice."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'net.conv0_(0, 8)_8': 8, 'net.conv1_(0, 16)_16': 16}\n"
]
}
],
"source": [
"print(mutator.choice_template)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can change the dict to prune the model."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MyDynamicModel(\n",
" (data_preprocessor): BaseDataPreprocessor()\n",
" (net): Sequential(\n",
" (conv0): DynamicConv2d(\n",
" 3, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)\n",
" (mutable_attrs): ModuleDict(\n",
" (in_channels): MutableChannelContainer(num_channels=3, activated_channels=3)\n",
" (out_channels): MutableChannelContainer(num_channels=8, activated_channels=6)\n",
" )\n",
" )\n",
" (relu): ReLU()\n",
" (conv1): DynamicConv2d(\n",
" 8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)\n",
" (mutable_attrs): ModuleDict(\n",
" (in_channels): MutableChannelContainer(num_channels=8, activated_channels=6)\n",
" (out_channels): MutableChannelContainer(num_channels=16, activated_channels=8)\n",
" )\n",
" )\n",
" )\n",
" (pool): AdaptiveAvgPool2d(output_size=1)\n",
" (head): DynamicLinear(\n",
" in_features=16, out_features=1000, bias=True\n",
" (mutable_attrs): ModuleDict(\n",
" (in_features): MutableChannelContainer(num_channels=16, activated_channels=8)\n",
" (out_features): MutableChannelContainer(num_channels=1000, activated_channels=1000)\n",
" )\n",
" )\n",
")\n"
]
}
],
"source": [
"mutator.set_choices(\n",
" {'net.conv0_(0, 8)_8': 0.75, 'net.conv1_(0, 16)_16': 0.5}\n",
")\n",
"print(model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Please refer to our documents for more choices related methods."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.9.12 ('mmlab')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "feec882ee78c63cb8d4b485f1b52bbb873bb9a7b094435863200c7afba202382"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}

17
mmrazor/models/mutators/channel_mutator/channel_mutator.py
View File

@ -129,13 +129,16 @@ class ChannelMutator(BaseMutator, Generic[ChannelUnitType]):
"""Config template of the mutator.
Args:
only_mutable_units (bool, optional): If only return config of
prunable units. Defaults to False.
with_unit_init_args (bool, optional): If return init_args of
units. Defaults to False.
with_channels (bool, optional): if return channel info.
Defaults to False.
only_mutable_units (bool, optional): Whether only return config of
prunable units. It can omit unmutable MutableChannelUnits
to decrease the length of the config. Defaults to False.
with_unit_init_args (bool, optional): Whether return init_args of
units. Let it be true, when you want to change the init
args of units. Defaults to False.
with_channels (bool, optional): Whether return channel info.
The channel info can initialization the units without
tracer. When you want to prune your model without a
tracer next time, let it be true. Defaults to False.
Example:
dict(
channel_unit_cfg = dict(

6
mmrazor/utils/index_dict.py
View File

@ -6,9 +6,9 @@ VT = TypeVar('VT') # Value type
class IndexDict(OrderedDict):
"""IndexDict inherents from OrderedDict[Tuple[int, int], VT]. Each
IndexDict object is a OrderDict object which using index(Tuple[int,int]) as
key and Any as value.
"""IndexDict inherits from OrderedDict[Tuple[int, int], VT]. Each IndexDict
object is a OrderDict object which using index(Tuple[int,int]) as key and
Any as value.
The key type is Tuple[a: int,b: int]. It indicates a range in
the [a,b).