mirror of
https://github.com/open-mmlab/mmselfsup.git
synced 2025-06-03 14:59:38 +08:00
[Docs] Translate three docs (#651)
* translate into Chinese * Update structures.md * translate into Chinese * Update models.md * Update models.md * Update models.md * Update structures.md * Update structures.md * Update structures.md * Update structures.md * Update transforms.md * Update transforms.md * Update transforms.md * Update transforms.md * Update transforms.md * Update models.md * Update docs/en/advanced_guides/models.md Co-authored-by: Yixiao Fang <36138628+fangyixiao18@users.noreply.github.com> * Update docs/en/advanced_guides/models.md Co-authored-by: Yixiao Fang <36138628+fangyixiao18@users.noreply.github.com> * Update docs/en/advanced_guides/models.md Co-authored-by: Yixiao Fang <36138628+fangyixiao18@users.noreply.github.com> * Update docs/en/advanced_guides/transforms.md Co-authored-by: Yixiao Fang <36138628+fangyixiao18@users.noreply.github.com> * Update structures.md * update * update Co-authored-by: Yixiao Fang <36138628+fangyixiao18@users.noreply.github.com> Co-authored-by: fangyixiao18 <fangyx18@hotmail.com>
This commit is contained in:
parent
c152d1451b
commit
0ad8faf345
@ -10,13 +10,19 @@ contains the following fix parts,
|
||||
|
||||
- algorithms, containing the full modules of a model and all sub-modules will be
|
||||
constructed in algorithms.
|
||||
- backbones, containing the backbones for each algorithm, e.g. ViT for MAE, and Swim Transformer
|
||||
for SimMIM.
|
||||
|
||||
- backbones, containing the backbones for each algorithm, e.g. ViT for MAE, and Swim Transformer for SimMIM.
|
||||
|
||||
- necks, some specifial modules, such as decoder, appended directly to the output of the backbone.
|
||||
|
||||
- heads, some specifial modules, such as mlp layers, appended to the output of the backbone or neck.
|
||||
- memories, some memory banks or queues in some algorithms, e.g. MoCov1/v2.
|
||||
|
||||
- memories, some memory banks or queues in some algorithms, e.g. MoCo v1/v2.
|
||||
|
||||
- losses, used to compute the loss between the predicted output and the target.
|
||||
|
||||
- target_generators, generating targets for self-supervised learning optimization, such as HOG, extracted features from other modules(DALL-E, CLIP), etc.
|
||||
|
||||
## Overview of modules in MMSelfSup
|
||||
|
||||
First, we will give an overview about existing modules in MMSelfSup. They will be displayed according to the categories
|
||||
@ -54,6 +60,7 @@ class BaseModel(_BaseModel):
|
||||
backbone: dict,
|
||||
neck: Optional[dict] = None,
|
||||
head: Optional[dict] = None,
|
||||
target_generator: Optional[dict] = None,
|
||||
pretrained: Optional[str] = None,
|
||||
data_preprocessor: Optional[Union[dict, nn.Module]] = None,
|
||||
init_cfg: Optional[dict] = None):
|
||||
@ -87,7 +94,7 @@ following section.
|
||||
## Overview these abstract functions in base model
|
||||
|
||||
The `forward` function is the entrance to the results. However, it is different from the default `forward` function in most PyTorch code, which
|
||||
only has one mode. You will mess all your logic in the `forward` function, limiting the scability. Just as shown in the code below, `forward` function in MMSelfSup has three modes, i) tensor, ii) loss and iii) predict.
|
||||
only has one mode. You will mess all your logic in the `forward` function, limiting the scalability. Just as shown in the code below, `forward` function in MMSelfSup has three modes, i) tensor, ii) loss and iii) predict.
|
||||
|
||||
```python
|
||||
def forward(self,
|
||||
|
@ -6,7 +6,7 @@
|
||||
|
||||
The same as those in other OpenMMLab repositories, MMSelfSup defines a data structure, called `SelfSupDataSample`, which is used to receive and pass data during the whole training/testing process.
|
||||
`SelfSupDataSample` inherits the `BaseDataElement` implemented in [MMEngine](https://github.com/open-mmlab/mmengine).
|
||||
We recommend users to refer to [BaseDataElement](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/tutorials/data_element.md)
|
||||
We recommend users to refer to [BaseDataElement](https://github.com/open-mmlab/mmengine/blob/main/docs/en/advanced_tutorials/data_element.md)
|
||||
for more in-depth introduction about the basics of `BaseDataElement`. In this tutorials, we mainly discuss some customized
|
||||
features in [SelfSupDataSample](mmselfsup.structures.SelfSupDataSample).
|
||||
|
||||
|
@ -1 +1,114 @@
|
||||
# Models
|
||||
# 模型
|
||||
|
||||
- [模型](#模型)
|
||||
- [MMSelfSup 模型概述](#mmselfsup-模型概述)
|
||||
- [用子模块来构造算法](#用子模块来构造算法)
|
||||
- [基础模型中的抽象函数](#基础模型中的抽象函数)
|
||||
|
||||
我们可以把模型看作算法的特征提取器或者损失生成器。在 MMSelfSup 中,模型主要包括以下几个部分:
|
||||
|
||||
- 算法,包括模型的全部模块和构造算法时需要用到的子模块。
|
||||
|
||||
- 主干,里面是每个算法的支柱,比如 MAE 中的 VIT 和 SimMIM 中的 Swin Transformer。
|
||||
|
||||
- 颈部,指一些特殊的模块,比如解码器,它直接增加脊柱部分的输出结果。
|
||||
|
||||
- 头部,指一些特殊的模块,比如多层感知器的层,它增加脊柱部分或者颈部部分的输出结果。
|
||||
|
||||
- 记忆,也就是一些算法中的存储体或者队列,比如 MoCo v1/v2。
|
||||
|
||||
- 损失,用于算输出的预测值和目标之间的损失。
|
||||
|
||||
- 目标生成器,为自监督学习生成优化目标,例如 HOG,其它模块抽取的特征(DALL-E,CLIP)等.
|
||||
|
||||
## MMSelfSup 模型概述
|
||||
|
||||
首先,我们纵览 MMSelfSup 中已有的模型。我们根据上述的分类来展示这些模型。
|
||||
|
||||
| 算法 | 主干 | 颈部 | 头部 | 损失 | 记忆 |
|
||||
| :--------------------: | :-----------------------------: | :--------------------------: | :----------------------------------: | :--------------------------------: | :--------------------: |
|
||||
| [`BarlowTwins`](TODO) | [`ResNet`](TODO) | [`NonLinearNeck`](TODO) | [`LatentCrossCorrelationHead`](TODO) | [`CrossCorrelationLoss`](TODO) | N/A |
|
||||
| [`DenseCL`](TODO) | [`ResNet`](TODO) | [`DenseCLNeck`](TODO) | [`ContrastiveHead`](TODO) | [`CrossEntropyLoss`](TODO) | N/A |
|
||||
| [`BYOL`](TODO) | [`ResNet`](TODO) | [`NonLinearNeck`](TODO) | [`LatentPredictHead`](TODO) | [`CosineSimilarityLoss`](TODO) | N/A |
|
||||
| [`CAE`](TODO) | [`CAEViT`](TODO) | [`CAENeck`](TODO) | [`CAEHead`](TODO) | [`CAELoss`](TODO) | N/A |
|
||||
| [`DeepCluster`](TODO) | [`ResNet`](TODO) | [`AvgPool2dNeck`](TODO) | [`ClsHead`](TODO) | [`CrossEntropyLoss`](TODO) | N/A |
|
||||
| [`MAE`](TODO) | [`MAEViT`](TODO) | [`MAEPretrainDecoder`](TODO) | [`MAEPretrainHead`](TODO) | [`MAEReconstructionLoss`](TODO) | N/A |
|
||||
| [`MoCo`](TODO) | [`ResNet`](TODO) | [`LinearNeck`](TODO) | [`ContrastiveHead`](TODO) | [`CrossEntropyLoss`](TODO) | N/A |
|
||||
| [`MoCov3`](TODO) | [`MoCoV3ViT`](TODO) | [`NonLinearNeck`](TODO) | [`MoCoV3Head`](TODO) | [`CrossEntropyLoss`](TODO) | N/A |
|
||||
| [`NPID`](TODO) | [`ResNet`](TODO) | [`LinearNeck`](TODO) | [`ContrastiveHead`](TODO) | [`CrossEntropyLoss`](TODO) | [`SimpleMemory`](TODO) |
|
||||
| [`ODC`](TODO) | [`ResNet`](TODO) | [`ODCNeck`](TODO) | [`ClsHead`](TODO) | [`CrossEntropyLoss`](TODO) | [`ODCMemory`](TODO) |
|
||||
| [`RelativeLoc`](TODO) | [`ResNet`](TODO) | [`RelativeLocNeck`](TODO) | [`ClsHead`](TODO) | [`CrossEntropyLoss`](TODO) | N/A |
|
||||
| [`RotationPred`](TODO) | [`ResNet`](TODO) | N/A | [`ClsHead`](TODO) | [`CrossEntropyLoss`](TODO) | N/A |
|
||||
| [`SimCLR`](TODO) | [`ResNet`](TODO) | [`NonLinearNeck`](TODO) | [`ContrastiveHead`](TODO) | [`CrossEntropyLoss`](TODO) | N/A |
|
||||
| [`SimMIM`](TODO) | [`SimMIMSwinTransformer`](TODO) | [`SimMIMNeck`](TODO) | [`SimMIMHead`](TODO) | [`SimMIMReconstructionLoss`](TODO) | N/A |
|
||||
| [`SimSiam`](TODO) | [`ResNet`](TODO) | [`NonLinearNeck`](TODO) | [`LatentPredictHead`](TODO) | [`CosineSimilarityLoss`](TODO) | N/A |
|
||||
| [`SwAV`](TODO) | [`ResNet`](TODO) | [`SwAVNeck`](TODO) | [`SwAVHead`](TODO) | [`SwAVLoss`](TODO) | N/A |
|
||||
|
||||
## 用子模块来构造算法
|
||||
|
||||
正如上表所述,每个算法都是主干,颈部,头部,损失和记忆的结合体。您可以从这些模块中任意选出若干部分来构建你自己的算法。如果需要定制化的模块,您可参考 [add_modules](./add_modules.md) 中的内容。
|
||||
MMSelfSup 提供一个基础模型,名为 `BaseModel`,所以的算法都应该继承这个基础模型,而且所有子模块(除了记忆部分)在基础模型中进行初始化。记忆部分在对应算法的 `__init__` 中被构造。损失部分在头部部分初始化时被构造。
|
||||
|
||||
```python
|
||||
class BaseModel(_BaseModel):
|
||||
|
||||
def __init__(self,
|
||||
backbone: dict,
|
||||
neck: Optional[dict] = None,
|
||||
head: Optional[dict] = None,
|
||||
target_generator: Optional[dict] = None,
|
||||
pretrained: Optional[str] = None,
|
||||
data_preprocessor: Optional[Union[dict, nn.Module]] = None,
|
||||
init_cfg: Optional[dict] = None):
|
||||
|
||||
if pretrained is not None:
|
||||
init_cfg = dict(type='Pretrained', checkpoint=pretrained)
|
||||
|
||||
if data_preprocessor is None:
|
||||
data_preprocessor = {}
|
||||
# The build process is in MMEngine, so we need to add scope here.
|
||||
data_preprocessor.setdefault('type',
|
||||
'mmselfsup.SelfSupDataPreprocessor')
|
||||
|
||||
super().__init__(
|
||||
init_cfg=init_cfg, data_preprocessor=data_preprocessor)
|
||||
|
||||
self.backbone = MODELS.build(backbone)
|
||||
|
||||
if neck is not None:
|
||||
self.neck = MODELS.build(neck)
|
||||
|
||||
if head is not None:
|
||||
self.head = MODELS.build(head)
|
||||
|
||||
```
|
||||
|
||||
正如上面代码所示,构造主干部分时需要配置,但是对颈部和头部而言这可有可无。除了构造算法之外,您还需要重写基础模型中的一些抽象函数才能得到正确结果,我们将在下一部分讨论这件事。
|
||||
|
||||
## 基础模型中的抽象函数
|
||||
|
||||
`forward` 函数是结果的入口。然而,它和大多数 Pytorch 代码中只有一种模式的 `forward` 函数不同。MMSelfSup 把所有的逻辑都混杂在 `forward` 中,从而限制了该方法的可拓展性。正如下面代码所示,MMSelfSup 中的 `forward` 函数根据不同模式进行前向处理,目前共有三种模式:张量,损失和预测。
|
||||
|
||||
```python
|
||||
def forward(self,
|
||||
batch_inputs: torch.Tensor,
|
||||
data_samples: Optional[List[SelfSupDataSample]] = None,
|
||||
mode: str = 'tensor'):
|
||||
if mode == 'tensor':
|
||||
feats = self.extract_feat(batch_inputs)
|
||||
return feats
|
||||
elif mode == 'loss':
|
||||
return self.loss(batch_inputs, data_samples)
|
||||
elif mode == 'predict':
|
||||
return self.predict(batch_inputs, data_samples)
|
||||
else:
|
||||
raise RuntimeError(f'Invalid mode "{mode}".')
|
||||
```
|
||||
|
||||
- 张量,如果模式为 `tensor`,`forward` 函数就返回从图片提取到的特征。您应该重写其中的 `extract_feat`部分才能让定制化的提取过程有效。
|
||||
|
||||
- 损失,如果模式为 `loss`,`forward` 函数就返回预测值与目标之间的损失。同样的,您应该重写其中的 `loss` 部分才能让定制化的提取过程有效。
|
||||
|
||||
- 预测,如果模式为 `predict`,`forward` 函数就返回预测结果,比如用您的算法预测得到的标签。如果需要,`predict`函数也需要重写。
|
||||
|
||||
本文中我们学习了 MMSelfSup 中的模型的基本组成部分,如果您想深入研究,可以参考每个算法的API文件。
|
||||
|
@ -1 +1,168 @@
|
||||
# Structures
|
||||
# 数据结构
|
||||
|
||||
- [数据结构](#数据结构)
|
||||
- [SelfSupDataSample 中的定制化的属性](#selfsupdatasample-中的定制化的属性)
|
||||
- [用 MMSelfSup 把数据打包给 SelfSupDataSample](#用-mmselfsup-把数据打包给-selfsupdatasample)
|
||||
|
||||
像 OpenMMLab 中其他仓库一样,MMSelfSup 也定义了一个数据结构,名为 `SelfSupDataSample` ,这个数据结构用于接收和传递整个训练和测试过程中的数据。
|
||||
`SelfSupDataSample` 继承 [MMEngine](https://github.com/open-mmlab/mmengine) 中使用的 `BaseDataElement`。如果需要深入了解 `BaseDataElement`,我们建议参考 [BaseDataElement](https://github.com/open-mmlab/mmengine/blob/main/docs/zh_cn/advanced_tutorials/data_element.md)。在这些教程中,我们主要讨论 [SelfSupDataSample](mmselfsup.structures.SelfSupDataSample) 中一些定制化的属性。
|
||||
|
||||
## SelfSupDataSample 中的定制化的属性
|
||||
|
||||
在 MMSelfSup 中,`SelfSupDataSample` 将模型需要的所有信息(除了图片)打包,比如 mask image modeling(MIM) 中请求的 `mask` 和前置任务中的 `pseudo_label` 。除了提供信息,它还能接受模型产生的信息,比如预测得分。为实现上述功能, `SelfSupDataSample` 定义以下五个属性:
|
||||
|
||||
- gt_label(标签数据),包含图片的真实标签。
|
||||
|
||||
- sample_idx(实例数据),包含一开始被数据集初始化的数据列表中的最近的图片的序号。
|
||||
|
||||
- mask(数据基类),包含 MIM 中的面具,比如 SimMIM 和 CAE。
|
||||
|
||||
- pred_label(标签数据),包含模型预测的标签。
|
||||
|
||||
- pseudo_label(数据基类),包含前置任务中用到的假的标签,比如 Relation Location 中的 location。
|
||||
|
||||
为了帮助使用者理解 SelfSupDataSample 中的基本思想,我们给出一个关于如何创建 `SelfSupDataSample` 实例并设置这些属性的简单例子。
|
||||
|
||||
```python
|
||||
import torch
|
||||
from mmselfsup.core import SelfSupDataSample
|
||||
from mmengine.data import LabelData, InstanceData, BaseDataElement
|
||||
|
||||
selfsup_data_sample = SelfSupDataSample()
|
||||
# 在 selfsup_data_sample 里加入真实标签数据
|
||||
# 真实标签数据的类型应与 LabelData 的类型一致
|
||||
selfsup_data_sample.gt_label = LabelData(value=torch.tensor([1]))
|
||||
|
||||
# 如果真实标签数据类型和 LabelData 不一致会报错
|
||||
selfsup_data_sample.gt_label = torch.tensor([1])
|
||||
# 报错: AssertionError: tensor([1]) should be a <class 'mmengine.data.label_data.LabelData'> but got <class 'torch.Tensor'>
|
||||
|
||||
# 给 selfsup_data_sample 加入样例数据
|
||||
# 同样的,样例数据里的值的类型应与 InstanceData 保持一致
|
||||
selfsup_data_sample.sample_idx = InstanceData(value=torch.tensor([1]))
|
||||
|
||||
# 给 selfsup_data_sample 加面具
|
||||
selfsup_data_sample.mask = BaseDataElement(value=torch.ones((3, 3)))
|
||||
|
||||
# 给 selfsup_data_sample 加假标签
|
||||
selfsup_data_sample.pseudo_label = InstanceData(location=torch.tensor([1, 2, 3]))
|
||||
|
||||
|
||||
# 创建这些属性后,您可轻而易举得取这些属性里的值
|
||||
print(selfsup_data_sample.gt_label.value)
|
||||
# 输出 tensor([1])
|
||||
print(selfsup_data_sample.mask.value.shape)
|
||||
# 输出 torch.Size([3, 3])
|
||||
```
|
||||
|
||||
## 用 MMSelfSup 把数据打包给 SelfSupDataSample
|
||||
|
||||
在把数据喂给模型之前, MMSelfSup 按照数据流程把数据打包进 `SelfSupDataSample` 。如果您不熟悉数据流程,可以参考 [data transform](https://github.com/open-mmlab/mmcv/blob/transforms/docs/zh_cn/understand_mmcv/data_transform.md)。我们用一个叫 [PackSelfSupInputs](mmselfsup.datasets.transforms.PackSelfSupInputs)的数据变换来打包数据。
|
||||
|
||||
```python
|
||||
class PackSelfSupInputs(BaseTransform):
|
||||
"""把数据打包并让格式能与函数输入匹配
|
||||
|
||||
需要的值:
|
||||
|
||||
- img
|
||||
|
||||
添加的值:
|
||||
|
||||
- data_sample
|
||||
- inputs
|
||||
|
||||
参数:
|
||||
key (str): 输入模型的图片的值,默认为 img 。
|
||||
algorithm_keys (List[str]): 和算法相关的组成部分的值,比如 mask 。默认为 [] 。
|
||||
pseudo_label_keys (List[str]): 假标签对应的属性。默认为 [] 。
|
||||
meta_keys (List[str]): 图片的 meta 信息的值。默认为 [] 。
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self,
|
||||
key: Optional[str] = 'img',
|
||||
algorithm_keys: Optional[List[str]] = [],
|
||||
pseudo_label_keys: Optional[List[str]] = [],
|
||||
meta_keys: Optional[List[str]] = []) -> None:
|
||||
assert isinstance(key, str), f'key should be the type of str, instead \
|
||||
of {type(key)}.'
|
||||
|
||||
self.key = key
|
||||
self.algorithm_keys = algorithm_keys
|
||||
self.pseudo_label_keys = pseudo_label_keys
|
||||
self.meta_keys = meta_keys
|
||||
|
||||
def transform(self,
|
||||
results: Dict) -> Dict[torch.Tensor, SelfSupDataSample]:
|
||||
"""打包数据的方法。
|
||||
|
||||
参数:
|
||||
results (Dict): 数据变换返回的字典。
|
||||
|
||||
返回:
|
||||
Dict:
|
||||
|
||||
- 'inputs' (List[torch.Tensor]): 模型前面的数据。
|
||||
- 'data_sample' (SelfSupDataSample): 前面数据的注释信息。
|
||||
"""
|
||||
packed_results = dict()
|
||||
if self.key in results:
|
||||
img = results[self.key]
|
||||
# if img is not a list, convert it to a list
|
||||
if not isinstance(img, List):
|
||||
img = [img]
|
||||
for i, img_ in enumerate(img):
|
||||
if len(img_.shape) < 3:
|
||||
img_ = np.expand_dims(img_, -1)
|
||||
img_ = np.ascontiguousarray(img_.transpose(2, 0, 1))
|
||||
img[i] = to_tensor(img_)
|
||||
packed_results['inputs'] = img
|
||||
|
||||
data_sample = SelfSupDataSample()
|
||||
if len(self.pseudo_label_keys) > 0:
|
||||
pseudo_label = InstanceData()
|
||||
data_sample.pseudo_label = pseudo_label
|
||||
|
||||
# gt_label, sample_idx, mask, pred_label 在此设置
|
||||
for key in self.algorithm_keys:
|
||||
self.set_algorithm_keys(data_sample, key, results)
|
||||
|
||||
# 除 gt_label, sample_idx, mask, pred_label 外的值会被设为假标签的属性
|
||||
for key in self.pseudo_label_keys:
|
||||
# convert data to torch.Tensor
|
||||
value = to_tensor(results[key])
|
||||
setattr(data_sample.pseudo_label, key, value)
|
||||
|
||||
img_meta = {}
|
||||
for key in self.meta_keys:
|
||||
img_meta[key] = results[key]
|
||||
data_sample.set_metainfo(img_meta)
|
||||
packed_results['data_sample'] = data_sample
|
||||
|
||||
return packed_results
|
||||
|
||||
@classmethod
|
||||
def set_algorithm_keys(self, data_sample: SelfSupDataSample, key: str,
|
||||
results: Dict) -> None:
|
||||
"""设置 SelfSupDataSample 中算法的值."""
|
||||
value = to_tensor(results[key])
|
||||
if key == 'sample_idx':
|
||||
sample_idx = InstanceData(value=value)
|
||||
setattr(data_sample, 'sample_idx', sample_idx)
|
||||
elif key == 'mask':
|
||||
mask = InstanceData(value=value)
|
||||
setattr(data_sample, 'mask', mask)
|
||||
elif key == 'gt_label':
|
||||
gt_label = LabelData(value=value)
|
||||
setattr(data_sample, 'gt_label', gt_label)
|
||||
elif key == 'pred_label':
|
||||
pred_label = LabelData(value=value)
|
||||
setattr(data_sample, 'pred_label', pred_label)
|
||||
else:
|
||||
raise AttributeError(f'{key} is not a attribute of \
|
||||
SelfSupDataSample')
|
||||
```
|
||||
|
||||
在 SelfSupDataSample 中 `algorithm_keys` 是除了 `pseudo_label` 的数据属性, `pseudo_label_keys` 是 SelfSupDataSample 中假标签对应的分支属性。
|
||||
感谢读完整个教程。有问题的话可以在 GitHub 上提 issue,我们会尽快联系您。
|
||||
|
@ -1 +1,80 @@
|
||||
# Transforms
|
||||
# 数据变化
|
||||
|
||||
- [数据变化](#数据变化)
|
||||
- [数据变换概述](#数据变换概述)
|
||||
- [MultiView 简介](#multiview-简介)
|
||||
- [PackSelfSupInputs 简介](#packselfsupinputs-简介)
|
||||
|
||||
## 数据变换概述
|
||||
|
||||
在 [add_transforms](./add_transforms.md) 中我们介绍了如何构建 `Pipeline` 。 `Pipeline` 里有一系列的数据变换。MMSelfSup 中数据变换主要分为三类:
|
||||
|
||||
1. 处理数据用到的数据变换。[processing.py](https://github.com/open-mmlab/mmselfsup/blob/1.x/mmselfsup/datasets/transforms/processing.py) 中定义了独特的数据变换,比如`RandomCrop`, `RandomResizedCrop` 和 `RandomGaussianBlur`。我们也可以用其它仓库的数据变换,比如 MMCV 中的 `LoadImageFromFile`。
|
||||
|
||||
2. 不同视角看同一照片的数据变换打包器。这个定义在 [wrappers.py](https://github.com/open-mmlab/mmselfsup/blob/1.x/mmselfsup/datasets/transforms/wrappers.py)。
|
||||
|
||||
3. 将数据变换使得数据能输入算法中。这个定义在 [formatting.py](https://github.com/open-mmlab/mmselfsup/blob/1.x/mmselfsup/datasets/transforms/formatting.py)。
|
||||
|
||||
总的来说,我们用的是如下的这些数据变换。我们将详细讨论最后两种数据变换。
|
||||
|
||||
| 类别 | 作用 |
|
||||
| :-------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------: |
|
||||
| [`BEiTMaskGenerator`](mmselfsup.datasets.BEiTMaskGenerator) | 为图像产生随机掩码,参考自 `BEiT` |
|
||||
| [`SimMIMMaskGenerator`](mmselfsup.datasets.SimMIMMaskGenerator) | 产生随机块状掩码,参考自 `SimMIM` |
|
||||
| [`ColorJitter`](mmselfsup.datasets.ColorJitter) | 随机改变图像亮度,对比度,饱和度和色调 |
|
||||
| [`RandomCrop`](mmselfsup.datasets.RandomCrop) | 随机裁切图像 |
|
||||
| [`RandomGaussianBlur`](mmselfsup.datasets.RandomGaussianBlur) | 随机高斯模糊,参考自, `SimCLR` |
|
||||
| [`RandomResizedCrop`](mmselfsup.datasets.RandomResizedCrop) | 随机裁切图像,并调整大小到特定比例 |
|
||||
| [`RandomResizedCropAndInterpolationWithTwoPic`](mmselfsup.datasets.RandomResizedCropAndInterpolationWithTwoPic) | 随机裁切图像,并调整大小到特定比例,可以给定不同的插值方法 |
|
||||
| [`RandomSolarize`](mmselfsup.datasets.RandomSolarize) | 随机曝光调整,参考自 `BYOL` |
|
||||
| [`RotationWithLabels`](mmselfsup.datasets.RotationWithLabels) | 旋转预测 |
|
||||
| [`RandomPatchWithLabels`](mmselfsup.datasets.RandomPatchWithLabels) | 随机分块 |
|
||||
| [`RandomRotation`](mmselfsup.datasets.RandomRotation) | 随机旋转图像 |
|
||||
| [`MultiView`](mmselfsup.datasets.transforms.MultiView) | 多角度图像的封装器 |
|
||||
| [`PackSelfSupInputs`](mmselfsup.datasets.PackSelfSupInputs) | 打包数据为可以送入算法的格式 |
|
||||
|
||||
## MultiView 简介
|
||||
|
||||
我们为一些算法定义了名为 [`MultiView`](mmselfsup.datasets.transforms.MultiView) 的多角度照片输入的封装器,比如 MoCo 系列,SimCLR,SwAV 等。在配置文件中,我们能这样定义:
|
||||
|
||||
```python
|
||||
pipeline = [
|
||||
dict(type='MultiView',
|
||||
num_views=2,
|
||||
transforms=[
|
||||
[dict(type='Resize', scale=224),]
|
||||
])
|
||||
]
|
||||
```
|
||||
|
||||
这意味着数据管道里面有两个角度。
|
||||
|
||||
我们也可以这样定义有不同角度的数据管道:
|
||||
|
||||
```python
|
||||
pipeline = [
|
||||
dict(type='MultiView',
|
||||
num_views=[2, 6],
|
||||
transforms=[
|
||||
[
|
||||
dict(type='Resize', scale=224)],
|
||||
[
|
||||
dict(type='Resize', scale=224),
|
||||
dict(type='RandomSolarize')],
|
||||
])
|
||||
]
|
||||
```
|
||||
|
||||
这意味着有两个数据管道,他们分别有两个角度和六个角度。在 [imagenet_mocov1.py](https://github.com/open-mmlab/mmselfsup/blob/1.x/configs/selfsup/_base_/datasets/imagenet_mocov1.py) 和 [imagenet_mocov2.py](https://github.com/open-mmlab/mmselfsup/blob/1.x/configs/selfsup/_base_/datasets/imagenet_mocov2.py) 和 [imagenet_swav_mcrop-2-6.py](https://github.com/open-mmlab/mmselfsup/blob/1.x/configs/selfsup/_base_/datasets/imagenet_swav_mcrop-2-6.py) 中有更多例子。
|
||||
|
||||
## PackSelfSupInputs 简介
|
||||
|
||||
我们定义了一个名为 [`PackSelfSupInputs`](mmselfsup.datasets.transforms.PackSelfSupInputs) 的类来将数据转换为能输入算法中的格式。这种转换通常在数据管道的最后,就像下面这样:
|
||||
|
||||
```python
|
||||
train_pipeline = [
|
||||
dict(type='LoadImageFromFile'),
|
||||
dict(type='MultiView', num_views=2, transforms=[view_pipeline]),
|
||||
dict(type='PackSelfSupInputs', meta_keys=['img_path'])
|
||||
]
|
||||
```
|
||||
|
Loading…
x
Reference in New Issue
Block a user