[Enhance] Substitute the environment variable in config file (#744)
* Add read environment variable function in config * Add UT * enable int case, split predefined and environment * Update py config * Add new attributes env_variables in config * Add examples in docstring * Fix comments * Add tutorials * Add en tutorials * Refactor config docs according to comments * Fix comments * Change function namepull/897/head
jbwang1997
GitHub
parent
acf21607be
commit
c46f891a97
docs
en/advanced_tutorials
resources/config
zh_cn/advanced_tutorials
mmengine/config
tests
data/config/py_config
test_config
|
|
@ -14,6 +14,8 @@ wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/c
|
|||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/my_module.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/optimizer_cfg.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/predefined_var.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/replace_data_root.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/replace_num_classes.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/refer_base_var.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_delete_key.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_lr0.01.py
|
||||
|
|
@ -490,6 +492,98 @@ Config (path: ./example.py): {'model': {'type': 'CustomModel', 'in_channels': [1
|
|||
|
||||
:::
|
||||
|
||||
### Replace fields with environment variables
|
||||
|
||||
When a field is deeply nested, we need to add a long prefix at the command line to locate it. To alleviate this problem, MMEngine allows users to substitute fields in configuration with environment variables.
|
||||
|
||||
Before parsing the configuration file, the program will search all `{{$ENV_VAR:DEF_VAL}}` fields and substitute those sections with environment variables. Here, `ENV_VAR` is the name of the environment variable used to replace this section, `DEF_VAL` is the default value if `ENV_VAR` is not set.
|
||||
|
||||
When we want to modify the dataset path at the command line, we can take `replace_data_root.py` as an example:
|
||||
|
||||
```python
|
||||
dataset_type = 'CocoDataset'
|
||||
data_root = '{{$DATASET:/data/coco/}}'
|
||||
dataset=dict(ann_file= data_root + 'train.json')
|
||||
```
|
||||
|
||||
If we run `demo_train.py` to parse this configuration file.
|
||||
|
||||
```bash
|
||||
python demo_train.py replace_data_root.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/data/coco/', 'dataset': {'ann_file': '/data/coco/train.json'}}
|
||||
```
|
||||
|
||||
Here, we don't set the environment variable `DATASET`. Thus, the program directly replaces `{{$DATASET:/data/coco/}}` with the default value `/data/coco/`. If we set `DATASET` at the command line:
|
||||
|
||||
```bash
|
||||
DATASET=/new/dataset/path/ python demo_train.py replace_data_root.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/new/dataset/path/train.json'}}
|
||||
```
|
||||
|
||||
The value of `data_root` has been substituted with the value of `DATASET` as `/new/dataset/path`.
|
||||
|
||||
It is noteworthy that both `--cfg-options` and `{{$ENV_VAR:DEF_VAL}}` allow users to modify fields in command line. But there is a small difference between those two methods. Environment variable substitution occurs before the configuration parsing. If the replaced field is also involved in other fields assignment, the environment variable substitution will also affect the other fields.
|
||||
|
||||
We take `demo_train.py` and `replace_data_root.py` for example. If we replace `data_root` by setting `--cfg-options data_root='/new/dataset/path'`:
|
||||
|
||||
```bash
|
||||
python demo_train.py replace_data_root.py --cfg-options data_root='/new/dataset/path/'
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/data/coco/train.json'}}
|
||||
```
|
||||
|
||||
As we can see, only `data_root` has been modified. `dataset.ann_file` is still the default value.
|
||||
|
||||
In contrast, if we replace `data_root` by setting `DATASET=/new/dataset/path`:
|
||||
|
||||
```bash
|
||||
DATASET=/new/dataset/path/ python demo_train.py replace_data_root.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/new/dataset/path/train.json'}}
|
||||
```
|
||||
|
||||
Both `data_root` and `dataset.ann_file` have been modified.
|
||||
|
||||
Environment variables can also be used to replace other types of fields. We can use `{{'$ENV_VAR:DEF_VAL'}}` or `{{"$ENV_VAR:DEF_VAL"}}` format to ensure the configuration file conforms to python syntax.
|
||||
|
||||
We can take `replace_num_classes.py` as an example:
|
||||
|
||||
```
|
||||
model=dict(
|
||||
bbox_head=dict(
|
||||
num_classes={{'$NUM_CLASSES:80'}}))
|
||||
```
|
||||
|
||||
If we run `demo_train.py` to parse this configuration file.
|
||||
|
||||
```bash
|
||||
python demo_train.py replace_num_classes.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_num_classes.py): {'model': {'bbox_head': {'num_classes': 80}}}
|
||||
```
|
||||
|
||||
Let us set the environment variable `NUM_CLASSES`
|
||||
|
||||
```bash
|
||||
NUM_CLASSES=20 python demo_train.py replace_num_classes.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_num_classes.py): {'model': {'bbox_head': {'num_classes': 20}}}
|
||||
```
|
||||
|
||||
### import the custom module
|
||||
|
||||
If we customize a module and register it into the corresponding registry, could we directly build it from the configuration file as the previous [section](#how-to-use-config) does? The answer is "I don't know" since I'm not sure the registration process has been triggered. To solve this "unknown" case, `Config` provides the `custom_imports` function, to make sure your module could be registered as expected.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,3 @@
|
|||
dataset_type = 'CocoDataset'
|
||||
data_root = '{{$DATASET:/data/coco/}}'
|
||||
dataset = dict(ann_file=data_root + 'train.json')
|
||||
|
|
@ -0,0 +1 @@
|
|||
model = dict(bbox_head=dict(num_classes={{'$NUM_CLASSES:80'}}))
|
||||
|
|
@ -15,6 +15,8 @@ wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/c
|
|||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/optimizer_cfg.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/predefined_var.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/refer_base_var.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/replace_data_root.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/replace_num_classes.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_delete_key.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_lr0.01.py
|
||||
wget https://raw.githubusercontent.com/open-mmlab/mmengine/main/docs/resources/config/resnet50_runtime.py
|
||||
|
|
@ -490,6 +492,98 @@ Config (path: ./example.py): {'model': {'type': 'CustomModel', 'in_channels': [1
|
|||
|
||||
:::
|
||||
|
||||
### 使用环境变量替换配置
|
||||
|
||||
当要修改的配置嵌套很深时，我们在命令行中需要加上很长的前缀来进行定位。为了更方便地在命令行中修改配置，MMEngine 提供了一套通过环境变量来替换配置的方法。
|
||||
|
||||
在解析配置文件之前，MMEngine 会搜索所有的 `{{$ENV_VAR:DEF_VAL}}` 字段，并使用特定的环境变量来替换这一部分。这里 `ENV_VAR` 为替换这一部分所用的环境变量，`DEF_VAL` 为没有设置环境变量时的默认值。
|
||||
|
||||
例如，当我们想在命令行中修改数据集路径时，我们可以在配置文件 `replace_data_root.py` 中这样写：
|
||||
|
||||
```python
|
||||
dataset_type = 'CocoDataset'
|
||||
data_root = '{{$DATASET:/data/coco/}}'
|
||||
dataset=dict(ann_file= data_root + 'train.json')
|
||||
```
|
||||
|
||||
当我们运行 `demo_train.py` 来读取这个配置文件时：
|
||||
|
||||
```bash
|
||||
python demo_train.py replace_data_root.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/data/coco/', 'dataset': {'ann_file': '/data/coco/train.json'}}
|
||||
```
|
||||
|
||||
这里没有设置环境变量 `DATASET`, 程序直接使用默认值 `/data/coco/` 来替换 `{{$DATASET:/data/coco/}}`。如果在命令行前设置设置环境变量则会有如下结果：
|
||||
|
||||
```bash
|
||||
DATASET=/new/dataset/path/ python demo_train.py replace_data_root.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/new/dataset/path/train.json'}}
|
||||
```
|
||||
|
||||
`data_root` 被替换成了环境变量 `DATASET` 的值 `/new/dataset/path/`。
|
||||
|
||||
值得注意的是，`--cfg-options` 与 `{{$ENV_VAR:DEF_VAL}}` 都可以在命令行改变配置文件的值，但他们还有一些区别。环境变量的替换发生在配置文件解析之前。如果该配置还参与到其他配置的定义时，环境变量替换也会影响到其他配置，而 `--cfg-options` 只会改变要修改的配置文件的值。
|
||||
|
||||
我们以 `demo_train.py` 与 `replace_data_root.py` 为例。 如果我们通过配置 `--cfg-options data_root='/new/dataset/path'` 来修改 `data_root`：
|
||||
|
||||
```bash
|
||||
python demo_train.py replace_data_root.py --cfg-options data_root='/new/dataset/path/'
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/data/coco/train.json'}}
|
||||
```
|
||||
|
||||
从输出结果上看，只有 `data_root` 被修改为新的值。`dataset.ann_file` 依然保持原始值。
|
||||
|
||||
作为对比，如果我们通过配置 `DATASET=/new/dataset/path` 来修改 `data_root`:
|
||||
|
||||
```bash
|
||||
DATASET=/new/dataset/path/ python demo_train.py replace_data_root.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_data_root.py): {'dataset_type': 'CocoDataset', 'data_root': '/new/dataset/path/', 'dataset': {'ann_file': '/new/dataset/path/train.json'}}
|
||||
```
|
||||
|
||||
`data_root` 与 `dataset.ann_file` 同时被修改了。
|
||||
|
||||
环境变量也可以用来替换字符串以外的配置，这时可以使用 `{{'$ENV_VAR:DEF_VAL'}}` 或者 `{{"$ENV_VAR:DEF_VAL"}}` 格式。`''` 与 `""` 用来保证配置文件合乎 python 语法。
|
||||
|
||||
例如，当我们想替换模型预测的类别数时，可以在配置文件 `replace_num_classes.py` 中这样写：
|
||||
|
||||
```
|
||||
model=dict(
|
||||
bbox_head=dict(
|
||||
num_classes={{'$NUM_CLASSES:80'}}))
|
||||
```
|
||||
|
||||
当我们运行 `demo_train.py` 来读取这个配置文件时：
|
||||
|
||||
```bash
|
||||
python demo_train.py replace_num_classes.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_num_classes.py): {'model': {'bbox_head': {'num_classes': 80}}}
|
||||
```
|
||||
|
||||
当设置 `NUM_CLASSES` 环境变量后：
|
||||
|
||||
```bash
|
||||
NUM_CLASSES=20 python demo_train.py replace_num_classes.py
|
||||
```
|
||||
|
||||
```
|
||||
Config (path: replace_num_classes.py): {'model': {'bbox_head': {'num_classes': 20}}}
|
||||
```
|
||||
|
||||
### 导入自定义 Python 模块
|
||||
|
||||
将配置与注册器结合起来使用时，如果我们往注册器中注册了一些自定义的类，就可能会遇到一些问题。因为读取配置文件的时候，这部分代码可能还没有被执行到，所以并未完成注册过程，从而导致构建自定义类的时候报错。
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ from .utils import (RemoveAssignFromAST, _get_external_cfg_base_path,
|
|||
BASE_KEY = '_base_'
|
||||
DELETE_KEY = '_delete_'
|
||||
DEPRECATION_KEY = '_deprecation_'
|
||||
RESERVED_KEYS = ['filename', 'text', 'pretty_text']
|
||||
RESERVED_KEYS = ['filename', 'text', 'pretty_text', 'env_variables']
|
||||
|
||||
if platform.system() == 'Windows':
|
||||
import regex as re
|
||||
|
|
@ -130,7 +130,8 @@ class Config:
|
|||
def __init__(self,
|
||||
cfg_dict: dict = None,
|
||||
cfg_text: Optional[str] = None,
|
||||
filename: Optional[Union[str, Path]] = None):
|
||||
filename: Optional[Union[str, Path]] = None,
|
||||
env_variables: Optional[dict] = None):
|
||||
filename = str(filename) if isinstance(filename, Path) else filename
|
||||
if cfg_dict is None:
|
||||
cfg_dict = dict()
|
||||
|
|
@ -151,11 +152,15 @@ class Config:
|
|||
else:
|
||||
text = ''
|
||||
super().__setattr__('_text', text)
|
||||
if env_variables is None:
|
||||
env_variables = dict()
|
||||
super().__setattr__('_env_variables', env_variables)
|
||||
|
||||
@staticmethod
|
||||
def fromfile(filename: Union[str, Path],
|
||||
use_predefined_variables: bool = True,
|
||||
import_custom_modules: bool = True) -> 'Config':
|
||||
import_custom_modules: bool = True,
|
||||
use_environment_variables: bool = True) -> 'Config':
|
||||
"""Build a Config instance from config file.
|
||||
|
||||
Args:
|
||||
|
|
@ -169,14 +174,18 @@ class Config:
|
|||
Config: Config instance built from config file.
|
||||
"""
|
||||
filename = str(filename) if isinstance(filename, Path) else filename
|
||||
cfg_dict, cfg_text = Config._file2dict(filename,
|
||||
use_predefined_variables)
|
||||
cfg_dict, cfg_text, env_variables = Config._file2dict(
|
||||
filename, use_predefined_variables, use_environment_variables)
|
||||
if import_custom_modules and cfg_dict.get('custom_imports', None):
|
||||
try:
|
||||
import_modules_from_strings(**cfg_dict['custom_imports'])
|
||||
except ImportError as e:
|
||||
raise ImportError('Failed to custom import!') from e
|
||||
return Config(cfg_dict, cfg_text=cfg_text, filename=filename)
|
||||
return Config(
|
||||
cfg_dict,
|
||||
cfg_text=cfg_text,
|
||||
filename=filename,
|
||||
env_variables=env_variables)
|
||||
|
||||
@staticmethod
|
||||
def fromstring(cfg_str: str, file_format: str) -> 'Config':
|
||||
|
|
@ -288,6 +297,67 @@ class Config:
|
|||
with open(temp_config_name, 'w', encoding='utf-8') as tmp_config_file:
|
||||
tmp_config_file.write(config_file)
|
||||
|
||||
@staticmethod
|
||||
def _substitute_env_variables(filename: str, temp_config_name: str):
|
||||
"""Substitute environment variables in config with actual values.
|
||||
|
||||
Sometimes, we want to change some items in the config with environment
|
||||
variables. For examples, we expect to change dataset root by setting
|
||||
``DATASET_ROOT=/dataset/root/path`` in the command line. This can be
|
||||
easily achieved by writing lines in the config as follows
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
data_root = '{{$DATASET_ROOT:/default/dataset}}/images'
|
||||
|
||||
|
||||
Here, ``{{$DATASET_ROOT:/default/dataset}}`` indicates using the
|
||||
environment variable ``DATASET_ROOT`` to replace the part between
|
||||
``{{}}``. If the ``DATASET_ROOT`` is not set, the default value
|
||||
``/default/dataset`` will be used.
|
||||
|
||||
Environment variables not only can replace items in the string, they
|
||||
can also substitute other types of data in config. In this situation,
|
||||
we can write the config as below
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
model = dict(
|
||||
bbox_head = dict(num_classes={{'$NUM_CLASSES:80'}}))
|
||||
|
||||
|
||||
For details, Please refer to docs/zh_cn/tutorials/config.md .
|
||||
|
||||
Args:
|
||||
filename (str): Filename of config.
|
||||
temp_config_name (str): Temporary filename to save substituted
|
||||
config.
|
||||
"""
|
||||
with open(filename, encoding='utf-8') as f:
|
||||
config_file = f.read()
|
||||
regexp = r'\{\{[\'\"]?\s*\$(\w+)\s*\:\s*(\S*?)\s*[\'\"]?\}\}'
|
||||
keys = re.findall(regexp, config_file)
|
||||
env_variables = dict()
|
||||
for var_name, value in keys:
|
||||
regexp = r'\{\{[\'\"]?\s*\$' + var_name + r'\s*\:\s*' \
|
||||
+ value + r'\s*[\'\"]?\}\}'
|
||||
if var_name in os.environ:
|
||||
value = os.environ[var_name]
|
||||
env_variables[var_name] = value
|
||||
print_log(
|
||||
f'Using env variable `{var_name}` with value of '
|
||||
f'{value} to replace item in config.',
|
||||
logger='current')
|
||||
if not value:
|
||||
raise KeyError(f'`{var_name}` cannot be found in `os.environ`.'
|
||||
f' Please set `{var_name}` in environment or '
|
||||
'give a default value.')
|
||||
config_file = re.sub(regexp, value, config_file)
|
||||
|
||||
with open(temp_config_name, 'w', encoding='utf-8') as tmp_config_file:
|
||||
tmp_config_file.write(config_file)
|
||||
return env_variables
|
||||
|
||||
@staticmethod
|
||||
def _pre_substitute_base_vars(filename: str,
|
||||
temp_config_name: str) -> dict:
|
||||
|
|
@ -361,8 +431,10 @@ class Config:
|
|||
return cfg
|
||||
|
||||
@staticmethod
|
||||
def _file2dict(filename: str,
|
||||
use_predefined_variables: bool = True) -> Tuple[dict, str]:
|
||||
def _file2dict(
|
||||
filename: str,
|
||||
use_predefined_variables: bool = True,
|
||||
use_environment_variables: bool = True) -> Tuple[dict, str, dict]:
|
||||
"""Transform file to variables dictionary.
|
||||
|
||||
Args:
|
||||
|
|
@ -391,6 +463,11 @@ class Config:
|
|||
temp_config_file.name)
|
||||
else:
|
||||
shutil.copyfile(filename, temp_config_file.name)
|
||||
# Substitute environment variables
|
||||
env_variables = dict()
|
||||
if use_environment_variables:
|
||||
env_variables = Config._substitute_env_variables(
|
||||
temp_config_file.name, temp_config_file.name)
|
||||
# Substitute base variables from placeholders to strings
|
||||
base_var_dict = Config._pre_substitute_base_vars(
|
||||
temp_config_file.name, temp_config_file.name)
|
||||
|
|
@ -401,8 +478,12 @@ class Config:
|
|||
for base_cfg_path in Config._get_base_files(temp_config_file.name):
|
||||
base_cfg_path, scope = Config._get_cfg_path(
|
||||
base_cfg_path, filename)
|
||||
_cfg_dict, _cfg_text = Config._file2dict(base_cfg_path)
|
||||
_cfg_dict, _cfg_text, _env_variables = Config._file2dict(
|
||||
filename=base_cfg_path,
|
||||
use_predefined_variables=use_predefined_variables,
|
||||
use_environment_variables=use_environment_variables)
|
||||
cfg_text_list.append(_cfg_text)
|
||||
env_variables.update(_env_variables)
|
||||
duplicate_keys = base_cfg_dict.keys() & _cfg_dict.keys()
|
||||
if len(duplicate_keys) > 0:
|
||||
raise KeyError('Duplicate key is not allowed among bases. '
|
||||
|
|
@ -481,7 +562,7 @@ class Config:
|
|||
cfg_text_list.append(cfg_text)
|
||||
cfg_text = '\n'.join(cfg_text_list)
|
||||
|
||||
return cfg_dict, cfg_text
|
||||
return cfg_dict, cfg_text, env_variables
|
||||
|
||||
@staticmethod
|
||||
def _dict_to_config_dict(cfg: dict,
|
||||
|
|
@ -703,6 +784,11 @@ class Config:
|
|||
"""get config text."""
|
||||
return self._text
|
||||
|
||||
@property
|
||||
def env_variables(self) -> dict:
|
||||
"""get used environment variables."""
|
||||
return self._env_variables
|
||||
|
||||
@property
|
||||
def pretty_text(self) -> str:
|
||||
"""get formatted python config text."""
|
||||
|
|
@ -823,8 +909,9 @@ class Config:
|
|||
def __iter__(self):
|
||||
return iter(self._cfg_dict)
|
||||
|
||||
def __getstate__(self) -> Tuple[dict, Optional[str], Optional[str]]:
|
||||
return (self._cfg_dict, self._filename, self._text)
|
||||
def __getstate__(self) -> Tuple[dict, Optional[str], Optional[str], dict]:
|
||||
return (self._cfg_dict, self._filename, self._text,
|
||||
self._env_variables)
|
||||
|
||||
def __deepcopy__(self, memo):
|
||||
cls = self.__class__
|
||||
|
|
@ -843,11 +930,13 @@ class Config:
|
|||
|
||||
return other
|
||||
|
||||
def __setstate__(self, state: Tuple[dict, Optional[str], Optional[str]]):
|
||||
_cfg_dict, _filename, _text = state
|
||||
def __setstate__(self, state: Tuple[dict, Optional[str], Optional[str],
|
||||
dict]):
|
||||
_cfg_dict, _filename, _text, _env_variables = state
|
||||
super().__setattr__('_cfg_dict', _cfg_dict)
|
||||
super().__setattr__('_filename', _filename)
|
||||
super().__setattr__('_text', _text)
|
||||
super().__setattr__('_text', _env_variables)
|
||||
|
||||
def dump(self, file: Optional[Union[str, Path]] = None):
|
||||
"""Dump config to file or return config text.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,4 @@
|
|||
# Copyright (c) OpenMMLab. All rights reserved.
|
||||
item1 = '{{ $ITEM1: }}'
|
||||
item2 = '{{ $ITEM2:default_value }}'
|
||||
item3 = {{' $ITEM3:80 '}}
|
||||
|
|
@ -376,6 +376,42 @@ class TestConfig:
|
|||
with open(substituted_cfg) as f:
|
||||
assert f.read() == expected_text
|
||||
|
||||
def test_substitute_environment_vars(self, tmp_path):
|
||||
cfg = tmp_path / 'tmp_cfg1.py'
|
||||
substituted_cfg = tmp_path / 'tmp_cfg2.py'
|
||||
|
||||
cfg_text = 'a={{$A:}}\n'
|
||||
with open(cfg, 'w') as f:
|
||||
f.write(cfg_text)
|
||||
with pytest.raises(KeyError):
|
||||
Config._substitute_env_variables(cfg, substituted_cfg)
|
||||
|
||||
os.environ['A'] = 'text_A'
|
||||
Config._substitute_env_variables(cfg, substituted_cfg)
|
||||
with open(substituted_cfg) as f:
|
||||
assert f.read() == 'a=text_A\n'
|
||||
os.environ.pop('A')
|
||||
|
||||
cfg_text = 'b={{$B:80}}\n'
|
||||
with open(cfg, 'w') as f:
|
||||
f.write(cfg_text)
|
||||
Config._substitute_env_variables(cfg, substituted_cfg)
|
||||
with open(substituted_cfg) as f:
|
||||
assert f.read() == 'b=80\n'
|
||||
|
||||
os.environ['B'] = '100'
|
||||
Config._substitute_env_variables(cfg, substituted_cfg)
|
||||
with open(substituted_cfg) as f:
|
||||
assert f.read() == 'b=100\n'
|
||||
os.environ.pop('B')
|
||||
|
||||
cfg_text = 'c={{"$C:80"}}\n'
|
||||
with open(cfg, 'w') as f:
|
||||
f.write(cfg_text)
|
||||
Config._substitute_env_variables(cfg, substituted_cfg)
|
||||
with open(substituted_cfg) as f:
|
||||
assert f.read() == 'c=80\n'
|
||||
|
||||
def test_pre_substitute_base_vars(self, tmp_path):
|
||||
cfg_path = osp.join(self.data_path, 'config',
|
||||
'py_config/test_pre_substitute_base_vars.py')
|
||||
|
|
@ -436,6 +472,7 @@ class TestConfig:
|
|||
|
||||
self._simple_load()
|
||||
self._predefined_vars()
|
||||
self._environment_vars()
|
||||
self._base_variables()
|
||||
self._merge_from_base()
|
||||
self._code_in_config()
|
||||
|
|
@ -477,9 +514,10 @@ class TestConfig:
|
|||
filename = f'{file_format}_config/{name}.{file_format}'
|
||||
|
||||
cfg_file = osp.join(self.data_path, 'config', filename)
|
||||
cfg_dict, cfg_text = Config._file2dict(cfg_file)
|
||||
cfg_dict, cfg_text, env_variables = Config._file2dict(cfg_file)
|
||||
assert isinstance(cfg_text, str)
|
||||
assert isinstance(cfg_dict, dict)
|
||||
assert isinstance(env_variables, dict)
|
||||
|
||||
def _get_file_path(self, file_path):
|
||||
if platform.system() == 'Windows':
|
||||
|
|
@ -534,6 +572,31 @@ class TestConfig:
|
|||
assert Config.fromfile(cfg_file)['item1'] == self._get_file_path(
|
||||
osp.dirname(cfg_file))
|
||||
|
||||
def _environment_vars(self):
|
||||
# test parse predefined_var in config
|
||||
cfg_file = osp.join(self.data_path,
|
||||
'config/py_config/test_environment_var.py')
|
||||
|
||||
with pytest.raises(KeyError):
|
||||
Config._file2dict(cfg_file)
|
||||
|
||||
os.environ['ITEM1'] = '60'
|
||||
cfg_dict_dst = dict(item1='60', item2='default_value', item3=80)
|
||||
assert Config._file2dict(cfg_file)[0]['item1'] == cfg_dict_dst['item1']
|
||||
assert Config._file2dict(cfg_file)[0]['item2'] == cfg_dict_dst['item2']
|
||||
assert Config._file2dict(cfg_file)[0]['item3'] == cfg_dict_dst['item3']
|
||||
|
||||
os.environ['ITEM2'] = 'new_value'
|
||||
os.environ['ITEM3'] = '50'
|
||||
cfg_dict_dst = dict(item1='60', item2='new_value', item3=50)
|
||||
assert Config._file2dict(cfg_file)[0]['item1'] == cfg_dict_dst['item1']
|
||||
assert Config._file2dict(cfg_file)[0]['item2'] == cfg_dict_dst['item2']
|
||||
assert Config._file2dict(cfg_file)[0]['item3'] == cfg_dict_dst['item3']
|
||||
|
||||
os.environ.pop('ITEM1')
|
||||
os.environ.pop('ITEM2')
|
||||
os.environ.pop('ITEM3')
|
||||
|
||||
def _merge_from_base(self):
|
||||
cfg_file = osp.join(self.data_path,
|
||||
'config/py_config/test_merge_from_base_single.py')
|
||||
|
|
|
|||
Loading…
Reference in New Issue