onion_config is a Python package designed for easy configuration management. It supports loading and validating configuration data from environment variables and configuration files in JSON and YAML formats. It is a Pydantic based custom configuration package for Python projects.
- Main config based on Pydantic schema - https://pypi.org/project/pydantic
- Load environment variables - https://pypi.org/project/python-dotenv
- Load from multiple configs directories
- Load configs from YAML and JSON files
- Update the default config with additional configurations (
extra_dirdirectory) - Pre-load hook function to modify config data before loading and validation
- Validate config values with Pydantic validators
- Config as dictionary or Pydantic model (with type hints)
- Pre-defined base config schema for common config (
BaseConfig) - Base for custom config loader (
ConfigLoader) - Support Pydantic-v1 and Pydantic-v2
- Install Python (>= v3.9) and pip (>= 23):
- [RECOMMENDED] Miniconda (v3)
- [arm64/aarch64] Miniforge (v3)
- [Python virutal environment] venv
[OPTIONAL] For DEVELOPMENT environment:
- Install git
- Setup an SSH key (video tutorial)
Tip
Skip this step, if you're going to install the package directly from PyPi or GitHub repository.
2.1. Prepare projects directory (if not exists):
# Create projects directory: mkdir -pv ~/workspaces/projects # Enter into projects directory: cd ~/workspaces/projects2.2. Follow one of the below options [A], [B] or [C]:
OPTION A. Clone the repository:
git clone https://github.com/bybatkhuu/module.python-config.git && \ cd module.python-configOPTION B. Clone the repository (for DEVELOPMENT: git + ssh key):
git clone git@github.com:bybatkhuu/module.python-config.git && \ cd module.python-configOPTION C. Download source code:
- Download archived zip file from releases.
- Extract it into the projects directory.
Note
Choose one of the following methods to install the package [A ~ E]:
OPTION A. [RECOMMENDED] Install from PyPi:
Warning
If you wanted to use Pydantic-v1, but if you already installed pydantic-settings and pydantic-core, remove it before installing Pydantic-v1:
pip uninstall -y pydantic-settings pip uninstall -y pydantic-core # Then install with Pydantic-v1: pip install -U onion-config[pydantic-v1]Warning
If you wanted to use Pydantic-v2, but if you already installed onion-config package just by
pip install -U onion-config command, and this will not install pydantic-settings.
For this case, 'env_prefix' WILL NOT WORK for BaseConfig or BaseSettings without pydantic-settings! This is Pydantic-v2's problem, and there could be some other problems.
So fix these issues re-install onion-config with pydantic-settings:
# Install with pydantic-settings for Pydantic-v2: pip install -U onion-config[pydantic-settings]OPTION B. Install latest version directly from GitHub repository:
# Pydantic-v1: pip install git+https://github.com/bybatkhuu/module.python-config.git[pydantic-v1] # Pydantic-v2: pip install git+https://github.com/bybatkhuu/module.python-config.git[pydantic-settings]OPTION C. Install from the downloaded source code:
# Install directly from the source code: # Pydantic-v1: pip install .[pydantic-v1] # Pydantic-v2: pip install .[pydantic-settings] # Or install with editable mode (for DEVELOPMENT): # Pydantic-v1: pip install -e .[pydantic-v1] # Pydantic-v2: pip install -e .[pydantic-settings]OPTION D. Install from pre-built release files:
- Download
.whlor.tar.gzfile from releases - Install with pip:
# Pydantic-v1: # Install from .whl file: pip install ./onion_config-[VERSION]-py3-none-any.whl[pydantic-v1] # Or install from .tar.gz file: pip install ./onion_config-[VERSION].tar.gz[pydantic-v1] # Pydantic-v2: # Install from .whl file: pip install ./onion_config-[VERSION]-py3-none-any.whl[pydantic-settings] # Or install from .tar.gz file: pip install ./onion_config-[VERSION].tar.gz[pydantic-settings]OPTION E. Copy the module into the project directory (for testing):
# Install python dependencies: pip install -r ./requirements/requirements.core.txt # Pydantic-v1: pip install -r ./requirements/requirements.pydantic-v1.txt # Pydantic-v2: pip install -r ./requirements/requirements.pydantic-settings.txt # Copy the module source code into the project: cp -r ./src/onion_config [PROJECT_DIR] # For example: cp -r ./src/onion_config /some/path/project/ENV=productionexamples/simple/configs/1.base.yml:
env: test app: name: "My App" version: "0.0.1" nested: key: "value"examples/simple/configs/2.extra.yml:
app: name: "New App" nested: some: "value" description: "Description of my app." another_val: extra: 1import pprint from loguru import logger try: import pydantic_settings _has_pydantic_settings = True except ImportError: _has_pydantic_settings = False from onion_config import ConfigLoader, BaseConfig class ConfigSchema(BaseConfig): env: str = "local" try: config: ConfigSchema = ConfigLoader(config_schema=ConfigSchema).load() except Exception: logger.exception("Failed to load config:") exit(2) if __name__ == "__main__": logger.info(f"All: {config}") logger.info(f"App name: {config.app['name']}") if _has_pydantic_settings: # Pydantic-v2: logger.info(f"Config:\n{pprint.pformat(config.model_dump())}\n") else: # Pydantic-v1: logger.info(f"Config:\n{pprint.pformat(config.dict())}\n")Run the examples/simple:
cd ./examples/simple python ./main.pyOutput:
2023-09-01 00:00:00.000 | INFO | __main__:<module>:29 - All: env='production' another_val={'extra': 1} app={'name': 'New App', 'version': '0.0.1', 'nested': {'key': 'value', 'some': 'value'}, 'description': 'Description of my app.'} 2023-09-01 00:00:00.000 | INFO | __main__:<module>:30 - App name: New App 2023-09-01 00:00:00.000 | INFO | __main__:<module>:35 - Config: {'another_val': {'extra': 1}, 'app': {'description': 'Description of my app.', 'name': 'New App', 'nested': {'key': 'value', 'some': 'value'}, 'version': '0.0.1'}, 'env': 'production'}ENV=development DEBUG=true APP_NAME="Old App" ONION_CONFIG_EXTRA_DIR="extra_configs"ENV=production APP_NAME="New App" APP_SECRET="my_secret"examples/advanced/configs/config.yml:
env: local app: name: "My App" port: 9000 bind_host: "0.0.0.0" version: "0.0.1" ignore_val: "Ignore me" logger: output: "file"examples/advanced/configs/logger.json:
{ "logger": { "level": "info", "output": "stdout" } }examples/advanced/configs_2/config.yml:
extra: config: key1: 1examples/advanced/configs_2/config_2.yml:
extra: config: key2: 2examples/advanced/extra_configs/extra.json:
{ "extra": { "type": "json" } }from enum import Enum from typing import Union import pydantic from pydantic import Field, SecretStr _has_pydantic_settings = False if "2.0.0" <= pydantic.__version__: try: from pydantic_settings import SettingsConfigDict _has_pydantic_settings = True except ImportError: pass from onion_config import BaseConfig # Environments as Enum: class EnvEnum(str, Enum): LOCAL = "local" DEVELOPMENT = "development" TEST = "test" DEMO = "demo" STAGING = "staging" PRODUCTION = "production" # App config schema: class AppConfig(BaseConfig): name: str = Field("App", min_length=2, max_length=32) bind_host: str = Field("localhost", min_length=2, max_length=128) port: int = Field(8000, ge=80, lt=65536) secret: SecretStr = Field(..., min_length=8, max_length=64) version: str = Field(..., min_length=5, max_length=16) description: Union[str, None] = Field(None, min_length=4, max_length=64) if _has_pydantic_settings: # Pydantic-v2: model_config = SettingsConfigDict(extra="ignore", env_prefix="APP_") else: # Pydantic-v1: class Config: extra = "ignore" env_prefix = "APP_" # Main config schema: class ConfigSchema(BaseConfig): env: EnvEnum = Field(EnvEnum.LOCAL) debug: bool = Field(False) app: AppConfig = Field(...)from loguru import logger from onion_config import ConfigLoader from schema import ConfigSchema # Pre-load function to modify config data before loading and validation: def _pre_load_hook(config_data: dict) -> dict: config_data["app"]["port"] = "80" config_data["extra_val"] = "Something extra!" return config_data config = None try: _config_loader = ConfigLoader( config_schema=ConfigSchema, configs_dirs=["configs", "configs_2", "/not_exists/path/configs_3"], env_file_paths=[".env", ".env.base", ".env.prod"], pre_load_hook=_pre_load_hook, config_data={"base": "start_value"}, warn_mode="ALWAYS", ) # Main config object: config: ConfigSchema = _config_loader.load() except Exception: logger.exception("Failed to load config:") exit(2)import pprint from loguru import logger try: import pydantic_settings _has_pydantic_settings = True except ImportError: _has_pydantic_settings = False from config import config if __name__ == "__main__": logger.info(f"All: {config}") logger.info(f"ENV: {config.env}") logger.info(f"DEBUG: {config.debug}") logger.info(f"Extra: {config.extra_val}") logger.info(f"Logger: {config.logger}") logger.info(f"App: {config.app}") logger.info(f"Secret: '{config.app.secret.get_secret_value()}'\n") if _has_pydantic_settings: # Pydantic-v2: logger.info(f"Config:\n{pprint.pformat(config.model_dump())}\n") else: # Pydantic-v1: logger.info(f"Config:\n{pprint.pformat(config.dict())}\n") try: # This will raise ValidationError config.app.port = 8443 except Exception as e: logger.error(f"{e}\n")Run the examples/advanced:
cd ./examples/advanced python ./main.pyOutput:
2023-09-01 00:00:00.000 | INFO | onion_config._base:load:143 - Loading all configs... 2023-09-01 00:00:00.000 | WARNING | onion_config._base:_load_dotenv_file:201 - '/home/user/workspaces/projects/onion_config/examples/advanced/.env' file is not exist! 2023-09-01 00:00:00.000 | WARNING | onion_config._base:_load_configs_dir:257 - '/not_exists/path/configs_3' directory is not exist! 2023-09-01 00:00:00.000 | SUCCESS | onion_config._base:load:171 - Successfully loaded all configs! 2023-09-01 00:00:00.000 | INFO | __main__:<module>:19 - All: env=<EnvEnum.PRODUCTION: 'production'> debug=True app=AppConfig(name='New App', bind_host='0.0.0.0', port=80, secret=SecretStr('**********'), version='0.0.1', description=None) extra={'config': {'key1': 1, 'key2': 2}, 'type': 'json'} extra_val='Something extra!' logger={'output': 'stdout', 'level': 'info'} base='start_value' 2023-09-01 00:00:00.000 | INFO | __main__:<module>:20 - ENV: production 2023-09-01 00:00:00.000 | INFO | __main__:<module>:21 - DEBUG: True 2023-09-01 00:00:00.000 | INFO | __main__:<module>:22 - Extra: Something extra! 2023-09-01 00:00:00.000 | INFO | __main__:<module>:23 - Logger: {'output': 'stdout', 'level': 'info'} 2023-09-01 00:00:00.000 | INFO | __main__:<module>:24 - App: name='New App' bind_host='0.0.0.0' port=80 secret=SecretStr('**********') version='0.0.1' description=None 2023-09-01 00:00:00.000 | INFO | __main__:<module>:25 - Secret: 'my_secret' 2023-09-01 00:00:00.000 | INFO | __main__:<module>:30 - Config: {'app': {'bind_host': '0.0.0.0', 'description': None, 'name': 'New App', 'port': 80, 'secret': SecretStr('**********'), 'version': '0.0.1'}, 'base': 'start_value', 'debug': True, 'env': <EnvEnum.PRODUCTION: 'production'>, 'extra': {'config': {'key1': 1, 'key2': 2}, 'type': 'json'}, 'extra_val': 'Something extra!', 'logger': {'level': 'info', 'output': 'stdout'}} 2023-09-01 00:00:00.000 | ERROR | __main__:<module>:36 - "AppConfig" is immutable and does not support item assignment👍
# ENV=development # DEBUG=true ONION_CONFIG_EXTRA_DIR="./extra_configs"To run tests, run the following command:
# Install core dependencies: pip install -r ./requirements/requirements.core.txt # Pydantic-v1: pip install -r ./requirements/requirements.pydantic-v1.txt # Pydantic-v2: pip install -r ./requirements/requirements.pydantic-settings.txt # Install python test dependencies: pip install -r ./requirements.test.txt # Run tests: python -m pytest -sv -o log_cli=true # Or use the test script: ./scripts/test.sh -l -v -cTo build the python package, run the following command:
# Install python build dependencies: pip install -r ./requirements/requirements.build.txt # Build python package: python -m build # Or use the build script: ./scripts/build.shTo build the documentation, run the following command:
# Install python documentation dependencies: pip install -r ./requirements/requirements.docs.txt # Serve documentation locally (for development): mkdocs serve # Or use the docs script: ./scripts/docs.sh # Or build documentation: mkdocs build # Or use the docs script: ./scripts/docs.sh -b- https://docs.pydantic.dev
- https://github.com/pydantic/pydantic
- https://docs.pydantic.dev/latest/usage/pydantic_settings
- https://github.com/pydantic/pydantic-settings
- https://saurabh-kumar.com/python-dotenv
- https://github.com/theskumar/python-dotenv
- https://packaging.python.org/tutorials/packaging-projects