Merge pull request #3370 from vkarak/feat/yaml-config

[feat] Enable YAML configuration files

Author: vkarak

docs/config_reference.rst

@@ -6,9 +6,22 @@ ReFrame's behavior can be configured through its configuration file, environment
 An option can be specified via multiple paths (e.g., a configuration file parameter and an environment variable), in which case command-line options precede environment variables, which in turn precede configuration file options.
 This section provides a complete reference guide of the configuration options of ReFrame that can be set in its configuration file or specified using environment variables.
 
-ReFrame's configuration is in JSON syntax.
-The full schema describing it can be found in |schemas/config.json|_ file.
-The final configuration for ReFrame is validated against this schema.
+ReFrame's configuration is a JSON object that is stored in either a Python, JSON or YAML file.
+In case of Python configuration, the configuration object must be stored in the special ``site_configuration`` variable:
+
+.. code-block:: python
+
+ site_configuration = {
+ .. # The configuration details.
+ }
+
+The final configuration is validated against the schema |schemas/config.json|_.
+See also :ref:`manpage-configuration` for understanding how ReFrame builds its final configuration.
+
+.. warning::
+ .. versionchanged:: 4.8
+
+ Raw JSON configuration files are deprecated.
 
 The syntax we use to describe the different configuration objects follows the convention: ``OBJECT[.OBJECT]*.PROPERTY``.
 Even if a configuration object contains a list of other objects, this is not reflected in the above syntax, as all objects in a certain list are homogeneous.
@@ -83,6 +96,10 @@ It consists of the following properties, which we also call conventionally *conf
  2. Shell commands: Any string not prefixed with ``py::`` will be treated as a shell command and will be executed *during auto-detection* to retrieve the hostname.
  The standard output of the command will be used.
 
+ .. note::
+
+ For YAML configuration files the ``py::`` prefixed strings cannot refer to user-defined functions.
+
  If the :option:`--system` option is not passed, ReFrame will try to autodetect the current system trying the methods in this list successively, until one of them succeeds.
  The resulting name will be matched against the :attr:`~config.systems.hostnames` patterns of each system and the system that matches first will be used as the current one.
 
@@ -2288,3 +2305,43 @@ A *device info object* in ReFrame's configuration is used to hold information ab
  :default: ``None``
 
  Number of devices of this type inside the system partition.
+
+
+Dynamic configuration
+=====================
+
+One advantage of ReFrame's configuration is that it is programmable, especially if you are using the Python files.
+Since the configuration is loaded as a Python module, you can generate parts of the configuration dynamically.
+
+The YAML configuration on the other hand is more static, although not fully.
+Code generation can still be used with the YAML configuration as it is treated as a Jinja2 template, where ReFrame provides the following bindings:
+
+- ``getenv(<envvar>)``: Retrieve an environment variable.
+- ``gid``: The real group id of the ReFrame process.
+- ``group``: The group name of the ReFrame process.
+- ``hostname``: The local host's hostname.
+- ``uid``: The real user id of the ReFrame process.
+- ``user``: The user name of the ReFrame process.
+
+These are two examples of YAML logging configuration that uses one of those bindings:
+
+.. code-block:: yaml
+
+ logging:
+ - handlers:
+ - type: file
+ name: reframe-{{ hostname }}.log
+ level: debug2
+ format: "[%(asctime)s.%(msecs)03d] %(levelname)s: %(check_info)s: %(message)s"
+ append: false
+
+
+.. code-block:: yaml
+
+ logging:
+ - handlers:
+ - type: file
+ name: reframe-{{ getenv("FOO") }}.log
+ level: debug2
+ format: "[%(asctime)s.%(msecs)03d] %(levelname)s: %(check_info)s: %(message)s"
+ append: false

docs/manpage.rst

@@ -2256,16 +2256,27 @@ Whenever an environment variable is associated with a configuration option, its
  ================================== ==================
 
 
+.. _manpage-configuration:
 
-Configuration File
-==================
+Configuration
+=============
+
+ReFrame's configuration can be stored in one or multiple configuration files.
+Two configuration file types are supported: Python and YAML.
+
+.. note::
+
+ .. versionchanged:: 4.8
+
+ The JSON configuration files are deprecated.
+
+The configuration of ReFrame defines the systems and environments to test as well as parameters controlling the framework's behavior.
 
-The configuration file of ReFrame defines the systems and environments to test as well as parameters controlling the framework's behavior.
+To determine its final configuration, ReFrame executes the following steps:
 
-ReFrame loads multiple configuration files to determine its final configuration.
-First, it loads unconditionally its builtin configuration which is located in ``${RFM_INSTALL_PREFIX}/reframe/core/settings.py``.
-If the :envvar:`RFM_CONFIG_PATH` environment variable is defined, ReFrame will look for configuration files named either ``settings.py`` or ``settings.json`` (in that order) in every location in the path and will load them.
-Finally, the :option:`--config-file` option is processed and any configuration files specified will also be loaded.
+- First, it unconditionally loads the builtin configuration which is located in ``${RFM_INSTALL_PREFIX}/reframe/core/settings.py``.
+- Second, if the :envvar:`RFM_CONFIG_PATH` environment variable is defined, ReFrame will look for configuration files named either ``settings.py`` or ``settings.yaml`` or ``settings.json`` (in that order) in every location in the path and will load them.
+- Finally, the :option:`--config-file` option is processed and any configuration files specified will also be loaded.
 
 For a complete reference of the available configuration options, please refer to the :doc:`reframe.settings(8) <config_reference>` man page.
 

docs/requirements.txt

@@ -1,6 +1,10 @@
 archspec==0.2.5
 docutils==0.18.1
+jinja2==3.0.3; python_version == '3.6'
+jinja2==3.1.4; python_version >= '3.7'
 jsonschema==3.2.0
+PyYAML==6.0.1; python_version < '3.8'
+PyYAML==6.0.2; python_version >= '3.8'
 semver==2.13.0; python_version == '3.6'
 semver==3.0.4; python_version >= '3.7'
 Sphinx==5.3.0; python_version < '3.8'

docs/tutorial.rst

@@ -458,6 +458,25 @@ Note also that the system and environment specification in the test run output i
 ReFrame has determined that the ``default`` partition and the ``baseline`` environment satisfy the test constraints and thus it has run the test with this partition/environment combination.
 
 
+YAML configuration
+------------------
+
+.. versionadded:: 4.8
+
+Apart from Python, ReFrame's configuration can be specified in a YAML file.
+The advantage is a more compact configuration, but it's not fully programmable as is the Python configuration.
+Below is the same configuration file presented above, but in YAML:
+
+
+.. literalinclude:: ../examples/tutorial/config/baseline.yaml
+ :caption:
+ :lines: 6-
+
+If you are using multiple configuration files, ReFrame allows you to "mix and match" the configuration file types: some of them can be in Python, others in YAML.
+
+Note that YAML configuration files offer a minimal programmability as they are essentially `Jinja2 <https://jinja.palletsprojects.com/>`__ templates where a few variables are substituted by ReFrame.
+Check the configuration reference for more information.
+
 .. _compiling-the-test-code:
 
 Compiling the test code

examples/tutorial/config/baseline.yaml

@@ -0,0 +1,18 @@
+# Copyright 2016-2025 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
+# ReFrame Project Developers. See the top-level LICENSE file for details.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+systems:
+- name: tutorialsys
+ descr: Example system
+ hostnames: ['myhost']
+ partitions:
+ - name: default
+ descr: Example partition
+ scheduler: local
+ launcher: local
+ environs: ['baseline']
+environments:
+- name: baseline
+ features: ['stream']

reframe/core/config.py

@@ -8,19 +8,23 @@
 import fnmatch
 import functools
 import importlib
+import io
 import itertools
 import json
 import jsonschema
 import os
 import re
+import socket
+import yaml
+from jinja2.sandbox import SandboxedEnvironment
 
 import reframe
 import reframe.core.settings as settings
 import reframe.utility as util
 import reframe.utility.jsonext as jsonext
 import reframe.utility.osext as osext
 from reframe.core.environments import normalize_module_list
-from reframe.core.exceptions import (ConfigError, ReframeFatalError)
+from reframe.core.exceptions import ConfigError, ReframeFatalError
 from reframe.core.logging import getlogger
 from reframe.utility import ScopedDict
 
@@ -351,14 +355,37 @@ def load_config_python(self, filename):
  def load_config_json(self, filename):
  with open(filename) as fp:
  try:
- config = json.loads(fp.read())
+ config = json.load(fp)
  except json.JSONDecodeError as e:
  raise ConfigError(
  f"invalid JSON syntax in configuration file '{filename}'"
  ) from e
 
  self.update_config(config, filename)
 
+ def load_config_yaml(self, filename):
+ bindings = {
+ 'getenv': os.getenv,
+ 'gid': os.getgid(),
+ 'group': osext.osgroup(),
+ 'hostname': socket.gethostname(),
+ 'uid': os.getuid(),
+ 'user': osext.osuser(),
+ }
+
+ with open(filename) as fp:
+ environment = SandboxedEnvironment()
+ template = environment.from_string(fp.read())
+ yaml_src = template.render(**bindings)
+ try:
+ config = yaml.safe_load(io.StringIO(yaml_src))
+ except Exception as err:
+ raise ConfigError(
+ f'invalid YAML syntax in configuration file `{filename}`'
+ ) from err
+
+ self.update_config(config, filename)
+
  def _setup_autodect_methods(self):
  def _py_meth(m):
  try:
@@ -429,7 +456,7 @@ def _detect_system(self):
  'the `--system` option')
 
  getlogger().debug(f'Retrieved hostname: {hostname!r}')
- getlogger().debug(f'Looking for a matching configuration entry')
+ getlogger().debug('Looking for a matching configuration entry')
  for system in self._site_config['systems']:
  for patt in system['hostnames']:
  if re.match(patt, hostname):
@@ -665,7 +692,13 @@ def load_config(*filenames):
  _, ext = os.path.splitext(f)
  if ext == '.py':
  ret.load_config_python(f)
+ elif ext == '.yaml' or ext == '.yml':
+ ret.load_config_yaml(f)
  elif ext == '.json':
+ getlogger().warning(
+ f'{f}: JSON configuration files are deprecated; '
+ 'please use either a Python or YAML configuration'
+ )
  ret.load_config_json(f)
  else:
  raise ConfigError(f"unknown configuration file type: '{f}'")

reframe/utility/typecheck.py

@@ -46,6 +46,7 @@
  A tuple with elements of type :class:`T`.
 
 .. py:data:: Tuple[T1,T2,...,Tn]
+ :noindex:
 
  A tuple with ``n`` elements, whose types are exactly :class:`T1`,
  :class:`T2`, ..., :class:`Tn` in that order.

requirements.txt

@@ -6,6 +6,8 @@ filelock==3.12.2; python_version == '3.7'
 filelock==3.16.1; python_version == '3.8'
 filelock==3.17.0; python_version > '3.8'
 importlib_metadata==4.0.1; python_version < '3.8'
+jinja2==3.0.3; python_version == '3.6'
+jinja2==3.1.4; python_version >= '3.7'
 jsonschema==3.2.0
 lxml==5.2.0; python_version < '3.8' and platform_machine == 'aarch64'
 lxml==5.3.0; python_version >= '3.8' or platform_machine != 'aarch64'

setup.cfg

@@ -34,9 +34,12 @@ install_requires =
  filelock<=3.16.1; python_version == '3.8'
  filelock<=3.12.2; python_version == '3.7'
  filelock<=3.4.1; python_version == '3.6'
+ jinja2==3.0.3; python_version == '3.6'
+ jinja2
  jsonschema
  lxml==5.2.0; python_version < '3.8' and platform_machine == 'aarch64'
  lxml==5.3.0; python_version >= '3.8' or platform_machine != 'aarch64'
+ PyYAML==6.0.1; python_version < '3.8'
  PyYAML
  requests
  requests <= 2.27.1; python_version == '3.6'

unittests/resources/config/bindings.yaml

@@ -0,0 +1,22 @@
+# Copyright 2016-2025 Swiss National Supercomputing Centre (CSCS/ETH Zurich)
+# ReFrame Project Developers. See the top-level LICENSE file for details.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+# Config file for testing the YAML bindings
+
+systems:
+- name: testsys
+ hostnames: ['.*']
+ partitions:
+ - name: default
+ scheduler: local
+ launcher: local
+ environs: ['builtin']
+ extras:
+ getenv: {{ getenv("_FOO_") }}
+ gid: {{ gid }}
+ group: {{ group }}
+ hostname: {{ hostname }}
+ uid: {{ uid }}
+ user: {{ user }}