Support for optional encryption

[fabiob]

Hi! Thanks for the great work on this little gem, I'm using it for several projects.

Recently I've had to work around the code for a situation that I believe is not uncommon: the settings file is encrypted on the development environment (to easily share the secrets using Git) but it points to a non-encrypted file in production (where it is mounted from a secret volume on Kubernetes, and exposed as unencrypted).

After using my workaround for a few months, I've decided it was time to share them with the project. This PR is the result.

Aside from the use case I've described above, while working on unit tests another one came to my mind: when the settings is loaded from a combination of yaml files, some of them are encrypted using SOPS and some are not. While SOPS allows for partially encrypted files (using encrypted_regex), I think it might be useful to fall back to basic YAML and JSON parsing when the file is not encrypted, instead of throwing an exception.

Hopefully you'll find those as valid use cases for the library. Let me know what you think.

P.S.: this also fixed #19
P.S. 2: I can't use pixi on my NixOS, so I've used uv and updated the pyproject.toml as a result. Let me know if it breaks something on pixi side.

[pavelzw]

looks mostly good, some comments

[pavelzw]

is this = True needed? We are never using this part since we override this in SopsJsonSettingsSource and SopsYamlSettingsSource

Suggested change

	allow_unencrypted: bool = True
	allow_unencrypted: bool

[fabiob]

The ideia was to allow users to use this mixin to write additional settings classes, maybe for Toml or Ini. Setting a default value establishes a default behavior and prevents an AttributeError on the method implementation if the user does not override the attribute.

But I guess we can simplify that and just remove the attribute altogether.

[pavelzw]

exposing this is imo not necessary as it's an internal implementation detail

Suggested change

from .mixin import SopsConfigFileSourceMixin

[fabiob]

Again, the ideia was to allow users to use this mixin to write additional settings classes. But I should have updated the README to clarify this idea.

[pavelzw]

Suggested change

"SopsConfigFileSourceMixin",

s.a.

[pavelzw]

why not also put this behavior in SOPSConfigSettingsSource? We could theoretically try to parse the json / yaml directly using python and see whether a sops key is set and if it's not set, just parse directly when allow_unencrypted?

[fabiob]

That's what my original workaround was about, but I found it somewhat ugly, as I needed to re-detect the file format using the extension:

class OptionalSOPSConfigSettingsSource(SOPSConfigSettingsSource): """  Tries to load settings from SOPS files. If the file is not encrypted, it will be loaded as a YAML or JSON file.  """ def _read_file(self, file_path: Path) -> dict[str, Any]: try: return super()._read_file(file_path) except SopsyCommandFailedError as exc: if not re.match("sops metadata not found", str(exc)): raise match file_path.suffix: case ".yaml": return yaml.safe_load(file_path.read_text()) case ".json": return json.loads(file_path.read_text()) case _ as ext: msg = f"Unsupported file extension: {ext}" raise ValueError(msg) from exc

After reading more about yours and @samuelcolvin 's code, I though the cleanest way was to override the Sops{Format}SettingsSource as I did in this PR. What do you think?

[fabiob]

Let me try to clarify my approach a little more:

The current SOPSConfigSettingsSource does not call the original Yaml or Json implementation, mostly because it does not need to — it delegates to sopsy. But once we add the option to work with unencrypted files, we have to parse the file somehow.

SOPSConfigSettingsSource.__init__ calls ConfigFileSourceMixin._read_files, which then calls back to SOPSConfigSettingsSource._read_file, which is the part that calls sopsy. However, by the time we get here, we lost the information of whether the file was supposed to be handled as JSON or YAML. We need to detect it again, either by extension (which my previous hack did) or by reading the first character. sopsy does use the latter approach in their get_dict function, it works for them since they can be pretty sure there's no BOM as they're handing sops decrypt output, but we don't have the same guarantee.

So my approach in this PR was to inherit the original pydantic-settings SopsJsonSettingsSource and SopsYamlSettingsSource classes. This way we can use super() calls to invoke the original implementation of _read_file(...) once we detect the file is not encrypted. This follows the principle of least surprise, it works just like the original YAML or JSON code from pydantic-settings, with the additional step of trying to decrypt the file before the original code takes over.

P.S.: now that I wrote this last paragraph, I think we can also improve on the POLS by calling sops.decrypt(to_dict=False), write to a temporary file and let the original implementation take over.