ci: Remove toxgen check

.github/workflows/ci.yml

@@ -33,28 +33,6 @@ jobs:
  pip install tox
  tox -e linters
 
- check-ci-config:
- name: Check CI config
- runs-on: ubuntu-latest
- timeout-minutes: 10
-
- steps:
- - uses: actions/checkout@v5.0.0
- with:
- ref: ${{ github.event.pull_request.head.sha }}
- fetch-depth: 0
- - uses: actions/setup-python@v6
- with:
- python-version: 3.12
-
- - name: Detect unexpected changes to tox.ini or CI
- run: |
- pip install -e .
- pip install -r scripts/populate_tox/requirements.txt
- python scripts/populate_tox/populate_tox.py --fail-on-changes
- pip install -r scripts/split_tox_gh_actions/requirements.txt
- python scripts/split_tox_gh_actions/split_tox_gh_actions.py --fail-on-changes
-
  build_lambda_layer:
  name: Build Package
  runs-on: ubuntu-latest

scripts/populate_tox/README.md

@@ -14,25 +14,24 @@ combination of hardcoded and generated entries.
 
 The `populate_tox.py` script fills out the auto-generated part of that template.
 It does this by querying PyPI for each framework's package and its metadata and
-then determining which versions make sense to test to get good coverage.
+then determining which versions it makes sense to test to get good coverage.
 
 By default, the lowest supported and latest version of a framework are always
 tested, with a number of releases in between:
 - If the package has majors, we pick the highest version of each major.
 - If the package doesn't have multiple majors, we pick two versions in between
  lowest and highest.
 
-#### Caveats
+Each test suite requires at least some configuration to be added to
+`TEST_SUITE_CONFIG` in `scripts/populate_tox/config.py`. If you're adding a new
+integration, check out the [Add a new test suite](#add-a-new-test-suite) section.
 
-- Make sure the integration name is the same everywhere. If it consists of
- multiple words, use an underscore instead of a hyphen.
+## Test suite config
 
-## Defining constraints
-
-The `TEST_SUITE_CONFIG` dictionary defines, for each integration test suite,
-the main package (framework, library) to test with; any additional test
-dependencies, optionally gated behind specific conditions; and optionally
-the Python versions to test on.
+The `TEST_SUITE_CONFIG` dictionary in `scripts/populate_tox/config.py` defines,
+for each integration test suite, the main package (framework, library) to test
+with; any additional test dependencies, optionally gated behind specific
+conditions; and optionally the Python versions to test on.
 
 Constraints are defined using the format specified below. The following sections
 describe each key.
@@ -58,7 +57,7 @@ in [packaging.specifiers](https://packaging.pypa.io/en/stable/specifiers.html).
 
 ### `package`
 
-The name of the third party package as it's listed on PyPI. The script will
+The name of the third-party package as it's listed on PyPI. The script will
 be picking different versions of this package to test.
 
 This key is mandatory.
@@ -69,15 +68,15 @@ The test dependencies of the test suite. They're defined as a dictionary of
 `rule: [package1, package2, ...]` key-value pairs. All packages
 in the package list of a rule will be installed as long as the rule applies.
 
-`rule`s are predefined. Each `rule` must be one of the following:
+Each `rule` must be one of the following:
  - `*`: packages will be always installed
  - a version specifier on the main package (e.g. `<=0.32`): packages will only
  be installed if the main package falls into the version bounds specified
  - specific Python version(s) in the form `py3.8,py3.9`: packages will only be
  installed if the Python version matches one from the list
 
 Rules can be used to specify version bounds on older versions of the main
-package's dependencies, for example. If e.g. Flask tests generally need
+package's dependencies, for example. If Flask tests generally need
 Werkzeug and don't care about its version, but Flask older than 3.0 needs
 a specific Werkzeug version to work, you can say:
 
@@ -176,7 +175,7 @@ be expressed like so:
 ### `integration_name`
 
 Sometimes, the name of the test suite doesn't match the name of the integration.
-For example, we have the `openai_base` and `openai_notiktoken` test suites, both
+For example, we have the `openai-base` and `openai-notiktoken` test suites, both
 of which are actually testing the `openai` integration. If this is the case, you
 can use the `integration_name` key to define the name of the integration. If not
 provided, it will default to the name of the test suite.
@@ -193,6 +192,11 @@ greater than 2, as the oldest and latest supported versions will always be
 picked. Additionally, if there is a recent prerelease, it'll also always be
 picked (this doesn't count towards `num_versions`).
 
+For instance, `num_versions` set to `2` will only test the first supported and
+the last release of the package. `num_versions` equal to `3` will test the first
+supported, the last release, and one release in between; `num_versions` set to `4`
+will test an additional release in between. In all these cases, if there is
+a recent prerelease, it'll be picked as well in addition to the picked versions.
 
 ## How-Tos
 
@@ -202,9 +206,10 @@ picked (this doesn't count towards `num_versions`).
  in `integrations/__init__.py`. This should be the lowest version of the
  framework that we can guarantee works with the SDK. If you've just added the
  integration, you should generally set this to the latest version of the framework
- at the time.
+ at the time, unless you've verified the integration works for earlier versions
+ as well.
 2. Add the integration and any constraints to `TEST_SUITE_CONFIG`. See the
- "Defining constraints" section for the format.
+ [Test suite config](#test-suite-config) section for the format.
 3. Add the integration to one of the groups in the `GROUPS` dictionary in
  `scripts/split_tox_gh_actions/split_tox_gh_actions.py`.
 4. Run `scripts/generate-test-files.sh` and commit the changes.

scripts/populate_tox/config.py

@@ -1,7 +1,6 @@
 # The TEST_SUITE_CONFIG dictionary defines, for each integration test suite,
-# the main package (framework, library) to test with; any additional test
-# dependencies, optionally gated behind specific conditions; and optionally
-# the Python versions to test on.
+# at least the main package (framework, library) to test with. Additional
+# test dependencies, Python versions to test on, etc. can also be defined here.
 #
 # See scripts/populate_tox/README.md for more info on the format and examples.
 

scripts/populate_tox/populate_tox.py

@@ -130,7 +130,8 @@ def _save_to_cache(package: str, version: Version, release: Optional[dict]) -> N
 
 
 def _prefilter_releases(
- integration: str, releases: dict[str, dict], older_than: Optional[datetime] = None
+ integration: str,
+ releases: dict[str, dict],
 ) -> tuple[list[Version], Optional[Version]]:
  """
  Filter `releases`, removing releases that are for sure unsupported.
@@ -178,9 +179,6 @@ def _prefilter_releases(
 
  uploaded = datetime.fromisoformat(meta["upload_time_iso_8601"])
 
- if older_than is not None and uploaded > older_than:
- continue
-
  if CUTOFF is not None and uploaded < CUTOFF:
  continue
 
@@ -224,7 +222,7 @@ def _prefilter_releases(
 
 
 def get_supported_releases(
- integration: str, pypi_data: dict, older_than: Optional[datetime] = None
+ integration: str, pypi_data: dict
 ) -> tuple[list[Version], Optional[Version]]:
  """
  Get a list of releases that are currently supported by the SDK.
@@ -236,17 +234,15 @@ def get_supported_releases(
  We return the list of supported releases and optionally also the newest
  prerelease, if it should be tested (meaning it's for a version higher than
  the current stable version).
-
- If an `older_than` timestamp is provided, no release newer than that will be
- considered.
  """
  package = pypi_data["info"]["name"]
 
  # Get a consolidated list without taking into account Python support yet
  # (because that might require an additional API call for some
  # of the releases)
  releases, latest_prerelease = _prefilter_releases(
- integration, pypi_data["releases"], older_than
+ integration,
+ pypi_data["releases"],
  )
 
  def _supports_lowest(release: Version) -> bool:
@@ -665,32 +661,10 @@ def _normalize_release(release: dict) -> dict:
  return normalized
 
 
-def main(fail_on_changes: bool = False) -> dict[str, list]:
+def main() -> dict[str, list]:
  """
  Generate tox.ini from the tox.jinja template.
-
- The script has two modes of operation:
- - fail on changes mode (if `fail_on_changes` is True)
- - normal mode (if `fail_on_changes` is False)
-
- Fail on changes mode is run on every PR to make sure that `tox.ini`,
- `tox.jinja` and this script don't go out of sync because of manual changes
- in one place but not the other.
-
- Normal mode is meant to be run as a cron job, regenerating tox.ini and
- proposing the changes via a PR.
  """
- print(f"Running in {'fail_on_changes' if fail_on_changes else 'normal'} mode.")
- last_updated = get_last_updated()
- if fail_on_changes:
- # We need to make the script ignore any new releases after the last updated
- # timestamp so that we don't fail CI on a PR just because a new package
- # version was released, leading to unrelated changes in tox.ini.
- print(
- f"Since we're in fail_on_changes mode, we're only considering "
- f"releases before the last tox.ini update at {last_updated.isoformat()}."
- )
-
  global MIN_PYTHON_VERSION, MAX_PYTHON_VERSION
  meta = _fetch_sdk_metadata()
  sdk_python_versions = _parse_python_versions_from_classifiers(
@@ -736,12 +710,7 @@ def main(fail_on_changes: bool = False) -> dict[str, list]:
 
  # Get the list of all supported releases
 
- # If in fail-on-changes mode, ignore releases newer than `last_updated`
- older_than = last_updated if fail_on_changes else None
-
- releases, latest_prerelease = get_supported_releases(
- integration, pypi_data, older_than
- )
+ releases, latest_prerelease = get_supported_releases(integration, pypi_data)
 
  if not releases:
  print(" Found no supported releases.")
@@ -778,9 +747,6 @@ def main(fail_on_changes: bool = False) -> dict[str, list]:
  }
  )
 
- if fail_on_changes:
- old_file_hash = get_file_hash()
-
  write_tox_file(packages)
 
  # Sort the release cache file
@@ -798,36 +764,13 @@ def main(fail_on_changes: bool = False) -> dict[str, list]:
  ):
  releases_cache.write(json.dumps(release) + "\n")
 
- if fail_on_changes:
- new_file_hash = get_file_hash()
- if old_file_hash != new_file_hash:
- raise RuntimeError(
- dedent(
- """
- Detected that `tox.ini` is out of sync with
- `scripts/populate_tox/tox.jinja` and/or
- `scripts/populate_tox/populate_tox.py`. This might either mean
- that `tox.ini` was changed manually, or the `tox.jinja`
- template and/or the `populate_tox.py` script were changed without
- regenerating `tox.ini`.
-
- Please don't make manual changes to `tox.ini`. Instead, make the
- changes to the `tox.jinja` template and/or the `populate_tox.py`
- script (as applicable) and regenerate the `tox.ini` file by
- running scripts/generate-test-files.sh
- """
- )
- )
- print("Done checking tox.ini. Looking good!")
- else:
- print(
- "Done generating tox.ini. Make sure to also update the CI YAML "
- "files to reflect the new test targets."
- )
+ print(
+ "Done generating tox.ini. Make sure to also update the CI YAML "
+ "files to reflect the new test targets."
+ )
 
  return packages
 
 
 if __name__ == "__main__":
- fail_on_changes = len(sys.argv) == 2 and sys.argv[1] == "--fail-on-changes"
- main(fail_on_changes)
+ main()