SAT: test strictness level migration tooling (create issues script) (#19020)

Author: alafanechere

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/README.md

@@ -0,0 +1,36 @@
+# Tooling for `test_strictness_level` migration
+
+This directory contains scripts that can help us manage the migration of connectors's `acceptance-test-config.yml` to `high` test strictness level.
+Before running these scripts you need to set up a local virtual environment in the **current directory**:
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+## Requirements
+* [GitHub CLI](https://cli.github.com/) (`brew install gh`)
+
+## Create migration issue for GA connectors (`create_issues.py`)
+This script will create one issue per GA connectors to migrate to `high` test strictness level.
+
+### What it does:
+1. Find all GA connectors in `../../../../../airbyte-config/init/src/main/resources/seed/source_definitions.yaml`
+2. Generate an issue content (title, body, labels, project), using `./templates/issue.md.j2`
+3. Find an already existing issue with the same title.
+4. Create the issue and return its url if it does not exist.
+
+Issues get created with the following labels:
+* `area/connectors`
+* `team/connectors-python` 
+* `type/enhancement` 
+* `test-strictness-level`
+
+Issues are added to the following project: `SAT-high-test-strictness-level`
+
+### How to run:
+**Dry run**:
+`python create_issues.py`
+
+**Real execution**:
+`python create_issues.py --dry False`
+

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/__init__.py

@@ -0,0 +1,3 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/create_issues.py

@@ -0,0 +1,83 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#
+
+
+import argparse
+import json
+import logging
+import os
+import subprocess
+import tempfile
+
+from definitions import GA_DEFINITIONS
+from jinja2 import Environment, FileSystemLoader
+
+TEMPLATES_FOLDER = "./templates/"
+COMMON_ISSUE_LABELS = ["area/connectors", "team/connectors-python", "type/enhancement", "test-strictness-level"]
+GITHUB_PROJECT_NAME = "SAT-high-test-strictness-level"
+
+logging.basicConfig(level=logging.DEBUG)
+environment = Environment(loader=FileSystemLoader(TEMPLATES_FOLDER))
+
+parser = argparse.ArgumentParser(description="Create issues for migration of GA connectors to high test strictness level in SAT")
+parser.add_argument("-d", "--dry", default=True)
+
+
+def get_issue_content(source_definition):
+ issue_title = f"Source {source_definition['name']}: enable `high` test strictness level in SAT"
+
+ template = environment.get_template("issue.md.j2")
+ issue_body = template.render(connector_name=source_definition["name"], release_stage=source_definition["releaseStage"])
+ file_definition, issue_body_path = tempfile.mkstemp()
+
+ with os.fdopen(file_definition, "w") as tmp:
+ # do stuff with temp file
+ tmp.write(issue_body)
+
+ return {"title": issue_title, "body_file": issue_body_path, "labels": COMMON_ISSUE_LABELS}
+
+
+def create_issue(source_definition, dry_run=True):
+ issue_content = get_issue_content(source_definition)
+ list_command_arguments = ["gh", "issue", "list", "--state", "open", "--search", f"'{issue_content['title']}'", "--json", "url"]
+
+ create_command_arguments = [
+ "gh",
+ "issue",
+ "create",
+ "--title",
+ issue_content["title"],
+ "--body-file",
+ issue_content["body_file"],
+ "--project",
+ GITHUB_PROJECT_NAME,
+ ]
+ for label in issue_content["labels"]:
+ create_command_arguments += ["--label", label]
+
+ list_existing_issue_process = subprocess.Popen(list_command_arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ stdout, stderr = list_existing_issue_process.communicate()
+ existing_issues = json.loads(stdout.decode())
+ already_created = len(existing_issues) > 0
+ if already_created:
+ logging.warning(f"An issue was already created for this definition: {existing_issues[0]}")
+ if not already_created:
+ if not dry_run:
+ process = subprocess.Popen(create_command_arguments, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+ stdout, stderr = process.communicate()
+ if stderr:
+ logging.error(stderr.decode())
+ else:
+ created_issue_url = stdout.decode()
+ logging.info(f"Created issue for {source_definition['name']}: {created_issue_url}")
+ else:
+ logging.info(f"[DRY RUN]: {' '.join(create_command_arguments)}")
+ os.remove(issue_content["body_file"])
+
+
+if __name__ == "__main__":
+ args = parser.parse_args()
+ dry_run = False if args.dry == "False" or args.dry == "false" else True
+ for definition in GA_DEFINITIONS:
+ create_issue(definition, dry_run=dry_run)

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/definitions.py

@@ -0,0 +1,21 @@
+#
+# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
+#
+
+
+import yaml
+
+SOURCE_DEFINITIONS_FILE_PATH = "../../../../../airbyte-config/init/src/main/resources/seed/source_definitions.yaml"
+
+
+def read_source_definitions():
+ with open(SOURCE_DEFINITIONS_FILE_PATH, "r") as source_definitions_file:
+ return yaml.safe_load(source_definitions_file)
+
+
+def find_by_release_stage(source_definitions, release_stage):
+ return [definition for definition in source_definitions if definition.get("releaseStage") == release_stage]
+
+
+ALL_DEFINITIONS = read_source_definitions()
+GA_DEFINITIONS = find_by_release_stage(ALL_DEFINITIONS, "generally_available")

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/requirements.txt

@@ -0,0 +1,4 @@
+Jinja2==3.1.2
+MarkupSafe==2.1.1
+pyaml==21.10.1
+PyYAML==6.0

airbyte-integrations/bases/source-acceptance-test/tools/strictness_level_migration/templates/issue.md.j2

@@ -0,0 +1,15 @@
+## What
+A `test_strictness_level` field was introduced to Source Acceptance Tests (SAT).
+{{ connector_name }} is a {{ release_stage }} connector, we want it to have a `high` test strictness level.
+
+**This will help**:
+- maximize the SAT coverage on this connector.
+- document its potential weaknesses in term of test coverage.
+
+## How
+1. Migrate the existing `acceptance-test-config.yml` file to the latest configuration format. (See instructions [here](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/source-acceptance-test/README.md#L61))
+2. Enable `high` test strictness level in `acceptance-test-config.yml`. (See instructions [here](https://github.com/airbytehq/airbyte/blob/master/docs/connector-development/testing-connectors/source-acceptance-tests-reference.md#L240))
+3. Commit changes on `acceptance-test-config.yml` and open a PR.
+4. Run SAT with the `/test` command on the branch.
+5. If tests are failing please fix the failing test or use `bypass_reason` fields to explain why a specific test can't be run.
+