🎉 Source e2e test: support custom schema #9720

tuliren · 2022-01-23T11:44:16Z

What

Add new mode in the current source-e2e-test connector to allow users to specify arbitrary catalog with single or multiple streams.
Add a cloud variant of the testing source.
Resolve Create a mock source connector #9532.

TODOs

Add cloud variant.

UI

Single-stream continuous feed mode

Multi-stream continuous feed mode

🚨 User Impact 🚨

None.
The legacy modes are not changed. They are simply filtered out in the cloud variant of this connector.

Pre-merge Checklist

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- ~~Connector's bootstrap.md. See description and examples~~
- docs/SUMMARY.md
- docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
- docs/integrations/README.md
- airbyte-integrations/builds.md
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

tuliren · 2022-01-23T12:18:34Z

/test connector=connectors/source-e2e-test

🕑 connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1735869155
❌ connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1735869155
🐛

tuliren · 2022-01-23T12:41:23Z

/test connector=connectors/source-e2e-test

🕑 connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1735916910
✅ connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1735916910
No Python unittests run

tuliren · 2022-01-23T20:00:36Z

/test connector=connectors/source-e2e-test

🕑 connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1736866343
✅ connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1736866343
No Python unittests run

tuliren · 2022-01-24T01:06:16Z

/test connector=connectors/source-e2e-test

🕑 connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1737549944
✅ connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1737549944
No Python unittests run

❌ connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1737549944
🐛

🕑 connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1737549944
✅ connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1737549944
No Python unittests run

tuliren · 2022-01-24T01:06:22Z

/test connector=connectors/source-e2e-test-cloud

🕑 connectors/source-e2e-test-cloud https://github.com/airbytehq/airbyte/actions/runs/1737550109
✅ connectors/source-e2e-test-cloud https://github.com/airbytehq/airbyte/actions/runs/1737550109
No Python unittests run

tuliren · 2022-01-24T04:42:10Z

To unblock Subodh, I will publish and merge this PR now. But feel free to leave any comment or request changes. I will update the connector in a follow-up PR.

tuliren · 2022-01-24T04:42:25Z

/publish connector=connectors/source-e2e-test

🕑 connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1738099825
✅ connectors/source-e2e-test https://github.com/airbytehq/airbyte/actions/runs/1738099825

tuliren · 2022-01-24T04:42:31Z

/publish connector=connectors/source-e2e-test-cloud

🕑 connectors/source-e2e-test-cloud https://github.com/airbytehq/airbyte/actions/runs/1738099908
✅ connectors/source-e2e-test-cloud https://github.com/airbytehq/airbyte/actions/runs/1738099908

edgao · 2022-01-24T17:17:36Z

...rce-e2e-test/src/main/java/io/airbyte/integrations/source/e2e_test/ContinuousFeedSource.java

+
+ final SchemaStore schemaStore = new SchemaStore(true);
+ final Schema schema = schemaStore.loadSchemaJson(Jsons.serialize(stream.getStream().getJsonSchema()));
+ final Generator generator = new Generator(ContinuousFeedConstants.MOCK_JSON_CONFIG, schemaStore, random.get());


nitpick: I think this is equivalent to just passing new Random(seed), since you're not passing the threadlocal into a separate thread

Good point. Will do.

edgao · 2022-01-24T17:20:09Z

...rce-e2e-test/src/main/java/io/airbyte/integrations/source/e2e_test/ContinuousFeedSource.java

+ return endOfData();
+ }
+
+ if (messageIntervalMs.isPresent() && emittedMessages.get() != 0) {


checking my understanding: the iterator will be processed single-threadedly?

I think so. Looks like the read is handled in one thread only.

...rce-e2e-test/src/main/java/io/airbyte/integrations/source/e2e_test/ContinuousFeedSource.java

airbyte-integrations/connectors/source-e2e-test/src/main/resources/json_schema_draft_07.json

...e2e-test/src/test/java/io/airbyte/integrations/source/e2e_test/ContinuousFeedConfigTest.java

...ntegrations/connectors/source-e2e-test/src/test/resources/parse_mock_catalog_test_cases.json

edgao · 2022-01-24T17:57:03Z

...rce-test/src/main/java/io/airbyte/integrations/standardtest/source/SourceAcceptanceTest.java

 });
 }

+ protected void assertRecordMessages(final List<AirbyteRecordMessage> recordMessages) {


can you add a javadoc comment explaining what this method should do?

Sure. Will do.

edgao · 2022-01-24T17:59:13Z

...egration/java/io/airbyte/integrations/source/e2e_test/CloudTestingSourcesAcceptanceTest.java

+import java.util.concurrent.ThreadLocalRandom;
+
+/**
+ * This acceptance test is mostly the same as {@code ContinuousFeedSourceAcceptanceTest}. The only


could this just extend ContinuousFeedSourceAcceptanceTest and override getImageName?

I think this is possible. However, I was not able to make gradle work with this set up. Will leave a TODO for now.

edgao · 2022-01-24T17:59:55Z

airbyte-integrations/connectors/source-e2e-test-cloud/README.md

@@ -0,0 +1,67 @@
+# End-to-End Testing Source Cloud Variant
+
+This is the Cloud variant of the [E2E Test Source](https://docs.airbyte.io/integrations/sources/e2e-test). It only allows the "continuous feed" mode with finite number of record messages.


can you add an explanation of why this is necessary+separate from the non-cloud variant?

Sure. Good point.

github-actions bot added the area/connectors Connector related issues label Jan 23, 2022

tuliren temporarily deployed to more-secrets January 23, 2022 12:20 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 23, 2022 12:20 Inactive

tuliren temporarily deployed to more-secrets January 23, 2022 12:42 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 23, 2022 12:43 Inactive

Add continuous feed mode to source e2e-test

dba33fe

tuliren force-pushed the liren/mock-api-source branch from bb2b890 to dba33fe Compare January 23, 2022 19:56

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 23, 2022

tuliren temporarily deployed to more-secrets January 23, 2022 19:57 Inactive

tuliren requested review from edgao and subodh1810 January 23, 2022 20:01

tuliren marked this pull request as ready for review January 23, 2022 20:01

octavia-squidington-iii temporarily deployed to more-secrets January 23, 2022 20:03 Inactive

Update connector catalog doc

da65c05

tuliren temporarily deployed to more-secrets January 23, 2022 20:07 Inactive

tuliren linked an issue Jan 23, 2022 that may be closed by this pull request

Create a mock source connector #9532

Closed

Fix sonar qube issues

a818e5a

tuliren temporarily deployed to more-secrets January 23, 2022 21:27 Inactive

tuliren added 2 commits January 23, 2022 17:02

Add cloud variant

0342d16

Format code

347c048

tuliren temporarily deployed to more-secrets January 24, 2022 01:07 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 24, 2022 01:08 Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 24, 2022 01:23 Inactive

tuliren added the area/benchmark label Jan 24, 2022

octavia-squidington-iii temporarily deployed to more-secrets January 24, 2022 04:44 Inactive

tuliren self-assigned this Jan 24, 2022

Add testing source connector to seed

5312a85

tuliren temporarily deployed to more-secrets January 24, 2022 04:55 Inactive

tuliren merged commit b269b9f into master Jan 24, 2022

tuliren deleted the liren/mock-api-source branch January 24, 2022 05:09

edgao reviewed Jan 24, 2022

View reviewed changes

octavia-squidington-iii mentioned this pull request Jan 24, 2022

Bump Airbyte version from 0.35.7-alpha to 0.35.8-alpha #9762

Merged

tuliren mentioned this pull request Jan 25, 2022

Address review comments for mock source #9773

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

🎉 Source e2e test: support custom schema #9720

🎉 Source e2e test: support custom schema #9720

Uh oh!

tuliren commented Jan 23, 2022 •

edited

Loading

tuliren commented Jan 23, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 23, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 23, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 24, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 24, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 24, 2022

tuliren commented Jan 24, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 24, 2022 •

edited by github-actions bot

Loading

edgao Jan 24, 2022

tuliren Jan 24, 2022 •

edited

Loading

edgao Jan 24, 2022

tuliren Jan 24, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

edgao Jan 24, 2022

tuliren Jan 24, 2022

edgao Jan 24, 2022

tuliren Jan 25, 2022

edgao Jan 24, 2022

tuliren Jan 25, 2022

Labels

4 participants

		@@ -0,0 +1,67 @@
		# End-to-End Testing Source Cloud Variant

		This is the Cloud variant of the [E2E Test Source](https://docs.airbyte.io/integrations/sources/e2e-test). It only allows the "continuous feed" mode with finite number of record messages.

🎉 Source e2e test: support custom schema #9720

🎉 Source e2e test: support custom schema #9720

Uh oh!

Conversation

tuliren commented Jan 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

TODOs

UI

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

tuliren commented Jan 23, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tuliren commented Jan 23, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tuliren commented Jan 23, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tuliren commented Jan 24, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tuliren commented Jan 24, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tuliren commented Jan 24, 2022

tuliren commented Jan 24, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tuliren commented Jan 24, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

tuliren Jan 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Labels

4 participants

tuliren commented Jan 23, 2022 •

edited

Loading

tuliren commented Jan 23, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 23, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 23, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 24, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 24, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 24, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 24, 2022 •

edited by github-actions bot

Loading

tuliren Jan 24, 2022 •

edited

Loading