🎉 Source Harvest: add OAuth 2.0 #7952

grubberr · 2021-11-14T17:46:09Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

How

Add an implementation of oAuth2 logic to the connector and the Java server part

Manual result of tests:

Redirect to the Harvest auth window:

2. Auth result:

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2021-11-16T13:44:34Z

/test connector=connectors/source-harvest

🕑 connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1467105606
❌ connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1467105606
🐛

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2021-11-16T14:30:16Z

/test connector=connectors/source-harvest

🕑 connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1467295934
❌ connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1467295934
🐛

grubberr · 2021-11-16T14:57:43Z

/test connector=connectors/source-harvest

🕑 connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1467409208
❌ connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1467409208
🐛

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2021-11-16T15:17:32Z

/test connector=connectors/source-harvest

🕑 connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1467502179
✅ connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1467502179
Python tests coverage:

 ---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover ------------------------------------------------ source_harvest/__init__.py 2 2 0% source_harvest/auth.py 13 13 0% source_harvest/source.py 33 33 0% source_harvest/streams.py 162 162 0% ------------------------------------------------ TOTAL 210 210 0% ---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover ------------------------------------------------------------------------ source_acceptance_test/__init__.py 2 0 100% source_acceptance_test/base.py 10 4 60% source_acceptance_test/config.py 75 8 89% source_acceptance_test/conftest.py 108 108 0% source_acceptance_test/plugin.py 47 47 0% source_acceptance_test/tests/__init__.py 4 0 100% source_acceptance_test/tests/test_core.py 200 94 53% source_acceptance_test/tests/test_full_refresh.py 38 27 29% source_acceptance_test/tests/test_incremental.py 69 38 45% source_acceptance_test/utils/__init__.py 6 0 100% source_acceptance_test/utils/asserts.py 37 2 95% source_acceptance_test/utils/common.py 41 24 41% source_acceptance_test/utils/compare.py 62 25 60% source_acceptance_test/utils/connector_runner.py 82 49 40% source_acceptance_test/utils/json_schema_helper.py 115 14 88% ------------------------------------------------------------------------ TOTAL 896 440 51%

antixar

Please add basic tests for Java part. You can use examples of other ready connectors.

airbyte-integrations/connectors/source-harvest/source_harvest/auth.py

grubberr · 2021-11-19T14:01:32Z

/test connector=connectors/source-harvest

🕑 connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1481293429
✅ connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1481293429
Python tests coverage:

 ---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover ------------------------------------------------ source_harvest/__init__.py 2 2 0% source_harvest/auth.py 11 11 0% source_harvest/source.py 33 33 0% source_harvest/streams.py 162 162 0% ------------------------------------------------ TOTAL 208 208 0% ---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover ------------------------------------------------------------------------ source_acceptance_test/__init__.py 2 0 100% source_acceptance_test/base.py 10 4 60% source_acceptance_test/config.py 75 8 89% source_acceptance_test/conftest.py 108 108 0% source_acceptance_test/plugin.py 47 47 0% source_acceptance_test/tests/__init__.py 4 0 100% source_acceptance_test/tests/test_core.py 200 94 53% source_acceptance_test/tests/test_full_refresh.py 38 27 29% source_acceptance_test/tests/test_incremental.py 69 38 45% source_acceptance_test/utils/__init__.py 6 0 100% source_acceptance_test/utils/asserts.py 37 2 95% source_acceptance_test/utils/common.py 41 24 41% source_acceptance_test/utils/compare.py 62 25 60% source_acceptance_test/utils/connector_runner.py 82 49 40% source_acceptance_test/utils/json_schema_helper.py 115 14 88% ------------------------------------------------------------------------ TOTAL 896 440 51%

ChristopheDuong · 2021-11-19T14:07:32Z

airbyte-integrations/connectors/source-harvest/source_harvest/spec.json

+ "complete_oauth_output_specification": {
+ "refresh_token": {
+ "type": "string",
+ "path_in_connector_config": ["credentials", 0, "refresh_token"]


Suggested change

"path_in_connector_config": ["credentials", 0, "refresh_token"]

"path_in_connector_config": ["credentials", "refresh_token"]

ChristopheDuong · 2021-11-19T14:07:43Z

airbyte-integrations/connectors/source-harvest/source_harvest/spec.json

+ "complete_oauth_server_output_specification": {
+ "client_id": {
+ "type": "string",
+ "path_in_connector_config": ["credentials", 0, "client_id"]


Suggested change

"path_in_connector_config": ["credentials", 0, "client_id"]

"path_in_connector_config": ["credentials", "client_id"]

ChristopheDuong · 2021-11-19T14:07:53Z

airbyte-integrations/connectors/source-harvest/source_harvest/spec.json

+ },
+ "client_secret": {
+ "type": "string",
+ "path_in_connector_config": ["credentials", 0, "client_secret"]


Suggested change

"path_in_connector_config": ["credentials", 0, "client_secret"]

"path_in_connector_config": ["credentials", "client_secret"]

ChristopheDuong · 2021-11-19T14:10:00Z

airbyte-integrations/connectors/source-harvest/source_harvest/spec.json

+ "oauth_config_specification": {
+ "complete_oauth_output_specification": {


Suggested change

"oauth_config_specification": {

"complete_oauth_output_specification": {

"oauth_config_specification": {

"predicate_key": ["credentials", "auth_type"],

"predicate_value": "Client",

"complete_oauth_output_specification": {

is it mistake here ?
I see that predicate_key not inside oauth_config_specification

airbyte/airbyte-api/src/main/openapi/config.yaml

Line 3255 in c5a7267

predicateKey:

oh you're right then it's:

Suggested change

"oauth_config_specification": {

"complete_oauth_output_specification": {

"predicate_key": ["credentials", "auth_type"],

"predicate_value": "Client",

"oauth_config_specification": {

"complete_oauth_output_specification": {

ChristopheDuong

The object inside:

complete_oauth_output_specification
complete_oauth_server_input_specification
complete_oauth_server_output_specification

needs to be JSON schema object, so you need to add a layer of

 "type": "object", "additionalProperties": false, "properties": {

ChristopheDuong · 2021-11-19T14:27:39Z

airbyte-integrations/connectors/source-harvest/source_harvest/spec.json

+ "complete_oauth_server_input_specification": {
+ "client_id": {
+ "type": "string"
+ },
+ "client_secret": {
+ "type": "string"
+ }
+ },


Suggested change

"complete_oauth_server_input_specification": {

"client_id": {

"type": "string"

},

"client_secret": {

"type": "string"

}

},

"complete_oauth_server_input_specification": {

"type": "object",

"additionalProperties": "false",

"properties": {

"client_id": {

"type": "string"

},

"client_secret": {

"type": "string"

}

}

},

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2021-11-19T16:15:24Z

/test connector=connectors/source-harvest

🕑 connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1481795172
✅ connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1481795172
Python tests coverage:

 ---------- coverage: platform linux, python 3.8.10-final-0 ----------- /actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-harvest/.venv/lib/python3.8/site-packages/coverage/control.py:761: CoverageWarning: No data was collected. (no-data-collected) Name Stmts Miss Cover self._warn("No data was collected.", slug="no-data-collected") ------------------------------------------------ source_harvest/__init__.py 2 2 0% source_harvest/auth.py 11 11 0% source_harvest/source.py 33 33 0% source_harvest/streams.py 162 162 0% ------------------------------------------------ TOTAL 208 208 0% ---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover ------------------------------------------------------------------------ source_acceptance_test/__init__.py 2 0 100% source_acceptance_test/base.py 10 4 60% source_acceptance_test/config.py 75 8 89% source_acceptance_test/conftest.py 108 108 0% source_acceptance_test/plugin.py 47 47 0% source_acceptance_test/tests/__init__.py 4 0 100% source_acceptance_test/tests/test_core.py 200 94 53% source_acceptance_test/tests/test_full_refresh.py 38 27 29% source_acceptance_test/tests/test_incremental.py 69 38 45% source_acceptance_test/utils/__init__.py 6 0 100% source_acceptance_test/utils/asserts.py 37 2 95% source_acceptance_test/utils/common.py 41 24 41% source_acceptance_test/utils/compare.py 62 25 60% source_acceptance_test/utils/connector_runner.py 82 49 40% source_acceptance_test/utils/json_schema_helper.py 115 14 88% ------------------------------------------------------------------------ TOTAL 896 440 51%

ChristopheDuong · 2021-11-19T16:18:59Z

tools/bin/ci_credentials.sh

 read_secrets source-google-workspace-admin-reports "$GOOGLE_WORKSPACE_ADMIN_REPORTS_TEST_CREDS"
 read_secrets source-greenhouse "$GREENHOUSE_TEST_CREDS"
 read_secrets source-greenhouse "$GREENHOUSE_TEST_CREDS_LIMITED" "config_users_only.json"
-read_secrets source-harvest "$HARVEST_INTEGRATION_TESTS_CREDS"


curious why are these removed?

We are switching from Github secrets to Google Secrets. New approach based on google secrets allow to dynamically generate secrets/config.json for current connector without this hard-coded variable.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2021-11-19T18:09:39Z

/publish connector=connectors/source-harvest

🕑 connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1482180172
✅ connectors/source-harvest https://github.com/airbytehq/airbyte/actions/runs/1482180172

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

* add OAuth 2.0 for source-harvest Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

add OAuth 2.0 for source-harvest

de803ec

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/connectors Connector related issues label Nov 14, 2021

grubberr linked an issue Nov 14, 2021 that may be closed by this pull request

Source Harvest: support oauth #7707

Closed

grubberr self-assigned this Nov 14, 2021

grubberr temporarily deployed to more-secrets November 14, 2021 17:47 Inactive

changelog added

524dfd1

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot added the area/documentation Improvements or additions to documentation label Nov 14, 2021

grubberr temporarily deployed to more-secrets November 14, 2021 17:50 Inactive

grubberr added 2 commits November 15, 2021 19:31

Merge branch 'master' into grubberr/7707-source-harvest

7939aef

oneOf added for spec.json

4f84a8c

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets November 15, 2021 21:03 Inactive

old_config.json, config_oauth.json added

9573015

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr requested review from antixar and bazarnov November 15, 2021 21:57

bazarnov approved these changes Nov 16, 2021

View reviewed changes

Merge branch 'master' into grubberr/7707-source-harvest

2fb98b8

grubberr temporarily deployed to more-secrets November 16, 2021 13:44 Inactive

jrhizor temporarily deployed to more-secrets November 16, 2021 13:47 Inactive

fix read_secret names HARVEST_* -> SECRET_*_HARVEST

62b6972

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets November 16, 2021 14:31 Inactive

jrhizor temporarily deployed to more-secrets November 16, 2021 14:32 Inactive

jrhizor temporarily deployed to more-secrets November 16, 2021 15:00 Inactive

bugfix: remove SECRET_*_HARVEST we don't need for GSM

4ad0e18

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets November 16, 2021 15:18 Inactive

jrhizor temporarily deployed to more-secrets November 16, 2021 15:19 Inactive

antixar suggested changes Nov 16, 2021

View reviewed changes

airbyte-integrations/connectors/source-harvest/source_harvest/auth.py Show resolved Hide resolved

airbyte-integrations/connectors/source-harvest/source_harvest/auth.py Outdated Show resolved Hide resolved

jrhizor temporarily deployed to more-secrets November 19, 2021 14:04 Inactive

ChristopheDuong reviewed Nov 19, 2021

View reviewed changes

ChristopheDuong suggested changes Nov 19, 2021

View reviewed changes

ChristopheDuong reviewed Nov 19, 2021

View reviewed changes

ChristopheDuong mentioned this pull request Nov 19, 2021

Change UI to use new OAuth API/Protocol #7915

Closed

advancedAuth fixed

ac50b02

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets November 19, 2021 16:15 Inactive

grubberr requested review from ChristopheDuong and antixar November 19, 2021 16:15

ChristopheDuong approved these changes Nov 19, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets November 19, 2021 16:17 Inactive

ChristopheDuong reviewed Nov 19, 2021

View reviewed changes

ordering added

3b4b4d5

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr temporarily deployed to more-secrets November 19, 2021 18:06 Inactive

jrhizor temporarily deployed to more-secrets November 19, 2021 18:12 Inactive

source_specs.yaml updated

202013a

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr merged commit 778b0f2 into master Nov 19, 2021

grubberr deleted the grubberr/7707-source-harvest branch November 19, 2021 18:23

grubberr temporarily deployed to more-secrets November 19, 2021 18:24 Inactive

This was referenced Nov 19, 2021

Bump Airbyte version from 0.32.4-alpha to 0.32.5-alpha #8144

Closed

Bump Airbyte version from 0.32.4-alpha to 0.32.5-alpha #8147

Closed

Bump Airbyte version from 0.32.4-alpha to 0.32.5-alpha #8153

Merged

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🎉 Source Harvest: add OAuth 2.0 (airbytehq#7952)

200dd9d

* add OAuth 2.0 for source-harvest Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

🎉 Source Harvest: add OAuth 2.0 #7952

🎉 Source Harvest: add OAuth 2.0 #7952

Uh oh!

grubberr commented Nov 14, 2021 •

edited

Loading

grubberr commented Nov 16, 2021 •

edited by github-actions bot

Loading

grubberr commented Nov 16, 2021 •

edited by github-actions bot

Loading

grubberr commented Nov 16, 2021 •

edited by github-actions bot

Loading

grubberr commented Nov 16, 2021 •

edited by github-actions bot

Loading

antixar left a comment

Uh oh!

Uh oh!

grubberr commented Nov 19, 2021 •

edited by github-actions bot

Loading

ChristopheDuong Nov 19, 2021

grubberr Nov 19, 2021

ChristopheDuong Nov 19, 2021

grubberr Nov 19, 2021

ChristopheDuong Nov 19, 2021

grubberr Nov 19, 2021

ChristopheDuong Nov 19, 2021

grubberr Nov 19, 2021

ChristopheDuong Nov 19, 2021

grubberr Nov 19, 2021

ChristopheDuong left a comment •

edited

Loading

ChristopheDuong Nov 19, 2021

grubberr Nov 19, 2021

grubberr commented Nov 19, 2021 •

edited by github-actions bot

Loading

ChristopheDuong Nov 19, 2021

grubberr Nov 19, 2021

grubberr commented Nov 19, 2021 •

edited by github-actions bot

Loading

Labels

6 participants

	"path_in_connector_config": ["credentials", 0, "refresh_token"]
	"path_in_connector_config": ["credentials", "refresh_token"]

		"oauth_config_specification": {
		"complete_oauth_output_specification": {

🎉 Source Harvest: add OAuth 2.0 #7952

🎉 Source Harvest: add OAuth 2.0 #7952

Uh oh!

Conversation

grubberr commented Nov 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

grubberr commented Nov 16, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

grubberr commented Nov 16, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

grubberr commented Nov 16, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

grubberr commented Nov 16, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

antixar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

grubberr commented Nov 19, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopheDuong left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grubberr commented Nov 19, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grubberr commented Nov 19, 2021 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Labels

6 participants

grubberr commented Nov 14, 2021 •

edited

Loading

grubberr commented Nov 16, 2021 •

edited by github-actions bot

Loading

grubberr commented Nov 16, 2021 •

edited by github-actions bot

Loading

grubberr commented Nov 16, 2021 •

edited by github-actions bot

Loading

grubberr commented Nov 16, 2021 •

edited by github-actions bot

Loading

grubberr commented Nov 19, 2021 •

edited by github-actions bot

Loading

ChristopheDuong left a comment •

edited

Loading

grubberr commented Nov 19, 2021 •

edited by github-actions bot

Loading

grubberr commented Nov 19, 2021 •

edited by github-actions bot

Loading