update some connectors to use YAML specs #12230

pedroslopez · 2022-04-20T23:45:00Z

What

#11668 introduced yaml spec support, now let's actually use it!

How

Update 5 connectors to use a spec.yaml:

stripe
salesforce
hubspot
google sheets
exchange-rates

I basically ran the old spec.json files through https://codebeautify.org/json-to-yaml and made minor tweaks (mainly change single quotes to double quotes)

Also updated some references to the old files and bumped connector versions.
Doc updates coming soon w/ references to new spec.yaml files.

pedroslopez · 2022-04-20T23:53:46Z

/test connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2198945566
✅ connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2198945566
Python tests coverage:

Name Stmts Miss Cover ------------------------------------------------------------------------ source_acceptance_test/utils/__init__.py 6 0 100% source_acceptance_test/tests/__init__.py 4 0 100% source_acceptance_test/__init__.py 2 0 100% source_acceptance_test/tests/test_full_refresh.py 52 2 96% source_acceptance_test/utils/asserts.py 37 2 95% source_acceptance_test/config.py 74 6 92% source_acceptance_test/utils/json_schema_helper.py 105 13 88% source_acceptance_test/utils/common.py 70 17 76% source_acceptance_test/utils/compare.py 62 23 63% source_acceptance_test/tests/test_core.py 285 106 63% source_acceptance_test/base.py 10 4 60% source_acceptance_test/utils/connector_runner.py 110 48 56% source_acceptance_test/tests/test_incremental.py 69 38 45% ------------------------------------------------------------------------ TOTAL 886 259 71% Name Stmts Miss Cover ----------------------------------------------- source_stripe/__init__.py 2 0 100% source_stripe/streams.py 235 66 72% source_stripe/source.py 22 11 50% ----------------------------------------------- TOTAL 259 77 70%

pedroslopez · 2022-04-20T23:54:12Z

/test connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2198946783
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2198946783
Python tests coverage:

Name Stmts Miss Cover ------------------------------------------------------------------------ source_acceptance_test/utils/__init__.py 6 0 100% source_acceptance_test/tests/__init__.py 4 0 100% source_acceptance_test/__init__.py 2 0 100% source_acceptance_test/tests/test_full_refresh.py 52 2 96% source_acceptance_test/utils/asserts.py 37 2 95% source_acceptance_test/config.py 74 6 92% source_acceptance_test/utils/json_schema_helper.py 105 13 88% source_acceptance_test/utils/common.py 70 17 76% source_acceptance_test/utils/compare.py 62 23 63% source_acceptance_test/tests/test_core.py 285 106 63% source_acceptance_test/base.py 10 4 60% source_acceptance_test/utils/connector_runner.py 110 48 56% source_acceptance_test/tests/test_incremental.py 69 38 45% ------------------------------------------------------------------------ TOTAL 886 259 71% Name Stmts Miss Cover -------------------------------------------------------- source_salesforce/__init__.py 2 0 100% source_salesforce/exceptions.py 8 1 88% source_salesforce/api.py 150 19 87% source_salesforce/streams.py 282 60 79% source_salesforce/rate_limiting.py 22 6 73% source_salesforce/source.py 75 33 56% source_salesforce/utils.py 8 7 12% -------------------------------------------------------- TOTAL 547 126 77% Name Stmts Miss Cover -------------------------------------------------------- source_salesforce/utils.py 8 0 100% source_salesforce/__init__.py 2 0 100% source_salesforce/source.py 75 6 92% source_salesforce/api.py 150 14 91% source_salesforce/exceptions.py 8 1 88% source_salesforce/rate_limiting.py 22 3 86% source_salesforce/streams.py 282 41 85% -------------------------------------------------------- TOTAL 547 65 88%

pedroslopez · 2022-04-21T00:15:32Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199015565
❌ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199015565
🐛 https://gradle.com/s/3qyv5nydwzd3g
Python short test summary info:

=========================== short test summary info ============================ FAILED test_core.py::TestSpec::test_oneof_usage[inputs0] - AssertionError: Th... =================== 1 failed, 26 passed in 118.06s (0:01:58) ===================

pedroslopez · 2022-04-21T00:23:22Z

/test connector=connectors/source-google-sheets

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199043935
✅ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199043935
Python tests coverage:

Name Stmts Miss Cover ------------------------------------------------------------------------ source_acceptance_test/utils/__init__.py 6 0 100% source_acceptance_test/tests/__init__.py 4 0 100% source_acceptance_test/__init__.py 2 0 100% source_acceptance_test/tests/test_full_refresh.py 52 2 96% source_acceptance_test/utils/asserts.py 37 2 95% source_acceptance_test/config.py 74 6 92% source_acceptance_test/utils/json_schema_helper.py 105 13 88% source_acceptance_test/utils/common.py 70 17 76% source_acceptance_test/utils/compare.py 62 23 63% source_acceptance_test/tests/test_core.py 285 106 63% source_acceptance_test/base.py 10 4 60% source_acceptance_test/utils/connector_runner.py 110 48 56% source_acceptance_test/tests/test_incremental.py 69 38 45% ------------------------------------------------------------------------ TOTAL 886 259 71% Name Stmts Miss Cover ----------------------------------------------------------------------- google_sheets_source/models/spreadsheet_values.py 12 0 100% google_sheets_source/models/spreadsheet.py 34 0 100% google_sheets_source/models/__init__.py 2 0 100% google_sheets_source/__init__.py 2 0 100% google_sheets_source/helpers.py 139 26 81% google_sheets_source/client.py 22 5 77% google_sheets_source/google_sheets_source.py 103 84 18% ----------------------------------------------------------------------- TOTAL 314 115 63%

Python short test summary info:

=========================== short test summary info ============================ SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config ================== 22 passed, 1 skipped in 121.90s (0:02:01) ===================

pedroslopez · 2022-04-21T00:43:38Z

/test connector=connectors/source-exchange-rates

🕑 connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199103908
✅ connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199103908
Python tests coverage:

Name Stmts Miss Cover ------------------------------------------------------------------------ source_acceptance_test/utils/__init__.py 6 0 100% source_acceptance_test/tests/__init__.py 4 0 100% source_acceptance_test/__init__.py 2 0 100% source_acceptance_test/tests/test_full_refresh.py 52 2 96% source_acceptance_test/utils/asserts.py 37 2 95% source_acceptance_test/config.py 74 6 92% source_acceptance_test/utils/json_schema_helper.py 105 13 88% source_acceptance_test/utils/common.py 70 17 76% source_acceptance_test/utils/compare.py 62 23 63% source_acceptance_test/tests/test_core.py 285 106 63% source_acceptance_test/base.py 10 4 60% source_acceptance_test/utils/connector_runner.py 110 48 56% source_acceptance_test/tests/test_incremental.py 69 38 45% ------------------------------------------------------------------------ TOTAL 886 259 71%

Python short test summary info:

=========================== short test summary info ============================ SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestFullRefresh.test_sequential_reads because not found in the config SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config ======================== 19 passed, 2 skipped in 16.33s ========================

pedroslopez · 2022-04-21T00:48:15Z

/test connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199117927
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199117927
Python tests coverage:

Name Stmts Miss Cover ------------------------------------------------------------------------ source_acceptance_test/utils/__init__.py 6 0 100% source_acceptance_test/tests/__init__.py 4 0 100% source_acceptance_test/__init__.py 2 0 100% source_acceptance_test/tests/test_full_refresh.py 52 2 96% source_acceptance_test/utils/asserts.py 37 2 95% source_acceptance_test/config.py 74 6 92% source_acceptance_test/utils/json_schema_helper.py 105 13 88% source_acceptance_test/utils/common.py 70 17 76% source_acceptance_test/utils/compare.py 62 23 63% source_acceptance_test/tests/test_core.py 285 106 63% source_acceptance_test/base.py 10 4 60% source_acceptance_test/utils/connector_runner.py 110 48 56% source_acceptance_test/tests/test_incremental.py 69 38 45% ------------------------------------------------------------------------ TOTAL 886 259 71% Name Stmts Miss Cover ------------------------------------------------ source_hubspot/errors.py 6 0 100% source_hubspot/__init__.py 2 0 100% source_hubspot/streams.py 707 56 92% source_hubspot/source.py 69 12 83% ------------------------------------------------ TOTAL 784 68 91%

pedroslopez · 2022-04-21T00:50:44Z

airbyte-integrations/connectors/source-stripe/setup.py

 packages=find_packages(),
 install_requires=MAIN_REQUIREMENTS,
- package_data={"": ["*.json", "schemas/*.json", "schemas/shared/*.json"]},
+ package_data={"": ["*.json", "*.yaml", "schemas/*.json", "schemas/shared/*.json"]},


note: not adding *.yaml here weirdly didn't seem to cause any issues 🤔

pedroslopez · 2022-04-21T00:51:08Z

airbyte-integrations/connectors/source-stripe/source_stripe/spec.yaml

@@ -0,0 +1,47 @@
+documentationUrl: https://docs.airbyte.io/integrations/sources/stripe
+connectionSpecification:
+ $schema: http://json-schema.org/draft-07/schema#


feels a little odd to be doing this in a yaml file, but I think it's still right?

pedroslopez · 2022-04-21T00:53:05Z

airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.yaml

+ type: string
+ title: Credentials Title
+ description: Name of the credentials set
+ const: OAuth Credentials


This is slightly different than the spec.json counterpart: previously default and enum were defined, but it looks like that's not necessary anymore and was causing test failures.

Yup! Thanks for the change

pedroslopez · 2022-04-21T01:05:24Z

airbyte-integrations/connectors/source-hubspot/setup.py


 MAIN_REQUIREMENTS = [
- "airbyte-cdk~=0.1.49",
+ "airbyte-cdk~=0.1.55",


Just pointing out that I noticed only hubspot has a patch version in the requirements. Others either don't have a version at all, or are set to ~=0.1

pedroslopez · 2022-04-21T01:45:48Z

/publish connector=connectors/source-stripe

🕑 connectors/source-stripe https://github.com/airbytehq/airbyte/actions/runs/2199283730
🚀 Successfully published connectors/source-stripe
❌ Couldn't auto-bump version for connectors/source-stripe

pedroslopez · 2022-04-21T01:46:20Z

/publish connector=connectors/source-salesforce

🕑 connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2199285286
🚀 Successfully published connectors/source-salesforce
🚀 Auto-bumped version for connectors/source-salesforce
✅ connectors/source-salesforce https://github.com/airbytehq/airbyte/actions/runs/2199285286

codecov · 2022-04-21T02:08:45Z

Codecov Report

❗ No coverage uploaded for pull request base (master@53799cb). Click here to learn what that means.
The diff coverage is n/a.

@@ Coverage Diff @@ ## master #12230 +/- ## ========================================= Coverage ? 82.93% ========================================= Files ? 21 Lines ? 1904 Branches ? 0 ========================================= Hits ? 1579 Misses ? 325 Partials ? 0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53799cb...16bc799. Read the comment docs.

pedroslopez · 2022-04-21T02:11:56Z

/publish connector=connectors/source-hubspot

🕑 connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199353483
🚀 Successfully published connectors/source-hubspot
🚀 Auto-bumped version for connectors/source-hubspot
✅ connectors/source-hubspot https://github.com/airbytehq/airbyte/actions/runs/2199353483

pedroslopez · 2022-04-21T02:13:07Z

/publish connector=connectors/source-google-sheets auto-bump-version=false

🕑 connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199356387
🚀 Successfully published connectors/source-google-sheets
✅ connectors/source-google-sheets https://github.com/airbytehq/airbyte/actions/runs/2199356387

pedroslopez · 2022-04-21T02:13:34Z

/publish connector=connectors/source-exchange-rates auto-bump-version=false

🕑 connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199357564
🚀 Successfully published connectors/source-exchange-rates
✅ connectors/source-exchange-rates https://github.com/airbytehq/airbyte/actions/runs/2199357564

* salesforce uses spec.yaml * stripe uses spec.yaml * revert unintended changes * bump stripe version * add yaml files to package_data * hubspot uses a spec.yaml * use double quotes in yaml files * google sheets uses spec.yaml * exhcange-rates uses spec.yaml * remove usage of default and enum in spec * bump hubspot cdk requirement * auto-bump connector version * auto-bump connector version * manually bump versions Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

pedroslopez added 2 commits April 20, 2022 19:42

salesforce uses spec.yaml

1ea0369

stripe uses spec.yaml

73e8efb

pedroslopez linked an issue Apr 20, 2022 that may be closed by this pull request

Update 5 connectors to use YAML spec #11891

Closed

github-actions bot added the area/connectors Connector related issues label Apr 20, 2022

pedroslopez added 2 commits April 20, 2022 19:48

revert unintended changes

a350121

bump stripe version

d7b3bd7

pedroslopez added 3 commits April 20, 2022 19:55

add yaml files to package_data

87e8609

hubspot uses a spec.yaml

f2cf0e6

use double quotes in yaml files

8f30a4e

google sheets uses spec.yaml

dd29c09

exhcange-rates uses spec.yaml

ff1d757

remove usage of default and enum in spec

741234b

pedroslopez commented Apr 21, 2022

View reviewed changes

bump hubspot cdk requirement

110e704

pedroslopez commented Apr 21, 2022

View reviewed changes

pedroslopez marked this pull request as ready for review April 21, 2022 01:05

pedroslopez requested review from Phlair and sherifnada April 21, 2022 01:07

sherifnada approved these changes Apr 21, 2022

View reviewed changes

auto-bump connector version

9e8a945

octavia-squidington-iii temporarily deployed to more-secrets April 21, 2022 02:06 Inactive

auto-bump connector version

e579adf

octavia-squidington-iii temporarily deployed to more-secrets April 21, 2022 02:28 Inactive

manually bump versions

16bc799

pedroslopez temporarily deployed to more-secrets April 21, 2022 02:37 Inactive

pedroslopez merged commit aa872d0 into master Apr 21, 2022

pedroslopez deleted the pedroslopez/yaml-spec-connectors branch April 21, 2022 02:59

octavia-squidington-iii mentioned this pull request Apr 21, 2022

Bump Airbyte version from 0.36.1-alpha to 0.36.2-alpha #12258

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

update some connectors to use YAML specs #12230

update some connectors to use YAML specs #12230

Uh oh!

pedroslopez commented Apr 20, 2022 •

edited

Loading

pedroslopez commented Apr 20, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 20, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez Apr 21, 2022

pedroslopez Apr 21, 2022

pedroslopez Apr 21, 2022

sherifnada Apr 21, 2022

pedroslopez Apr 21, 2022 •

edited

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

codecov bot commented Apr 21, 2022 •

edited

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

Labels

4 participants

update some connectors to use YAML specs #12230

update some connectors to use YAML specs #12230

Uh oh!

Conversation

pedroslopez commented Apr 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

pedroslopez commented Apr 20, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Apr 20, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez Apr 21, 2022

Choose a reason for hiding this comment

pedroslopez Apr 21, 2022

Choose a reason for hiding this comment

pedroslopez Apr 21, 2022

Choose a reason for hiding this comment

sherifnada Apr 21, 2022

Choose a reason for hiding this comment

pedroslopez Apr 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

codecov bot commented Apr 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Apr 21, 2022 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Labels

4 participants

pedroslopez commented Apr 20, 2022 •

edited

Loading

pedroslopez commented Apr 20, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 20, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez Apr 21, 2022 •

edited

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

codecov bot commented Apr 21, 2022 •

edited

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading

pedroslopez commented Apr 21, 2022 •

edited by github-actions bot

Loading