✨ source-file: prevent local file usage on cloud deployment mode #30984

pedroslopez · 2023-10-02T17:41:20Z

What

In order to remove source-file-secure, we need to integrate its funcionality to source-file. It uses the cdk helper introduced in #30980 to adapt the connector's behavior depending on the deployment mode.

close #30625

How

Integrate source-file-secure:

change spec behavior to remove local files on cloud deployment mode (logic from source-file-secure)
update runtime behavior to error based on deployment mode if local files are used (logic from source-file-secure)
add CAT tests based on deployment mode

vercel · 2023-10-02T17:41:23Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 16, 2023 5:49pm

pedroslopez · 2023-10-02T17:41:33Z

Current dependencies on/for this PR:

master
- PR ✨ source-file: prevent local file usage on cloud deployment mode #30984 👈

This comment was auto-generated by Graphite.

github-actions · 2023-10-02T17:41:41Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

airbyte-oss-build-runner · 2023-10-02T19:04:40Z

source-file test report (commit `bd49ade501`) - ❌

⏲️ Total pipeline duration: 20mn02s

Step	Result
Connector package install	❌
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-02T19:34:11Z

source-file test report (commit `bd49ade501`) - ❌

⏲️ Total pipeline duration: 20mn02s

Step	Result
Connector package install	❌
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-02T21:00:55Z

source-file test report (commit `da65ef3fa3`) - ❌

⏲️ Total pipeline duration: 20mn03s

Step	Result
Connector package install	❌
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-03T01:15:06Z

source-file test report (commit `da65ef3fa3`) - ❌

⏲️ Total pipeline duration: 20mn03s

Step	Result
Connector package install	❌
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-03T17:50:18Z

source-file test report (commit `f5627fb84a`) - ❌

⏲️ Total pipeline duration: 03mn20s

Step	Result
Connector package install	✅
Build source-file docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	❌
Acceptance tests	❌
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-03T18:20:00Z

source-file test report (commit `b2f972ce8c`) - ❌

⏲️ Total pipeline duration: 02mn37s

Step	Result
Connector package install	✅
Build source-file docker image for platform linux/x86_64	✅
Unit tests	✅
Integration tests	❌
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-ci/connectors/pipelines/pipelines/actions/environments.py

airbyte-integrations/connectors/source-file/setup.py

airbyte-oss-build-runner · 2023-10-06T22:38:48Z

source-file test report (commit `d2a9f741d6`) - ✅

⏲️ Total pipeline duration: 03mn29s

Step	Result
Connector package install	✅
Build source-file docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

erohmensing · 2023-10-06T22:47:08Z

@pedroslopez would it be doable to graphite split this into to prs, one which fixes the tests and one which does the migration to deployment_mode=cloud?

bnchrch

bnchrch · 2023-10-09T16:17:38Z

airbyte-integrations/connectors/source-file/source_file/source.py

+ for provider in spec.connectionSpecification["properties"]["provider"]["oneOf"]:
+ storage = provider["properties"]["storage"]
+
+ if "enum" in storage:


📚 comments as to what we are doing could be useful

particularly for

storage["const"] = storage.pop("default")

Good catch - I tried to write a comment but realized I didn't understand what this was doing (I copied it from source-file-secure). Seems like when it was introduced, the spec for source-file had enums everywhere and it should have been consts and for some reason it was patched here for source-file-secure. This is no longer the case and source-file now properly declares these as const, so this is not needed.

Removed the code!

airbyte-oss-build-runner · 2023-10-14T00:52:45Z

source-file test report (commit `2ffc7ab084`) - ❌

⏲️ Total pipeline duration: 03mn46s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-14T00:56:51Z

source-file test report (commit `4d995df4b5`) - ✅

⏲️ Total pipeline duration: 02mn27s

Step	Result
Build source-file docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-16T17:22:57Z

source-file test report (commit `240ffe35e1`) - ❌

⏲️ Total pipeline duration: 01mn25s

Step	Result
Build source-file docker image for platform(s) linux/x86_64	✅
Unit tests	❌
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-16T17:30:37Z

source-file test report (commit `240ffe35e1`) - ✅

⏲️ Total pipeline duration: 02mn39s

Step	Result
Build source-file docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

airbyte-oss-build-runner · 2023-10-16T17:53:18Z

source-file test report (commit `daa435b8de`) - ✅

⏲️ Total pipeline duration: 03mn21s

Step	Result
Build source-file docker image for platform(s) linux/x86_64	✅
Unit tests	✅
Integration tests	✅
Acceptance tests	✅
Code format checks	✅
Validate metadata for source-file	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-file test

…bytehq#30984) Co-authored-by: pedroslopez <pedroslopez@users.noreply.github.com>

pedroslopez mentioned this pull request Oct 2, 2023

CAT: add option to test with specific deployment_mode #30838

Merged

octavia-squidington-iii added the area/connectors Connector related issues label Oct 2, 2023

octavia-squidington-iii added the connectors/source/file label Oct 2, 2023

pedroslopez force-pushed the pedro/cat-deployment-mode branch from 6d6de39 to 1450bf3 Compare October 2, 2023 18:15

pedroslopez force-pushed the pedro/source-file-dep-mode branch from 0b3b8da to bd49ade Compare October 2, 2023 18:15

pedroslopez marked this pull request as ready for review October 2, 2023 18:43

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Oct 2, 2023

vercel bot deployed to Preview October 2, 2023 20:40 View deployment

Base automatically changed from pedro/cat-deployment-mode to master October 3, 2023 19:47

pedroslopez force-pushed the pedro/source-file-dep-mode branch from b2f972c to 481e531 Compare October 6, 2023 22:29

pedroslopez commented Oct 6, 2023

View reviewed changes

pedroslopez changed the title ~~add source-file-secure logic to source-file~~ ✨ source-file: add source-file-secure logic to source-file Oct 6, 2023

pedroslopez changed the title ~~✨ source-file: add source-file-secure logic to source-file~~ ✨ source-file: integrate source-file-secure logic Oct 6, 2023

pedroslopez changed the title ~~✨ source-file: integrate source-file-secure logic~~ ✨ source-file: prevent local file usage on cloud deployment mode Oct 6, 2023

pedroslopez requested review from a team, alafanechere and bnchrch October 6, 2023 22:42

pedroslopez changed the base branch from master to pedro/fix-source-file-test October 6, 2023 22:58

pedroslopez force-pushed the pedro/source-file-dep-mode branch from d2a9f74 to 57af86a Compare October 6, 2023 22:58

bnchrch approved these changes Oct 9, 2023

View reviewed changes

pedroslopez force-pushed the pedro/source-file-dep-mode branch from 57af86a to 2ffc7ab Compare October 14, 2023 00:46

pedroslopez requested review from a team as code owners October 14, 2023 00:46

pedroslopez requested a review from a team October 14, 2023 00:46

pedroslopez changed the base branch from pedro/fix-source-file-test to master October 14, 2023 00:46

pedroslopez force-pushed the pedro/source-file-dep-mode branch from 2ffc7ab to 4d995df Compare October 14, 2023 00:52

pedroslopez and others added 7 commits October 16, 2023 10:19

add source-file-secure logic to source-file

0eb4b11

acc tests

e78589a

Automated Commit - Formatting Changes

5979873

bump cdk

c1c2665

bump version

282afd6

rm the thing again

d1eba9a

this is no longer needed

240ffe3

pedroslopez force-pushed the pedro/source-file-dep-mode branch from 4d995df to 240ffe3 Compare October 16, 2023 17:19

pedroslopez enabled auto-merge (squash) October 16, 2023 17:39

pedroslopez disabled auto-merge October 16, 2023 17:46

rm override

daa435b

pedroslopez merged commit d4ef599 into master Oct 16, 2023

pedroslopez deleted the pedro/source-file-dep-mode branch October 16, 2023 20:00

pedroslopez mentioned this pull request Oct 16, 2023

🧹 remove source-file-secure #31477

Merged

ariesgun pushed a commit to ariesgun/airbyte that referenced this pull request Oct 23, 2023

✨ source-file: prevent local file usage on cloud deployment mode (air…

f0182d4

…bytehq#30984) Co-authored-by: pedroslopez <pedroslopez@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

✨ source-file: prevent local file usage on cloud deployment mode #30984

✨ source-file: prevent local file usage on cloud deployment mode #30984

Uh oh!

pedroslopez commented Oct 2, 2023 •

edited

Loading

vercel bot commented Oct 2, 2023 •

edited

Loading

pedroslopez commented Oct 2, 2023 •

edited

Loading

github-actions bot commented Oct 2, 2023 •

edited by pedroslopez

Loading

airbyte-oss-build-runner commented Oct 2, 2023

airbyte-oss-build-runner commented Oct 2, 2023

airbyte-oss-build-runner commented Oct 2, 2023

airbyte-oss-build-runner commented Oct 3, 2023

airbyte-oss-build-runner commented Oct 3, 2023

airbyte-oss-build-runner commented Oct 3, 2023

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

airbyte-oss-build-runner commented Oct 6, 2023

erohmensing commented Oct 6, 2023

bnchrch left a comment

bnchrch Oct 9, 2023

bnchrch Oct 9, 2023

pedroslopez Oct 16, 2023

airbyte-oss-build-runner commented Oct 14, 2023

airbyte-oss-build-runner commented Oct 14, 2023

airbyte-oss-build-runner commented Oct 16, 2023

airbyte-oss-build-runner commented Oct 16, 2023

airbyte-oss-build-runner commented Oct 16, 2023

Labels

6 participants

✨ source-file: prevent local file usage on cloud deployment mode #30984

✨ source-file: prevent local file usage on cloud deployment mode #30984

Uh oh!

Conversation

pedroslopez commented Oct 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

vercel bot commented Oct 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pedroslopez commented Oct 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Oct 2, 2023 • edited by pedroslopez Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

airbyte-oss-build-runner commented Oct 2, 2023

source-file test report (commit bd49ade501) - ❌

airbyte-oss-build-runner commented Oct 2, 2023

source-file test report (commit bd49ade501) - ❌

airbyte-oss-build-runner commented Oct 2, 2023

source-file test report (commit da65ef3fa3) - ❌

airbyte-oss-build-runner commented Oct 3, 2023

source-file test report (commit da65ef3fa3) - ❌

airbyte-oss-build-runner commented Oct 3, 2023

source-file test report (commit f5627fb84a) - ❌

airbyte-oss-build-runner commented Oct 3, 2023

source-file test report (commit b2f972ce8c) - ❌

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

airbyte-oss-build-runner commented Oct 6, 2023

source-file test report (commit d2a9f741d6) - ✅

erohmensing commented Oct 6, 2023

bnchrch left a comment

Choose a reason for hiding this comment

bnchrch Oct 9, 2023

Choose a reason for hiding this comment

bnchrch Oct 9, 2023

Choose a reason for hiding this comment

pedroslopez Oct 16, 2023

Choose a reason for hiding this comment

airbyte-oss-build-runner commented Oct 14, 2023

source-file test report (commit 2ffc7ab084) - ❌

airbyte-oss-build-runner commented Oct 14, 2023

source-file test report (commit 4d995df4b5) - ✅

airbyte-oss-build-runner commented Oct 16, 2023

source-file test report (commit 240ffe35e1) - ❌

airbyte-oss-build-runner commented Oct 16, 2023

source-file test report (commit 240ffe35e1) - ✅

airbyte-oss-build-runner commented Oct 16, 2023

source-file test report (commit daa435b8de) - ✅

Labels

6 participants

pedroslopez commented Oct 2, 2023 •

edited

Loading

vercel bot commented Oct 2, 2023 •

edited

Loading

pedroslopez commented Oct 2, 2023 •

edited

Loading

github-actions bot commented Oct 2, 2023 •

edited by pedroslopez

Loading

source-file test report (commit `bd49ade501`) - ❌

source-file test report (commit `bd49ade501`) - ❌

source-file test report (commit `da65ef3fa3`) - ❌

source-file test report (commit `da65ef3fa3`) - ❌

source-file test report (commit `f5627fb84a`) - ❌

source-file test report (commit `b2f972ce8c`) - ❌

source-file test report (commit `d2a9f741d6`) - ✅

source-file test report (commit `2ffc7ab084`) - ❌

source-file test report (commit `4d995df4b5`) - ✅

source-file test report (commit `240ffe35e1`) - ❌

source-file test report (commit `240ffe35e1`) - ✅

source-file test report (commit `daa435b8de`) - ✅