Skip to content

Conversation

@monai
Copy link
Contributor

@monai monai commented Dec 14, 2021

What

This PR adds initial support for Flexport API. For now, it provides the following streams:

Recommended reading order

  1. source.py

Pre-merge Checklist

Unit tests
~/projects/lt/airbyte/airbyte-integrations/connectors/source-flexport (flexport) [um] python -m pytest -v unit_tests/ [.venv] Test session starts (platform: darwin, Python 3.9.7, pytest 6.1.2, pytest-sugar 0.9.4) cachedir: .pytest_cache rootdir: /Users/juozas/projects/lt/airbyte, configfile: pytest.ini plugins: cov-2.12.1, sugar-0.9.4, requests-mock-1.9.3, mock-3.6.1, timeout-1.4.2 collecting ... airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_cursor_field[Shipments-updated_at] ✓ 3% ▍ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_get_updated_state[current0-latest0-expected0] ✓ 5% ▌ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_get_updated_state[current1-latest1-expected1] ✓ 8% ▊ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_get_updated_state[current2-latest2-expected2] ✓ 11% █▏ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_get_updated_state[current3-latest3-expected3] ✓ 13% █▍ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_slices[None-None-24-2050-01-01T00:00:00+00:00-2050-01-02T00:00:01+00:00] ✓ 16% █▋ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_slices[2050-01-02T00:00:00+00:00-None-48-2050-01-01T00:00:00+00:00-2050-01-03T00:00:01+00:00] ✓ 18% █▉ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_slices[None-stream_state2-1-2050-01-04T00:00:00+00:00-2050-01-04T00:00:01+00:00] ✓ 21% ██▏ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_slices[2050-01-05T00:00:00+00:00-stream_state3-48-2050-01-04T00:00:00+00:00-2050-01-06T00:00:01+00:00] ✓ 24% ██▍ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_slices[2052-01-01T00:00:00+00:00-stream_state4-25-2050-01-01T00:00:00+00:00-2052-01-02T00:00:01+00:00] ✓ 26% ██▋ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_slices[2050-04-01T00:00:00+00:00-stream_state5-13-2050-01-01T00:00:00+00:00-2050-04-02T00:00:01+00:00] ✓ 29% ██▉ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_slices[2050-01-31T00:00:00+00:00-stream_state6-5-2050-01-01T00:00:00+00:00-2050-02-01T00:00:01+00:00] ✓ 32% ███▎ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_slices[2050-01-01T23:59:59+00:00-stream_state7-24-2050-01-01T00:00:00+00:00-2050-01-02T00:00:01+00:00] ✓ 34% ███▌ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_supports_incremental ✓ 37% ███▊ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_source_defined_cursor ✓ 39% ███▉ airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py::test_stream_checkpoint_interval ✓ 42% ████▎ airbyte-integrations/connectors/source-flexport/unit_tests/test_source.py::test_check_connection[200-response0-expected0] ✓ 45% ████▌ airbyte-integrations/connectors/source-flexport/unit_tests/test_source.py::test_check_connection[401-response1-expected1] ✓ 47% ████▊ airbyte-integrations/connectors/source-flexport/unit_tests/test_source.py::test_check_connection[401-response2-expected2] ✓ 50% █████ airbyte-integrations/connectors/source-flexport/unit_tests/test_source.py::test_streams ✓ 53% █████▍ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_request_params[None-expected0] ✓ 55% █████▌ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_request_params[next_page_token1-expected1] ✓ 58% █████▊ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_next_page_token[response0-None] ✓ 61% ██████▏ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_next_page_token[response1-expected1] ✓ 63% ██████▍ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_next_page_token[response2-expected2] ✓ 66% ██████▋ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_next_page_token[response3-expected3] ✓ 68% ██████▉ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[400-None-expected0] ✓ 71% ███████▏ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[401-string_response-expected1] ✓ 74% ███████▍ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[200-response2-expected2] ✓ 76% ███████▋ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[402-response3-expected3] ✓ 79% ███████▉ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[200-response4-expected4] ✓ 82% ████████▎ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[403-response5-expected5] ✓ 84% ████████▌ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[200-None-expected6] ✓ 87% ████████▊ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[200-string_response-expected7] ✓ 89% ████████▉ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[200-response8-expected8] ✓ 92% █████████▎ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[200-response9-expected9] ✓ 95% █████████▌ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[200-response10-expected10] ✓ 97% █████████▊ airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py::test_parse_response[200-response11-expected11] ✓ 100% ██████████ ================================================================================================================== warnings summary ================================================================================================================== airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py: 16 warnings airbyte-integrations/connectors/source-flexport/unit_tests/test_source.py: 5 warnings airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py: 18 warnings /Users/juozas/projects/lt/airbyte/airbyte-integrations/connectors/source-flexport/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:38: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20. self._authenticator = NoAuth() airbyte-integrations/connectors/source-flexport/unit_tests/test_incremental_streams.py: 16 warnings airbyte-integrations/connectors/source-flexport/unit_tests/test_source.py: 5 warnings airbyte-integrations/connectors/source-flexport/unit_tests/test_streams.py: 18 warnings /Users/juozas/projects/lt/airbyte/airbyte-integrations/connectors/source-flexport/.venv/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20. return old_new1(cls, *args, **kwargs) -- Docs: https://docs.pytest.org/en/stable/warnings.html Results (0.72s): 38 passed 
Acceptance tests
~/projects/lt/airbyte/airbyte-integrations/connectors/source-flexport (flexport) [um] python -m pytest -p integration_tests.acceptance [.venv] Test session starts (platform: darwin, Python 3.9.7, pytest 6.1.2, pytest-sugar 0.9.4) cachedir: .pytest_cache rootdir: /Users/juozas/projects/lt/airbyte, configfile: pytest.ini plugins: cov-2.12.1, sugar-0.9.4, requests-mock-1.9.3, mock-3.6.1, timeout-1.4.2 collecting ... airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓ 4% ▌ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓ 9% ▉ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓ 13% █▍ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓ 17% █▊ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓ 22% ██▎ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓ 26% ██▋ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓ 30% ███▏ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓ 35% ███▌ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓ 39% ███▉ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓ 43% ████▍ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓ 48% ████▊ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓ 52% █████▎ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓ 57% █████▋ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs1] ✓ 61% ██████▏ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs2] ✓ 65% ██████▌ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs3] ✓ 70% ██████▉ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs4] ✓ 74% ███████▍ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓ 78% ███████▉ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs1] ✓ 83% ████████▍ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs2] ✓ 87% ████████▊ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs3] ✓ 91% █████████▎ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓ 96% █████████▋ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] ✓ 100% ██████████ Results (159.41s): 23 passed 

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/SUMMARY.md
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the connector is published, connector added to connector index as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

This change is Reviewable

@github-actions github-actions bot added the area/connectors Connector related issues label Dec 14, 2021
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Dec 14, 2021
@marcosmarxm
Copy link
Contributor

Thanks for the contribution @monai. Please change from draft to ready to review when the code is finished!

@alafanechere
Copy link
Contributor

alafanechere commented Dec 15, 2021

Hi @monai,
Let me know if I'm wrong, but according to your streams schema you decided to mimic the API responses given by Flexport. The one-to-many relationship is described by links to locations resources in a company record, and vice versa for the location resource. These urls contain the id of the location, e.g (https://api.flexport.com/network/locations/123456), I'd extract the id from this url and add the id field to the collection schema. This will allow a user to join between companies and locations after syncing both streams, and it will not lead to redundant data in the streams.

I'm not sure if I answered exactly the question you asked.

@monai
Copy link
Contributor Author

monai commented Dec 22, 2021

@alafanechere the PR is ready for review.

Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @monai again for this clean contribution. I made some minor suggestions.
Can I also ask you to create a UUID for your source and create a file for it in .airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/, you can check other connector's file for example. We'd also need you to add your connector in airbyte-config/init/src/main/resources/seed/source_definitions.yaml.
As discussed on slack we're waiting to have a Flexport sandbox account to run this in our CI.

@alafanechere alafanechere added the waiting-for-cred Waiting for CI credentials label Dec 23, 2021
@monai monai requested a review from alafanechere December 27, 2021 13:46
@monai
Copy link
Contributor Author

monai commented Jan 3, 2022

@alafanechere I've addressed your last comment in 0f45987. The PR is ready for review again.

Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @monai for these changes. Could you please provide a screenshot of the acceptance test passing? We do not yet have a Flexport sandbox account that could allow us to run these on our side. It's why I labeled the PR as waiting-for-credentials.

@monai
Copy link
Contributor Author

monai commented Jan 4, 2022

Acceptance tests
~/projects/lt/airbyte/airbyte-integrations/connectors/source-flexport (flexport) [A1um] python -m pytest -p integration_tests.acceptance [.venv] Test session starts (platform: darwin, Python 3.9.7, pytest 6.1.2, pytest-sugar 0.9.4) cachedir: .pytest_cache rootdir: /Users/juozas/projects/lt/airbyte, configfile: pytest.ini plugins: cov-2.12.1, sugar-0.9.4, requests-mock-1.9.3, mock-3.6.1, timeout-1.4.2 collecting ... airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓ 4% ▌ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓ 9% ▉ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓ 13% █▍ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓ 17% █▊ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓ 22% ██▎ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓ 26% ██▋ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓ 30% ███▏ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓ 35% ███▌ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓ 39% ███▉ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓ 43% ████▍ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓ 48% ████▊ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓ 52% █████▎ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓ 57% █████▋ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs1] ✓ 61% ██████▏ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs2] ✓ 65% ██████▌ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs3] ✓ 70% ██████▉ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs4] ✓ 74% ███████▍ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓ 78% ███████▉ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs1] ✓ 83% ████████▍ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs2] ✓ 87% ████████▊ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs3] ✓ 91% █████████▎ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_two_sequential_reads[inputs0] ✓ 96% █████████▋ airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_incremental.py::TestIncremental.test_state_with_abnormally_large_values[inputs0] ✓ 100% ██████████ Results (177.12s): 23 passed ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ~/projects/lt/airbyte/airbyte-integrations/connectors/source-flexport (flexport) [um] git rev-parse HEAD [.venv] 880a117a7b986f3e493180ed492cd58bfd5e7c39 
@alafanechere
Copy link
Contributor

alafanechere commented Jan 19, 2022

@monai Flexport remains silent about the creation of a sandbox account for Airbyte. I suggest we merge the connector and not publish it. It will allow users to build the connector themselves and still have this connector in our repo, we'll publish it eventually if Flexport replies to our sandbox request.

@monai
Copy link
Contributor Author

monai commented Jan 20, 2022

I suggest we merge the connector and not publish it.

I agree.

Copy link
Contributor

@alafanechere alafanechere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll merge without publishing as we miss a Flexport sandbox account.

@alafanechere alafanechere merged commit 9afbbff into airbytehq:master Jan 20, 2022
@jrhizor
Copy link
Contributor

jrhizor commented Jan 20, 2022

@alafanechere We can't add a connector to airbyte-config/init/src/main/resources/seed/source_definitions.yaml without publishing or it causes build failures https://github.com/airbytehq/airbyte/runs/4883175759?check_suite_focus=true. Will revert just that line in a separate PR.

@monai
Copy link
Contributor Author

monai commented Mar 1, 2022

@alafanechere how's the communication with Flexport? Any progress?

@alafanechere
Copy link
Contributor

@monai, no reply from their side...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation community waiting-for-cred Waiting for CI credentials

5 participants