Skip to content

Conversation

@grubberr
Copy link
Contributor

@grubberr grubberr commented Sep 29, 2022

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

Try to fix https://github.com/airbytehq/oncall/issues/598

How

_get_reporting_service method is caching instance of "ReportingServiceManager" for uniq pair (customer_id, account_id)
If we have a lot of such uniq pairs we cache a lot of ReportingServiceManager objects which are pretty heavy.

Recommended reading order

  1. x.java
  2. y.python

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
@grubberr grubberr self-assigned this Sep 29, 2022
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Sep 29, 2022
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
@grubberr grubberr changed the title Source Bing Ads: limit cache for reporting_service Source Bing Ads: Fix: limit cache size for ReportingServiceManager instances Sep 29, 2022
@grubberr
Copy link
Contributor Author

grubberr commented Sep 29, 2022

/test connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3153041635
❌ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3153041635
🐛 https://gradle.com/s/pza7gbsvocwle

Build Failed

Test summary info:

=========================== short test summary info ============================ FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai... SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config ============= 1 failed, 26 passed, 1 skipped in 386.39s (0:06:26) ============== 
@grubberr
Copy link
Contributor Author

grubberr commented Sep 29, 2022

/test connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3153319080
❌ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3153319080
🐛 https://gradle.com/s/b7j3e5a3m7i5c

Build Failed

Test summary info:

=========================== short test summary info ============================ FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai... SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config ============= 1 failed, 26 passed, 1 skipped in 396.71s (0:06:36) ============== 
Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
@grubberr
Copy link
Contributor Author

grubberr commented Sep 29, 2022

/test connector=connectors/source-bing-ads

🕑 connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3153929303
✅ connectors/source-bing-ads https://github.com/airbytehq/airbyte/actions/runs/3153929303
Python tests coverage:

Name Stmts Miss Cover ------------------------------------------------- source_bing_ads/__init__.py 2 0 100% source_bing_ads/source.py 259 9 97% source_bing_ads/cache.py 18 1 94% source_bing_ads/reports.py 133 17 87% source_bing_ads/client.py 93 14 85% ------------------------------------------------- TOTAL 505 41 92% Name Stmts Miss Cover Missing ---------------------------------------------------------------------------------- source_acceptance_test/base.py 10 4 60% 15-18 source_acceptance_test/config.py 83 6 93% 78-80, 84-86 source_acceptance_test/conftest.py 164 164 0% 6-282 source_acceptance_test/plugin.py 48 48 0% 6-104 source_acceptance_test/tests/test_core.py 329 111 66% 39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577 source_acceptance_test/tests/test_full_refresh.py 52 2 96% 34, 65 source_acceptance_test/tests/test_incremental.py 152 26 83% 21-23, 29-31, 36-43, 48-61, 239, 250-258 source_acceptance_test/utils/asserts.py 37 2 95% 57-58 source_acceptance_test/utils/common.py 77 17 78% 15-16, 24-30, 47-54, 64, 67 source_acceptance_test/utils/compare.py 62 23 63% 21-51, 68, 97-99 source_acceptance_test/utils/connector_runner.py 112 50 55% 23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149 source_acceptance_test/utils/json_schema_helper.py 105 13 88% 30-31, 38, 41, 65-68, 96, 120, 190-192 ---------------------------------------------------------------------------------- TOTAL 1358 466 66% 

Build Passed

Test summary info:

=========================== short test summary info ============================ SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config ================== 27 passed, 1 skipped in 592.04s (0:09:52) =================== 
@grubberr
Copy link
Contributor Author

grubberr commented Sep 29, 2022

/publish connector=connectors/source-bing-ads

🕑 Publishing the following connectors:
connectors/source-bing-ads
https://github.com/airbytehq/airbyte/actions/runs/3154271631


Connector Did it publish? Were definitions generated?
connectors/source-bing-ads

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@grubberr grubberr merged commit 434833c into master Sep 29, 2022
@grubberr grubberr deleted the grubberr/oncall-598-source-bing-ads branch September 29, 2022 20:52
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
…stances (airbytehq#17403) Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/bing-ads

6 participants