Source Pinterest - Adding missing columns for analytics streams #23457

artusiep · 2023-02-25T00:06:38Z

What

Adding missing metrics for pinterest source

Not all columns are available for synchronous endpoint so I have added all available columns:

- ADVERTISER_ID | #21326 - CLICKTHROUGH_1 | #22206 and #21326 - CLICKTHROUGH_1_GROSS | #21326 - CLICKTHROUGH_2 | #22206 and #21326 - ENGAGEMENT_1 | #22206 and #21326 - ENGAGEMENT_2 | #22206 and #21326 - PIN_ID | #21326 - PIN_PROMOTION_ID | #21326 - REPIN_1 | #21326 - REPIN_2 | #21326 - TOTAL_CLICK_ADD_TO_CART | #22206 and #21326 - TOTAL_CLICK_LEAD | #21326 - TOTAL_CUSTOM | #21326 - TOTAL_ENGAGEMENT | #21326 - TOTAL_ENGAGEMENT_LEAD | #21326 - TOTAL_LEAD | #21326 - TOTAL_OFFLINE_CHECKOUT | #21326 - TOTAL_VIEW_ADD_TO_CART | #21326 - TOTAL_VIEW_LEAD | #21326 - TOTAL_WEB_SESSIONS | #21326 - VIDEO_LENGTH | #21326 - WEB_SESSIONS_1 | #21326 - WEB_SESSIONS_2 | #21326 - VIDEO_P0_COMBINED_1 | Not Available in Synchronous API call - VIDEO_P100_COMPLETE_1 | Not Available in Synchronous API call - VIDEO_P25_COMBINED_1 | Not Available in Synchronous API call - VIDEO_P50_COMBINED_1 | Not Available in Synchronous API call - VIDEO_P75_COMBINED_1 | Not Available in Synchronous API call - VIDEO_P95_COMBINED_1 | Not Available in Synchronous API call - VIDEO_3SEC_VIEWS_1 | Not Available in Synchronous API call - VIDEO_MRC_VIEWS_1 | Not Available in Synchronous API call - APP_INSTALL_COST_PER_ACTION | Not Available in Synchronous API call

This contribution brings 22 columns to analytics columns.

windows_in_days parameter is not used anywhere, and it is not configurable so deleting that from conftest.

Unit test:

Closes #22206 #21326
Duplicates: [https://github.com//pull/22501]

How

Adding columns and making list of columns easier to review

🚨 User Impact 🚨

Pinterest source users would need to refresh source schema to pickup changes

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

artusiep · 2023-02-25T00:28:51Z

Tagging @sh4sh, @natalyjazzviolin

artusiep · 2023-02-25T00:42:21Z

Acceptance tests:

I do not really know how to make this change backward compatibile

artusiep · 2023-02-25T00:47:23Z

Tested with Airbyte docker with normalization

sajarin

Hey @artusiep,

Thanks for the contribution, it looks like you're adding around 22 new columns to the analytics streams.

In the conftest.py, what's the reason behind removing the "windows_in_days" value?

Also there seems to be some merge conflicts, can you fix them?

artusiep · 2023-03-08T11:26:38Z

Hey @sajarin

answering your question

Hey @artusiep,

Thanks for the contribution, it looks like you're adding around 22 new columns to the analytics streams.

Exactly this is the main purpose of this contribution. The list of columns are mentioned in Pull Request description

In the conftest.py, what's the reason behind removing the "windows_in_days" value?

windows_in_days parameter is simply not used anywhere, and it is not configurable. Take a look at: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-pinterest/source_pinterest/source.py#L44-L46 I just wanted do make a clean up of this variable as well

Also there seems to be some merge conflicts, can you fix them?

Merge conflict fixed

sajarin · 2023-03-13T15:53:24Z

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/4406909764
❌ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/4406909764
🐛 https://gradle.com/s/uyjo523c2lip6

Build Failed

Test summary info:

=========================== short test summary info ============================ FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - co... FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs1] - co... FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Stream board... SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical. ============= 3 failed, 43 passed, 1 skipped in 217.33s (0:03:37) ==============

artusiep · 2023-03-14T08:36:16Z

@sajarin I think the backward compatibility error is connected with adding these columns and changing the type of columns.
Changed type because it was failing on dbt normalization in BigQuery:
https://github.com/airbytehq/airbyte/pull/23457/files#diff-65c6cd00462455b47cc1ddbe4c50322833e2d6cb2c6c4997012d1c9d2c8f3b12L33-L35
https://github.com/airbytehq/airbyte/pull/23457/files#diff-254365ad0710b775423057cada068e509205a6a37317b9da247e7f6246893977L15-L20
https://github.com/airbytehq/airbyte/pull/23457/files#diff-a8cf2c89813a5bbf6f0ba32e0c2107c7630f38fbd7d86e53159f235b2423ceadL15-L17

marcosmarxm · 2023-03-17T14:16:03Z

airbyte-integrations/connectors/source-pinterest/integration_tests/sample_config.json

 "client_secret": "client_secret",
 "refresh_token": "refresh_token",
- "start_date": "2020-06-01",
- "window_in_days": 1


Why are you removing this parameter?

As it is mentioned above:

In the conftest.py, what's the reason behind removing the "windows_in_days" value?

windows_in_days parameter is simply not used anywhere, and it is not configurable. Take a look at: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-pinterest/source_pinterest/source.py#L44-L46 I just wanted do make a clean up of this variable as well

I can rollback that but it is not going to be used

marcosmarxm · 2023-03-17T14:16:39Z

...-integrations/connectors/source-pinterest/source_pinterest/schemas/ad_account_analytics.json

 },
 "CAMPAIGN_NAME": {
- "type": ["null", "number"]
+ "type": ["null", "string"]


Changing an existing schame you must edit the acceptance-test to skip the backward compatibility issue.

I will try to understand how this backward compatibility acceptance tests works in order to mitigate that

Hello @artusiep sorry the long delay here. I submitted a few changes to correct integration tests.

marcosmarxm · 2023-04-01T14:11:37Z

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/4583595799
✅ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/4583595799
Python tests coverage:

Name Stmts Miss Cover -------------------------------------------------- source_pinterest/utils.py 8 0 100% source_pinterest/__init__.py 2 0 100% source_pinterest/source.py 224 18 92% -------------------------------------------------- TOTAL 234 18 92%

Build Passed

Test summary info:

=========================== short test summary info ============================ SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical. SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:587: Backward compatibility tests are disabled for version 0.2.3. ================== 47 passed, 3 skipped in 208.23s (0:03:28) ===================

marcosmarxm

Thanks @artusiep I'll request the final approval from the connector team to release the new version of your changes. This is going to happens next week but everything is ok from your side. Sorry again for the long delay to review the contribution.

lazebnyi

LGTM! Need update strictness level for tests

airbyte-integrations/connectors/source-pinterest/acceptance-test-config.yml

marcosmarxm · 2023-04-03T16:24:15Z

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/4599028209
✅ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/4599028209
Python tests coverage:

Name Stmts Miss Cover -------------------------------------------------- source_pinterest/utils.py 8 0 100% source_pinterest/__init__.py 2 0 100% source_pinterest/source.py 224 18 92% -------------------------------------------------- TOTAL 234 18 92%

Build Passed

Test summary info:

=========================== short test summary info ============================ SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical. SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:587: Backward compatibility tests are disabled for version 0.2.3. ================== 47 passed, 3 skipped in 212.08s (0:03:32) ===================

lazebnyi

LGTM!

marcosmarxm · 2023-04-03T17:31:44Z

/publish connector=connectors/source-pinterest

🕑 Publishing the following connectors:
connectors/source-pinterest
https://github.com/airbytehq/airbyte/actions/runs/4599612994

Connector	Did it publish?	Were definitions generated?
connectors/source-pinterest	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

octavia-squidington-iii added area/connectors Connector related issues bounty community connectors/source/pinterest labels Feb 25, 2023

artusiep changed the title ~~Adding missing metrics for pinterest source~~ Adding missing columns for analytics streams for pinterest source Feb 25, 2023

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Feb 25, 2023

sajarin changed the title ~~Adding missing columns for analytics streams for pinterest source~~ Source Pinterest - Adding missing columns for analytics streams Mar 7, 2023

sajarin reviewed Mar 7, 2023

View reviewed changes

artusiep added 2 commits March 8, 2023 12:16

Adding missing metrics for pinterest source

6da6e84

Fixing formatting

779417c

artusiep force-pushed the 22206-extend-metrics-in-anatytics-stream branch from 96639a3 to df908ac Compare March 8, 2023 11:18

Adding new version to docs

e6a8ef9

artusiep force-pushed the 22206-extend-metrics-in-anatytics-stream branch from df908ac to e6a8ef9 Compare March 8, 2023 11:20

artusiep requested a review from sajarin March 8, 2023 18:16

Merge branch 'master' into 22206-extend-metrics-in-anatytics-stream

c6a8e1f

marcosmarxm suggested changes Mar 17, 2023

View reviewed changes

marcosmarxm added the contributor-program PRs submitted through the contributor program. label Mar 17, 2023

marcosmarxm requested a review from a team March 17, 2023 14:19

marcosmarxm added 2 commits April 1, 2023 11:07

update schemas

05d5d3e

merge master

8e88003

marcosmarxm added 2 commits April 1, 2023 11:29

remove minimal config catalog for tests

c9f25cd

removed expected file

4749b9c

marcosmarxm approved these changes Apr 1, 2023

View reviewed changes

marcosmarxm requested a review from lazebnyi April 3, 2023 13:51

lazebnyi approved these changes Apr 3, 2023

View reviewed changes

lazebnyi suggested changes Apr 3, 2023

View reviewed changes

airbyte-integrations/connectors/source-pinterest/acceptance-test-config.yml Outdated Show resolved Hide resolved

rollback test strictness level to high

cbc0753

marcosmarxm requested a review from lazebnyi April 3, 2023 16:54

lazebnyi approved these changes Apr 3, 2023

View reviewed changes

auto-bump connector version

d651f48

marcosmarxm merged commit 63850e5 into airbytehq:master Apr 3, 2023

marcosmarxm mentioned this pull request Apr 3, 2023

Source: Pinterest - Missing Fields from ad_analytics stream #21326

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Source Pinterest - Adding missing columns for analytics streams #23457

Source Pinterest - Adding missing columns for analytics streams #23457

Uh oh!

artusiep commented Feb 25, 2023 •

edited

Loading

artusiep commented Feb 25, 2023

artusiep commented Feb 25, 2023

artusiep commented Feb 25, 2023 •

edited

Loading

sajarin left a comment •

edited

Loading

artusiep commented Mar 8, 2023 •

edited

Loading

sajarin commented Mar 13, 2023 •

edited by github-actions bot

Loading

artusiep commented Mar 14, 2023

marcosmarxm Mar 17, 2023

artusiep Mar 17, 2023

marcosmarxm Mar 17, 2023

artusiep Mar 19, 2023

marcosmarxm Apr 1, 2023

marcosmarxm commented Apr 1, 2023 •

edited by github-actions bot

Loading

marcosmarxm left a comment

lazebnyi left a comment •

edited

Loading

Uh oh!

marcosmarxm commented Apr 3, 2023 •

edited by github-actions bot

Loading

lazebnyi left a comment

marcosmarxm commented Apr 3, 2023 •

edited by github-actions bot

Loading

Labels

5 participants

Source Pinterest - Adding missing columns for analytics streams #23457

Source Pinterest - Adding missing columns for analytics streams #23457

Uh oh!

Conversation

artusiep commented Feb 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

artusiep commented Feb 25, 2023

artusiep commented Feb 25, 2023

artusiep commented Feb 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

sajarin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

artusiep commented Mar 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

sajarin commented Mar 13, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Failed

artusiep commented Mar 14, 2023

marcosmarxm Mar 17, 2023

Choose a reason for hiding this comment

artusiep Mar 17, 2023

Choose a reason for hiding this comment

marcosmarxm Mar 17, 2023

Choose a reason for hiding this comment

artusiep Mar 19, 2023

Choose a reason for hiding this comment

marcosmarxm Apr 1, 2023

Choose a reason for hiding this comment

marcosmarxm commented Apr 1, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

marcosmarxm left a comment

Choose a reason for hiding this comment

lazebnyi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marcosmarxm commented Apr 3, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

lazebnyi left a comment

Choose a reason for hiding this comment

marcosmarxm commented Apr 3, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Labels

5 participants

artusiep commented Feb 25, 2023 •

edited

Loading

artusiep commented Feb 25, 2023 •

edited

Loading

sajarin left a comment •

edited

Loading

artusiep commented Mar 8, 2023 •

edited

Loading

sajarin commented Mar 13, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented Apr 1, 2023 •

edited by github-actions bot

Loading

lazebnyi left a comment •

edited

Loading

marcosmarxm commented Apr 3, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented Apr 3, 2023 •

edited by github-actions bot

Loading