🎉 New Source: Freshsales

[tuanchris]

What

• Create a new source for Freshales. API document here
• Participating in hacktoberfest-2021, issue no 33

How

Describe the solution

Connection ran successfully in local env.

Stream settings

Local JSON files

Example output

Recommended reading order

1. x.java
2. y.python

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/SUMMARY.md
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions
• Connector added to connector index like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions
• Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• Credentials added to Github CI. Instructions.
• /test connector=connectors/<name> command is passing.
• New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed.

[CLAassistant]

All committers have signed the CLA.

[marcosmarxm]

@tuanchris any possibility to run a sync with a postgres destinaiton and basic normalization on? =)

[tuanchris]

@marcosmarxm I actually did that with BigQuery, hope that it is fine too. All seems to be working well but this.

2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 6 of 10 OK created table model freshsales.upcoming_appointments.............................................. [�[32mCREATE TABLE (2.0 rows, 1.3 KB processed)�[0m in 3.54s] 2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 3 of 10 OK created table model freshsales.lost_deals......................................................... [�[32mCREATE TABLE (1.0 rows, 2.2 KB processed)�[0m in 3.67s] 2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 5 of 10 OK created table model freshsales.open_deals......................................................... [�[32mCREATE TABLE (1.0 rows, 2.2 KB processed)�[0m in 3.68s] 2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 8 of 10 OK created table model freshsales.won_deals.......................................................... [�[32mCREATE TABLE (1.0 rows, 2.2 KB processed)�[0m in 3.79s] 2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 4 of 10 OK created table model freshsales.past_appointments.................................................. [�[32mCREATE TABLE (2.0 rows, 1.3 KB processed)�[0m in 3.90s] 2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 10 of 10 OK created table model freshsales.accounts.......................................................... [�[32mCREATE TABLE (1.0 rows, 1.5 KB processed)�[0m in 3.70s] 2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 2 of 10 OK created table model freshsales.contacts........................................................... [�[32mCREATE TABLE (100.0 rows, 208.1 KB processed)�[0m in 4.00s] 2021-10-13 16:53:02 INFO () LineGobbler(voidCall):65 - 16:53:02 | 1 of 10 ERROR creating table model freshsales.completed_tasks................................................ [�[31mERROR�[0m in 4.04s] 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 16:53:03 | 7 of 10 OK created table model freshsales.sales_activities................................................... [�[32mCREATE TABLE (0.0 rows, 0.0 Bytes processed)�[0m in 4.11s] 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 16:53:03 | 9 of 10 ERROR creating table model freshsales.open_tasks..................................................... [�[31mERROR�[0m in 3.99s] 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 16:53:03 | 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 16:53:03 | Finished running 10 table models in 6.65s. 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - �[31mCompleted with 2 errors and 0 warnings:�[0m 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - �[33mDatabase Error in model completed_tasks (models/generated/airbyte_tables/freshsales/completed_tasks.sql)�[0m 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - Bad bool value: 1 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/freshsales/completed_tasks.sql 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - �[33mDatabase Error in model open_tasks (models/generated/airbyte_tables/freshsales/open_tasks.sql)�[0m 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - Bad bool value: 0 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_tables/freshsales/open_tasks.sql 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - 2021-10-13 16:53:03 INFO () LineGobbler(voidCall):65 - Done. PASS=8 WARN=0 ERROR=2 SKIP=0 TOTAL=10 

Should I change the boolean schema to integer or what is the preferred way of dealing with this?

[marcosmarxm]

Some comments, but overall looks good! I'll set up this locally and test. Can you check the comments?

[tuanchris]

@marcosmarxm Running all good locally with basic normalization to BigQuery now

[marcosmarxm]

thanks @tuanchris going to review and test the connector later today.

[marcosmarxm]

Hi @tuanchris the unit tests are not passing, can you check them?
The integration tests work great! So after correct then I think we can merge this contribution.

[marcosmarxm]

Unit tests are not passing.

================================================================================ short test summary info ================================================================================ FAILED unit_tests/test_source.py::test_check_connection - assert (False,\n ConnectionError(MaxRetryError('HTTPSConnectionPool(host="%3cmagicmock%20name=\'mock.__getitem__()\'%20id=\'... FAILED unit_tests/test_streams.py::test_request_params - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_next_page_token - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_parse_response - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_request_headers - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_http_method - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] - TypeError: __init__() missing 1 required positional argument: 'domain_name' FAILED unit_tests/test_streams.py::test_backoff_time - TypeError: __init__() missing 1 required positional argument: 'domain_name' 

[tuanchris]

all fixed @marcosmarxm

[marcosmarxm]

Unit tests are passing but Integration tests aren't.

=========================== short test summary info ============================ FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: Rec... FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0] =================== 2 failed, 10 passed, 1 skipped in 53.96s =================== 

Make sure ./gradlew airbyte-integrations:connectors:source-freshsales:integrationTest is working and share a screenshoot showing the result.

[tuanchris]

@marcosmarxm I'm stuck here, can you help take a look? Two problems that I'm facing is with the following message running integration test:

INFO detailed_logger acceptance_tests_logs/test_full_refresh.py__TestFullRefresh__test_sequential_reads[inputs0].txt:test_full_refresh.py:25 The two sequential reads should produce either equal set of records or one of them is a strict subset of the other 

So it seems that the results from Freshsales gave more columns than in the doc. I have consolidated all possible columns, change the type to ["null", "data_type"] but the error still persists.

Another warning that I'm seeing is like follow but I cannot debug it also.

WARNING root:connector_runner.py:112 Unable to parse connector's output 1 validation error for AirbyteMessage type field required (type=value_error.missing) 

[tuanchris]

@marcosmarxm Problem found. response was returning tokens, which changed with different requests. How do I remove this column? Removing it from the schema doesn't seem to help.

[marcosmarxm]

@tuanchris the problem is not the token data but the connector you built doesnt run a second full refresh properly.
All streams are Full Refresh | Overwrite. Can you check that?

The first sync: works and ingest data
Second: no data ingested.

[marcosmarxm]

Updating, looks maybe the Freshsales is throwing a Max call when trying to run the test. Could you try to add a sleep between record1 and record2 in accept tests?

[tuanchris]

@marcosmarxm I couldn't reproduce the failed attempt you show locally. Running two consecutive sync works for me

Furthermore, running the following test script for five minutes doesn't seem to throw any error:

auth = TokenAuthenticator(token=f'token={config["api_key"]}', auth_method="Token").get_auth_header() while True: url = f'https://{config["domain_name"]}/crm/sales/api/deals/view/17001600269' r = requests.get(filters_url, headers=auth) r.raise_for_status() print(r.json()) 

With the following config setting:

{ "streams": [ { "stream": { "name": "won_deals", "json_schema": {}, "supported_sync_modes": ["full_refresh"], "source_defined_primary_key": [["id"]] }, "sync_mode": "full_refresh", "destination_sync_mode": "overwrite" } ] } 

The integration test failed and I got this diff (only the token is different).

I still think that this is the culprit.

[tuanchris]

@marcosmarxm I removed the fc_widget_collaboration column and integration tests passed

[marcosmarxm]

thanks @tuanchris now integration test are working.