✨Source MailGun: Migrate Python CDK to Low-code CDK #29122

btkcodedev · 2023-08-05T08:07:46Z

What

Migrating Source Mailgun to Low-Code CDK
Closes #29120

How

Developed using (Configuration Based Source) low-code CDK

Tests

Integration & Acceptance

Full Test Results

🚨 User Impact 🚨

No breaking changes, migration to low-code

Migrate Connector: Source Mailgun

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md
- docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
- docs/integrations/README.md
- airbyte-integrations/builds.md
PR name follows PR naming conventions

github-actions · 2023-08-05T08:08:09Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

btkcodedev · 2023-08-05T09:13:59Z

Test results: Passed state:

btkcodedev · 2023-08-05T09:15:02Z

Waiting for #28471 for merging to update start_date

marcosmarxm · 2023-08-08T13:44:32Z

Waiting for #28471 for merging to update start_date

Go ahead and fix the start date directly here.

btkcodedev · 2023-08-08T13:46:54Z

@marcosmarxm It's Hitting with backward compatibility test errors, any suggestions?

airbyte-integrations/connectors/source-mailgun/acceptance-test-config.yml

…-config.yml Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

marcosmarxm

Left some comments, the connector is missing pagination.

marcosmarxm · 2023-08-11T14:45:02Z

airbyte-integrations/connectors/source-mailgun/metadata.yaml

+ enabled: false
+ cloud:
+ enabled: false


Why ar eyou changing this?

marcosmarxm · 2023-08-11T14:45:08Z

airbyte-integrations/connectors/source-mailgun/metadata.yaml

- enabled: true
- oss:
- enabled: true
+ releaseDate: TODO


Add a date here

marcosmarxm · 2023-08-11T14:47:00Z

airbyte-integrations/connectors/source-mailgun/source_mailgun/manifest.yaml

+ paginator:
+ type: NoPagination


Please add pagination as it was implemented before

airbyte/airbyte-integrations/connectors/source-mailgun/source_mailgun/source.py

Lines 42 to 49 in b78ad4e

def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, HttpUrl]]:

"""

:param response: the most recent response from the API

:return If there is another page in the result, a mapping (e.g: dict) containing information needed to query the next page in the response.

If there are no more pages in the result, return None.

"""

next_page: Optional[HttpUrl] = response.json().get("paging", {}).get("next")

return {"url": next_page} if next_page and self._pre_parse_response(response) else None

marcosmarxm · 2023-08-11T14:47:26Z

airbyte-integrations/connectors/source-mailgun/source_mailgun/manifest.yaml

+ stream_names:
+ - "domains"
+ - "events"
+


btkcodedev · 2023-08-12T18:25:28Z

All comments resolved
Test result passed state

btkcodedev · 2023-08-12T18:37:20Z

sajarin · 2023-08-14T19:10:47Z

@btkcodedev getting an error with the check command with the events stream:

{"type": "LOG", "log": {"level": "ERROR", "message": "Encountered an error trying to connect to stream events. Error: \n Traceback (most recent call last):\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/checks/check_stream.py\", line 42, in check_connection\n stream_is_available, reason = availability_strategy.check_availability(stream, logger, source)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/availability_strategy.py\", line 36, in check_availability\n stream_slice = get_first_stream_slice(stream)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/streams/utils/stream_helper.py\", line 21, in get_first_stream_slice\n stream.stream_slices(\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py\", line 126, in stream_slices\n return self.retriever.stream_slices()\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py\", line 348, in stream_slices\n return self.stream_slicer.stream_slices()\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py\", line 141, in stream_slices\n start_datetime = self._calculate_earliest_possible_value(self._select_best_end_datetime())\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/incremental/datetime_based_cursor.py\", line 146, in _calculate_earliest_possible_value\n earliest_possible_start_datetime = min(self.start_datetime.get_datetime(self.config), end_datetime)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/datetime/min_max_datetime.py\", line 58, in get_datetime\n time = self._parser.parse(str(self.datetime.eval(config, **additional_parameters)), datetime_format)\n File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/datetime/datetime_parser.py\", line 29, in parse\n parsed_datetime = datetime.datetime.strptime(str(date), format)\n File \"/usr/local/lib/python3.9/_strptime.py\", line 568, in _strptime_datetime\n tt, fraction, gmtoff_fraction = _strptime(data_string, format)\n File \"/usr/local/lib/python3.9/_strptime.py\", line 349, in _strptime\n raise ValueError(\"time data %r does not match format %r\" %\nValueError: time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'\n"}}

{"type": "LOG", "log": {"level": "ERROR", "message": "Check failed"}}

{"type": "CONNECTION_STATUS", "connectionStatus": {"status": "FAILED", "message": "\"Unable to connect to stream events - time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'\""}}

btkcodedev · 2023-08-15T01:31:52Z

@sajarin
Updated the start date for solving #27385,

Please update the secrets config to match the updated format (start_date: %Y-%m-%dT%H:%M:%SZ)
Ping via Slack (btkcodedev) for getting my creds if needed

sajarin · 2023-08-15T20:53:26Z

@btkcodedev thanks for the update, config validation is now passing but no records are being fetched. I pinged you on slack for credentials so we can get this moving.

changes addressed

sajarin · 2023-08-17T17:53:44Z

Thanks for the contribution @btkcodedev!

padala · 2023-08-21T04:01:34Z

@btkcodedev @sajarin Thanks for the contribution. I'm still facing the issue with sync resulting in 0 records when I tried this today on Airbyte Cloud. Is this already resolved?

@btkcodedev thanks for the update, config validation is now passing but no records are being fetched. I pinged you on slack for credentials so we can get this moving.

sajarin · 2023-08-21T14:49:02Z

@padala can you make a new issue and tag me? Perhaps an issue with the record selector

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Sajarin <sajarindider@gmail.com>

padala · 2023-09-01T17:35:06Z

@sajarin I've created a new issues here - #30105

Migrate to lowcode

5e49d09

octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation community connectors/source/mailgun labels Aug 5, 2023

btkcodedev changed the title ~~Migrate to lowcode~~ ✨Source MailGun: Migrate Python CDK to Low-code CDK Aug 5, 2023

Add incremental sync, format

bfc7976

btkcodedev marked this pull request as ready for review August 5, 2023 09:15

btkcodedev added 2 commits August 5, 2023 15:37

remove comments

450c431

fix config

42b3302

marcosmarxm added the hackathon-2023-08 label Aug 7, 2023

marcosmarxm mentioned this pull request Aug 8, 2023

🐛Source MailGun: Update start_date validation #28471

Closed

Merge branch 'master' into mailgunlowcode

83f028d

marcosmarxm suggested changes Aug 8, 2023

View reviewed changes

airbyte-integrations/connectors/source-mailgun/acceptance-test-config.yml Show resolved Hide resolved

btkcodedev and others added 2 commits August 8, 2023 19:20

Update airbyte-integrations/connectors/source-mailgun/acceptance-test…

80953f2

…-config.yml Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>

Merge branch 'master' into mailgunlowcode

e6a2569

marcosmarxm added the team/tse Technical Support Engineers label Aug 8, 2023

Fix start_date

8542dc3

btkcodedev requested a review from marcosmarxm August 8, 2023 14:25

btkcodedev mentioned this pull request Aug 8, 2023

mailgun #29120

Closed

marcosmarxm added the Hackathon External low-code/ui connector builder hackathons label Aug 9, 2023

btkcodedev added 2 commits August 10, 2023 00:50

Merge branch 'master' into mailgunlowcode

95d03f6

Merge branch 'master' into mailgunlowcode

7e27364

marcosmarxm previously requested changes Aug 11, 2023

View reviewed changes

Fix metadata, Add pagination

2e7c6d9

Merge branch 'master' into mailgunlowcode

8366ece

btkcodedev requested a review from marcosmarxm August 12, 2023 18:25

btkcodedev and others added 2 commits August 13, 2023 00:08

EOF

155ce99

Merge branch 'master' into mailgunlowcode

20c632a

Merge branch 'master' into mailgunlowcode

779a196

Merge branch 'master' into mailgunlowcode

d181abd

sajarin mentioned this pull request Aug 17, 2023

Run ci checks for ✨Source MailGun: Migrate Python CDK to Low-code CDK #29536

Closed

11 tasks

Merge branch 'master' into mailgunlowcode

f26459e

sajarin approved these changes Aug 17, 2023

View reviewed changes

sajarin merged commit 359d846 into airbytehq:master Aug 17, 2023

harrytou pushed a commit to KYVENetwork/airbyte that referenced this pull request Sep 1, 2023

✨Source MailGun: Migrate Python CDK to Low-code CDK (airbytehq#29122)

f21c6ef

Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Sajarin <sajarindider@gmail.com>

padala mentioned this pull request Sep 1, 2023

Source Mailgun: After recent update sync resulting in 0 records #30105

Closed

1 task

btkcodedev mentioned this pull request Sep 6, 2023

Source Mailgun: Invalid Date Field Validation on start_date #27385

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

✨Source MailGun: Migrate Python CDK to Low-code CDK #29122

✨Source MailGun: Migrate Python CDK to Low-code CDK #29122

Uh oh!

btkcodedev commented Aug 5, 2023 •

edited

Loading

github-actions bot commented Aug 5, 2023

btkcodedev commented Aug 5, 2023

btkcodedev commented Aug 5, 2023

marcosmarxm commented Aug 8, 2023

btkcodedev commented Aug 8, 2023

Uh oh!

marcosmarxm left a comment

marcosmarxm Aug 11, 2023

marcosmarxm Aug 11, 2023

marcosmarxm Aug 11, 2023

marcosmarxm Aug 11, 2023

btkcodedev commented Aug 12, 2023

btkcodedev commented Aug 12, 2023

sajarin commented Aug 14, 2023

btkcodedev commented Aug 15, 2023 •

edited

Loading

sajarin commented Aug 15, 2023

sajarin commented Aug 17, 2023

padala commented Aug 21, 2023

sajarin commented Aug 21, 2023

padala commented Sep 1, 2023

Labels

5 participants

	def next_page_token(self, response: requests.Response) -> Optional[Mapping[str, HttpUrl]]:
	"""
	:param response: the most recent response from the API
	:return If there is another page in the result, a mapping (e.g: dict) containing information needed to query the next page in the response.
	If there are no more pages in the result, return None.
	"""
	next_page: Optional[HttpUrl] = response.json().get("paging", {}).get("next")
	return {"url": next_page} if next_page and self._pre_parse_response(response) else None

✨Source MailGun: Migrate Python CDK to Low-code CDK #29122

✨Source MailGun: Migrate Python CDK to Low-code CDK #29122

Uh oh!

Conversation

btkcodedev commented Aug 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

Tests

🚨 User Impact 🚨

Community member or Airbyter

github-actions bot commented Aug 5, 2023

Before Merging a Connector Pull Request

btkcodedev commented Aug 5, 2023

btkcodedev commented Aug 5, 2023

marcosmarxm commented Aug 8, 2023

btkcodedev commented Aug 8, 2023

Uh oh!

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm Aug 11, 2023

Choose a reason for hiding this comment

marcosmarxm Aug 11, 2023

Choose a reason for hiding this comment

marcosmarxm Aug 11, 2023

Choose a reason for hiding this comment

marcosmarxm Aug 11, 2023

Choose a reason for hiding this comment

btkcodedev commented Aug 12, 2023

btkcodedev commented Aug 12, 2023

sajarin commented Aug 14, 2023

btkcodedev commented Aug 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

sajarin commented Aug 15, 2023

sajarin commented Aug 17, 2023

padala commented Aug 21, 2023

sajarin commented Aug 21, 2023

padala commented Sep 1, 2023

Labels

5 participants

btkcodedev commented Aug 5, 2023 •

edited

Loading

btkcodedev commented Aug 15, 2023 •

edited

Loading