Skip to content

Commit 45ad415

Browse files
xlahuertaVincent Kocvincentkococtavia-squidington-iii
authored
🎉 New Source: Pocket [low-code CDK] (#18655)
* Pocket Source Connector initial implementation * Update changelog with PR id * Style fixes & bootstrap.md * Delete abnormal_state.json * Update setup.py * Update invalid_config.json * Delete sample_state.json * Update retrieve.json * Update acceptance.py * Rename Pocket Extractor test * Add `schema_loader` as a workaround to prevent `EmptySchemaLoader` * auto-bump connector version Co-authored-by: Vincent Koc <koconder@users.noreply.github.com> Co-authored-by: Vincent Koc <koconder@gmail.com> Co-authored-by: Vincent Koc <vincentkoc@ieee.org> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
1 parent 7546450 commit 45ad415

File tree

28 files changed

+1049
-0
lines changed

28 files changed

+1049
-0
lines changed

‎airbyte-config/init/src/main/resources/icons/pocket.svg‎

Lines changed: 117 additions & 0 deletions
Loading

‎airbyte-config/init/src/main/resources/seed/source_definitions.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1066,6 +1066,14 @@
10661066
icon: plaid.svg
10671067
sourceType: api
10681068
releaseStage: alpha
1069+
- name: Pocket
1070+
sourceDefinitionId: b0dd65f1-081f-4731-9c51-38e9e6aa0ebf
1071+
dockerRepository: airbyte/source-pocket
1072+
dockerImageTag: 0.1.0
1073+
documentationUrl: https://docs.airbyte.com/integrations/sources/pocket
1074+
icon: pocket.svg
1075+
sourceType: api
1076+
releaseStage: alpha
10691077
- name: PokeAPI
10701078
sourceDefinitionId: 6371b14b-bc68-4236-bfbd-468e8df8e968
10711079
dockerRepository: airbyte/source-pokeapi

‎airbyte-config/init/src/main/resources/seed/source_specs.yaml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10180,6 +10180,100 @@
1018010180
supportsNormalization: false
1018110181
supportsDBT: false
1018210182
supported_destination_sync_modes: []
10183+
- dockerImage: "airbyte/source-pocket:0.1.0"
10184+
spec:
10185+
documentationUrl: "https://docs.airbyte.com/integrations/sources/pocket"
10186+
connectionSpecification:
10187+
$schema: "http://json-schema.org/draft-07/schema#"
10188+
title: "Pocket Spec"
10189+
type: "object"
10190+
required:
10191+
- "consumer_key"
10192+
- "access_token"
10193+
additionalProperties: true
10194+
properties:
10195+
consumer_key:
10196+
type: "string"
10197+
title: "Consumer Key"
10198+
description: "Your application's Consumer Key."
10199+
airbyte_secret: true
10200+
order: 0
10201+
access_token:
10202+
type: "string"
10203+
title: "Access Token"
10204+
description: "The user's Pocket access token."
10205+
airbyte_secret: true
10206+
order: 1
10207+
state:
10208+
type: "string"
10209+
title: "State"
10210+
description: "Select the state of the items to retrieve."
10211+
order: 2
10212+
enum:
10213+
- "unread"
10214+
- "archive"
10215+
- "all"
10216+
favorite:
10217+
type: "boolean"
10218+
title: "Is Favorite?"
10219+
description: "Retrieve only favorited items."
10220+
default: false
10221+
order: 3
10222+
tag:
10223+
type: "string"
10224+
title: "Tag Name"
10225+
description: "Return only items tagged with this tag name. Use _untagged_\
10226+
\ for retrieving only untagged items."
10227+
order: 4
10228+
content_type:
10229+
type: "string"
10230+
title: "Content Type"
10231+
description: "Select the content type of the items to retrieve."
10232+
order: 5
10233+
enum:
10234+
- "article"
10235+
- "video"
10236+
- "image"
10237+
sort:
10238+
type: "string"
10239+
title: "Sort By"
10240+
description: "Sort retrieved items by the given criteria."
10241+
order: 6
10242+
enum:
10243+
- "newest"
10244+
- "oldest"
10245+
- "title"
10246+
- "site"
10247+
detail_type:
10248+
type: "string"
10249+
title: "Detail Type"
10250+
description: "Select the granularity of the information about each item."
10251+
order: 7
10252+
enum:
10253+
- "simple"
10254+
- "complete"
10255+
search:
10256+
type: "string"
10257+
title: "Search Query"
10258+
description: "Only return items whose title or url contain the `search`\
10259+
\ string."
10260+
order: 8
10261+
domain:
10262+
type: "string"
10263+
title: "Domain"
10264+
description: "Only return items from a particular `domain`."
10265+
order: 9
10266+
since:
10267+
type: "string"
10268+
title: "Since"
10269+
description: "Only return items modified since the given timestamp."
10270+
pattern: "[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}"
10271+
examples:
10272+
- "2022-10-20 14:14:14"
10273+
order: 10
10274+
supportsNormalization: false
10275+
supportsDBT: false
10276+
supported_destination_sync_modes: []
1018310277
- dockerImage: "airbyte/source-pokeapi:0.1.5"
1018410278
spec:
1018510279
documentationUrl: "https://docs.airbyte.com/integrations/sources/pokeapi"

‎airbyte-integrations/connectors/source-pocket/.dockerignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*
2+
!Dockerfile
3+
!main.py
4+
!source_pocket
5+
!setup.py
6+
!secrets

‎airbyte-integrations/connectors/source-pocket/Dockerfile‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
FROM python:3.9.11-alpine3.15 as base
2+
3+
# build and load all requirements
4+
FROM base as builder
5+
WORKDIR /airbyte/integration_code
6+
7+
# upgrade pip to the latest version
8+
RUN apk --no-cache upgrade \
9+
&& pip install --upgrade pip \
10+
&& apk --no-cache add tzdata build-base
11+
12+
13+
COPY setup.py ./
14+
# install necessary packages to a temporary folder
15+
RUN pip install --prefix=/install .
16+
17+
# build a clean environment
18+
FROM base
19+
WORKDIR /airbyte/integration_code
20+
21+
# copy all loaded and built libraries to a pure basic image
22+
COPY --from=builder /install /usr/local
23+
# add default timezone settings
24+
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
25+
RUN echo "Etc/UTC" > /etc/timezone
26+
27+
# bash is installed for more convenient debugging.
28+
RUN apk --no-cache add bash
29+
30+
# copy payload code only
31+
COPY main.py ./
32+
COPY source_pocket ./source_pocket
33+
34+
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
35+
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
36+
37+
LABEL io.airbyte.version=0.1.0
38+
LABEL io.airbyte.name=airbyte/source-pocket

‎airbyte-integrations/connectors/source-pocket/README.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# Pocket Source
2+
3+
This is the repository for the Pocket configuration based source connector.
4+
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/pocket).
5+
6+
## Local development
7+
8+
#### Building via Gradle
9+
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.
10+
11+
To build using Gradle, from the Airbyte repository root, run:
12+
```
13+
./gradlew :airbyte-integrations:connectors:source-pocket:build
14+
```
15+
16+
#### Create credentials
17+
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/pocket)
18+
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_pocket/spec.yaml` file.
19+
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
20+
See `integration_tests/sample_config.json` for a sample config file.
21+
22+
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source pocket test creds`
23+
and place them into `secrets/config.json`.
24+
25+
### Locally running the connector docker image
26+
27+
#### Build
28+
First, make sure you build the latest Docker image:
29+
```
30+
docker build . -t airbyte/source-pocket:dev
31+
```
32+
33+
You can also build the connector image via Gradle:
34+
```
35+
./gradlew :airbyte-integrations:connectors:source-pocket:airbyteDocker
36+
```
37+
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
38+
the Dockerfile.
39+
40+
#### Run
41+
Then run any of the connector commands as follows:
42+
```
43+
docker run --rm airbyte/source-pocket:dev spec
44+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-pocket:dev check --config /secrets/config.json
45+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-pocket:dev discover --config /secrets/config.json
46+
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-pocket:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
47+
```
48+
## Testing
49+
50+
#### Acceptance Tests
51+
Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference) for more information.
52+
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
53+
54+
To run your integration tests with docker
55+
56+
### Using gradle to run tests
57+
All commands should be run from airbyte project root.
58+
To run unit tests:
59+
```
60+
./gradlew :airbyte-integrations:connectors:source-pocket:unitTest
61+
```
62+
To run acceptance and custom integration tests:
63+
```
64+
./gradlew :airbyte-integrations:connectors:source-pocket:integrationTest
65+
```
66+
67+
## Dependency Management
68+
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
69+
We split dependencies between two groups, dependencies that are:
70+
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
71+
* required for the testing need to go to `TEST_REQUIREMENTS` list
72+
73+
### Publishing a new version of the connector
74+
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
75+
1. Make sure your changes are passing unit and integration tests.
76+
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
77+
1. Create a Pull Request.
78+
1. Pat yourself on the back for being an awesome contributor.
79+
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

‎airbyte-integrations/connectors/source-pocket/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#

‎airbyte-integrations/connectors/source-pocket/acceptance-test-config.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# See [Source Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/source-acceptance-tests-reference)
2+
# for more information about how to configure these tests
3+
connector_image: airbyte/source-pocket:dev
4+
acceptance_tests:
5+
spec:
6+
tests:
7+
- spec_path: "source_pocket/spec.yaml"
8+
connection:
9+
tests:
10+
- config_path: "secrets/config.json"
11+
status: "succeed"
12+
- config_path: "integration_tests/invalid_config.json"
13+
status: "failed"
14+
discovery:
15+
tests:
16+
- config_path: "secrets/config.json"
17+
basic_read:
18+
tests:
19+
- config_path: "secrets/config.json"
20+
configured_catalog_path: "integration_tests/configured_catalog.json"
21+
empty_streams: []
22+
incremental:
23+
bypass_reason: "This connector does not implement incremental sync"
24+
full_refresh:
25+
tests:
26+
- config_path: "secrets/config.json"
27+
configured_catalog_path: "integration_tests/configured_catalog.json"

0 commit comments

Comments
 (0)