Skip to content

Commit 838aebe

Browse files
btkcodedevmarcosmarxmoctavia-squidington-iii
authored
🎉 New Source: SpaceX API [low-code cdk] (#18311)
* Init: New Source: SpaceX API * Chore: Removed duplicate and unwanted files * fix: Reconfigured schema and matched the read records. chore: Acceptance test run and results passed without errors * chore: Added documentation, updated airbyte docs specific to the connector * fix conflict * Chore: delete unwanted files * chore: Created bootstrap.md * chore: Update doc url, removed unwanted files * fix: Fixed schema after crosscheck with several schema validators * fix: resolved .vscode setting conflict, resolved merge conflict * feat: Added support for all endpoints * chore: updated documentation * feat: changes to stream path to accept options and id * fix: Resolve coomments * chore: resolved comments * chore: Delete unwanted files * chore: Update Readme.md after resolving merge conflict * update connector * add spacex api to source def * run format * add schemaloader * auto-bump connector version Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
1 parent 535e8da commit 838aebe

38 files changed

+2405
-0
lines changed

‎airbyte-config/init/src/main/resources/seed/source_definitions.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1184,6 +1184,13 @@
11841184
documentationUrl: https://docs.airbyte.com/integrations/sources/sonar-cloud
11851185
sourceType: api
11861186
releaseStage: alpha
1187+
- name: SpaceX API
1188+
sourceDefinitionId: 62235e65-af7a-4138-9130-0bda954eb6a8
1189+
dockerRepository: airbyte/source-spacex-api
1190+
dockerImageTag: 0.1.0
1191+
documentationUrl: https://docs.airbyte.com/integrations/sources/spacex-api
1192+
sourceType: api
1193+
releaseStage: alpha
11871194
- name: Square
11881195
sourceDefinitionId: 77225a51-cd15-4a13-af02-65816bd0ecf4
11891196
dockerRepository: airbyte/source-square

‎airbyte-config/init/src/main/resources/seed/source_specs.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11774,6 +11774,27 @@
1177411774
supportsNormalization: false
1177511775
supportsDBT: false
1177611776
supported_destination_sync_modes: []
11777+
- dockerImage: "airbyte/source-spacex-api:0.1.0"
11778+
spec:
11779+
documentationUrl: "https://docs.airbyte.com/integrations/sources/spacex-api"
11780+
connectionSpecification:
11781+
$schema: "http://json-schema.org/draft-07/schema#"
11782+
title: "Spacex Api Spec"
11783+
type: "object"
11784+
additionalProperties: true
11785+
properties:
11786+
id:
11787+
title: "Unique ID for specific source target"
11788+
type: "string"
11789+
desciption: "Optional, For a specific ID"
11790+
options:
11791+
title: "Configuration options for endpoints"
11792+
type: "string"
11793+
desciption: "Optional, Possible values for an endpoint. Example values for\
11794+
\ launches-latest, upcoming, past"
11795+
supportsNormalization: false
11796+
supportsDBT: false
11797+
supported_destination_sync_modes: []
1177711798
- dockerImage: "airbyte/source-square:0.1.4"
1177811799
spec:
1177911800
documentationUrl: "https://docs.airbyte.com/integrations/sources/square"

‎airbyte-integrations/connectors/source-spacex-api/.dockerignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*
2+
!Dockerfile
3+
!main.py
4+
!source_spacex_api
5+
!setup.py
6+
!secrets

‎airbyte-integrations/connectors/source-spacex-api/Dockerfile‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
FROM python:3.9.11-alpine3.15 as base
2+
3+
# build and load all requirements
4+
FROM base as builder
5+
WORKDIR /airbyte/integration_code
6+
7+
# upgrade pip to the latest version
8+
RUN apk --no-cache upgrade \
9+
&& pip install --upgrade pip \
10+
&& apk --no-cache add tzdata build-base
11+
12+
13+
COPY setup.py ./
14+
# install necessary packages to a temporary folder
15+
RUN pip install --prefix=/install .
16+
17+
# build a clean environment
18+
FROM base
19+
WORKDIR /airbyte/integration_code
20+
21+
# copy all loaded and built libraries to a pure basic image
22+
COPY --from=builder /install /usr/local
23+
# add default timezone settings
24+
COPY --from=builder /usr/share/zoneinfo/Etc/UTC /etc/localtime
25+
RUN echo "Etc/UTC" > /etc/timezone
26+
27+
# bash is installed for more convenient debugging.
28+
RUN apk --no-cache add bash
29+
30+
# copy payload code only
31+
COPY main.py ./
32+
COPY source_spacex_api ./source_spacex_api
33+
34+
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
35+
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]
36+
37+
LABEL io.airbyte.version=0.1.0
38+
LABEL io.airbyte.name=airbyte/source-spacex-api

‎airbyte-integrations/connectors/source-spacex-api/README.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# Spacex Api Source
2+
3+
This is the repository for the Spacex Api configuration based source connector.
4+
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/sources/spacex-api).
5+
6+
## Local development
7+
8+
### Prerequisites
9+
**To iterate on this connector, make sure to complete this prerequisites section.**
10+
11+
#### Minimum Python version required `= 3.9.0`
12+
13+
#### Build & Activate Virtual Environment and install dependencies
14+
From this connector directory, create a virtual environment:
15+
```
16+
python -m venv .venv
17+
```
18+
19+
This will generate a virtualenv for this module in `.venv/`. Make sure this venv is active in your
20+
development environment of choice. To activate it from the terminal, run:
21+
```
22+
source .venv/bin/activate
23+
pip install -r requirements.txt
24+
```
25+
If you are in an IDE, follow your IDE's instructions to activate the virtualenv.
26+
27+
Note that while we are installing dependencies from `requirements.txt`, you should only edit `setup.py` for your dependencies. `requirements.txt` is
28+
used for editable installs (`pip install -e`) to pull in Python dependencies from the monorepo and will call `setup.py`.
29+
If this is mumbo jumbo to you, don't worry about it, just put your deps in `setup.py` but install using `pip install -r requirements.txt` and everything
30+
should work as you expect.
31+
32+
#### Building via Gradle
33+
You can also build the connector in Gradle. This is typically used in CI and not needed for your development workflow.
34+
35+
To build using Gradle, from the Airbyte repository root, run:
36+
```
37+
./gradlew :airbyte-integrations:connectors:source-spacex-api:build
38+
```
39+
40+
#### Create credentials
41+
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/sources/spacex-api)
42+
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `source_spacex_api/spec.yaml` file.
43+
Note that any directory named `secrets` is gitignored across the entire Airbyte repo, so there is no danger of accidentally checking in sensitive information.
44+
See `integration_tests/sample_config.json` for a sample config file.
45+
46+
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `source spacex-api test creds`
47+
and place them into `secrets/config.json`.
48+
49+
### Locally running the connector docker image
50+
51+
#### Build
52+
First, make sure you build the latest Docker image:
53+
```
54+
docker build . -t airbyte/source-spacex-api:dev
55+
```
56+
57+
You can also build the connector image via Gradle:
58+
```
59+
./gradlew :airbyte-integrations:connectors:source-spacex-api:airbyteDocker
60+
```
61+
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
62+
the Dockerfile.
63+
64+
#### Run
65+
Then run any of the connector commands as follows:
66+
```
67+
docker run --rm airbyte/source-spacex-api:dev spec
68+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-spacex-api:dev check --config /secrets/config.json
69+
docker run --rm -v $(pwd)/secrets:/secrets airbyte/source-spacex-api:dev discover --config /secrets/config.json
70+
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/source-spacex-api:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
71+
```
72+
## Testing
73+
74+
#### Acceptance Tests
75+
Customize `acceptance-test-config.yml` file to configure tests. See [Source Acceptance Tests](https://docs.airbyte.io/connector-development/testing-connectors/source-acceptance-tests-reference) for more information.
76+
If your connector requires to create or destroy resources for use during acceptance tests create fixtures for it and place them inside integration_tests/acceptance.py.
77+
78+
To run your integration tests with docker
79+
80+
### Using gradle to run tests
81+
All commands should be run from airbyte project root.
82+
To run unit tests:
83+
```
84+
./gradlew :airbyte-integrations:connectors:source-spacex-api:unitTest
85+
```
86+
To run acceptance and custom integration tests:
87+
```
88+
./gradlew :airbyte-integrations:connectors:source-spacex-api:integrationTest
89+
```
90+
91+
## Dependency Management
92+
All of your dependencies should go in `setup.py`, NOT `requirements.txt`. The requirements file is only used to connect internal Airbyte dependencies in the monorepo for local development.
93+
We split dependencies between two groups, dependencies that are:
94+
* required for your connector to work need to go to `MAIN_REQUIREMENTS` list.
95+
* required for the testing need to go to `TEST_REQUIREMENTS` list
96+
97+
### Publishing a new version of the connector
98+
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
99+
1. Make sure your changes are passing unit and integration tests.
100+
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
101+
1. Create a Pull Request.
102+
1. Pat yourself on the back for being an awesome contributor.
103+
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

‎airbyte-integrations/connectors/source-spacex-api/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#
2+
# Copyright (c) 2022 Airbyte, Inc., all rights reserved.
3+
#

‎airbyte-integrations/connectors/source-spacex-api/acceptance-test-config.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# See [Source Acceptance Tests](https://docs.airbyte.com/connector-development/testing-connectors/source-acceptance-tests-reference)
2+
# for more information about how to configure these tests
3+
connector_image: airbyte/source-spacex-api:dev
4+
tests:
5+
spec:
6+
- spec_path: "source_spacex_api/spec.yaml"
7+
connection:
8+
- config_path: "secrets/config.json"
9+
status: "succeed"
10+
- config_path: "integration_tests/invalid_config.json"
11+
status: "failed"
12+
discovery:
13+
- config_path: "secrets/config.json"
14+
basic_read:
15+
- config_path: "secrets/config.json"
16+
configured_catalog_path: "integration_tests/configured_catalog.json"
17+
empty_streams: []
18+
full_refresh:
19+
- config_path: "secrets/config.json"
20+
configured_catalog_path: "integration_tests/configured_catalog.json"

‎airbyte-integrations/connectors/source-spacex-api/acceptance-test-docker.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/usr/bin/env sh
2+
3+
# Build latest connector image
4+
docker build . -t $(cat acceptance-test-config.yml | grep "connector_image" | head -n 1 | cut -d: -f2-)
5+
6+
# Pull latest acctest image
7+
docker pull airbyte/source-acceptance-test:latest
8+
9+
# Run
10+
docker run --rm -it \
11+
-v /var/run/docker.sock:/var/run/docker.sock \
12+
-v /tmp:/tmp \
13+
-v $(pwd):/test_input \
14+
airbyte/source-acceptance-test \
15+
--acceptance-test-config /test_input
16+

‎airbyte-integrations/connectors/source-spacex-api/bootstrap.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# SpaceX-API
2+
3+
The connector uses the v4 API documented here: https://github.com/r-spacex/SpaceX-API . It is
4+
straightforward HTTP REST API with no authentication.
5+
6+
## Dummy API key
7+
8+
Api key is mandate for this connector to work as it could be used in future enhancements.
9+
Just pass any dummy api key for establishing the connection. Example:123
10+
11+
## Implementation details
12+
13+
## Setup guide
14+
15+
### Step 1: Set up SpaceX connection
16+
17+
- Have a dummy API key (Example: 12345)
18+
- A specific id (If specific info is needed)
19+
20+
## Step 2: Generate schema for the endpoint
21+
22+
### Custom schema is generated and tested with different IDs
23+
24+
## Step 3: Spec, Secrets, and connector yaml files are configured with reference to the Airbyte documentation.
25+
26+
## In a nutshell:
27+
28+
1. Navigate to the Airbyte Open Source dashboard.
29+
2. Set the name for your source.
30+
3. Enter your `api_key`.
31+
5. Enter your `id` if needed. (Optional)
32+
6. Click **Set up source**.
33+
34+
* We use only GET methods, all endpoints are straightforward. We emit what we receive as HTTP response.
35+

‎airbyte-integrations/connectors/source-spacex-api/build.gradle‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
plugins {
2+
id 'airbyte-python'
3+
id 'airbyte-docker'
4+
id 'airbyte-source-acceptance-test'
5+
}
6+
7+
airbytePython {
8+
moduleDirectory 'source_spacex_api'
9+
}

0 commit comments

Comments
 (0)