Deploy-less Orchestration with GitHub Actions

In this tutorial, we will show you how to load AWS resources into a Postgres database by running CloudQuery as a GitHub Action, using the AWS source- and Postgresql destination integrations.

Prerequisites

Generating a CloudQuery API key

Downloading integrations requires users to be authenticated, normally this means running cloudquery login but that is not doable in a CI environment like GitHub Actions. The recommended way to handle this is to use an API key. More information on generating an API Key can be found here.

AWS Authentication

Since we’ll be running CloudQuery in the context of a GitHub Action runner, we’ll need to add AWS authentication.

To set up authentication with AWS from GitHub Actions you can follow the Configuring OpenID Connect in Amazon Web Services blog from GitHub.

The role that you create will be used by CloudQuery to download your cloud configuration, so you should also grant it permissions to read from your cloud (e.g. ReadOnlyAccess permission policy)

Creating the CloudQuery configuration file

Under the root of your repository, create a new cloudquery.yml file with the following content:

kind: source spec:  name: 'aws'  path: cloudquery/aws  registry: cloudquery  version: "v32.47.0"  tables: ['*']  destinations: ['postgresql'] --- kind: destination spec:  name: 'postgresql'  path: cloudquery/postgresql  registry: cloudquery  version: "v8.12.2"  spec:  connection_string: ${CQ_DSN} # The CQ_DSN environment variable will be set by GitHub Action workflow

For more configuration options, visit our docs

Creating the GitHub Action workflow

First we’ll need to create a GitHub secret with the name CQ_DSN and the value of the connection string to your PostgreSQL database.

Create a workflow file under .github/workflows/cloudquery.yml with the following content, and fill in <role-arn> and <region> according to the role you created in the prerequisites.

name: CloudQuery on:  schedule:  - cron: '0 3 * * *' # Run daily at 03:00 (3am)  jobs:  cloudquery:  permissions:  id-token: write  contents: read  runs-on: ubuntu-latest  steps:  - uses: actions/checkout@v3 # Checkout the code so we have access to the config file  - name: Configure AWS credentials # Setup AWS credentials (example)  uses: aws-actions/configure-aws-credentials@v1  with:  role-to-assume: <role-arn> # based on the role you created in the prerequisites  aws-region: <region> # based on the region you created the role in  - uses: cloudquery/setup-cloudquery@v3  name: Setup CloudQuery  with:  version: "v6.29.5"  - name: Sync with CloudQuery  run: cloudquery sync cloudquery.yml --log-console  env:  CLOUDQUERY_API_KEY: ${{ secrets.CLOUDQUERY_API_KEY }} # See https://cli-docs.cloudquery.io/docs/managing-cloudquery/deployments/generate-api-key  CQ_DSN: ${{ secrets.CQ_DSN }} # Connection string to a PostgreSQL database

Once committed to the default branch of the repository, the above workflow will run daily at 3 a.m. and will sync the AWS source integration with the PostgreSQL destination integration.

Warning GitHub automatically disables workflows on public repositories if no repository activity has occurred for 60 days. This may impact your sync if the repository does not receive regular commits.

Running CloudQuery in parallel to speed up sync time

By default, CloudQuery extracts all supported resources, which can take a bit of time, depending on the number of resources you have in your AWS account.

With the GitHub Actions matrix configuration, you can split the sync process into multiple jobs and run them in parallel.

To do so, create the following workflow file under .github/workflows/cloudquery-parallel.yml:

name: CloudQuery Parallel on:  schedule:  - cron: '0 3 * * *' # Run daily at 03:00 (3am) jobs:  cloudquery:  permissions:  id-token: write  contents: read  runs-on: ubuntu-latest  strategy:  matrix:  shard: [1/4, 2/4, 3/4, 4/4] # Split the sync into 4 parts  steps:  - uses: actions/checkout@v3 # Checkout the code so we have access to the config file  - name: Configure AWS credentials # Setup AWS credentials (example)  uses: aws-actions/configure-aws-credentials@v1  with:  role-to-assume: <role-arn> # based on the role you created in the prerequisites  aws-region: <region> # based on the region you created the role in  - uses: cloudquery/setup-cloudquery@v3  name: Setup CloudQuery  with:  version: "v6.29.5"  - name: Sync with CloudQuery  run: cloudquery sync cloudquery.yml --log-console --shard ${{ matrix.shard }}  env:  CLOUDQUERY_API_KEY: ${{ secrets.CLOUDQUERY_API_KEY }} # See https://cli-docs.cloudquery.io/docs/managing-cloudquery/deployments/generate-api-key  CQ_DSN: ${{ secrets.CQ_DSN }} # Connection string to a PostgreSQL database

Once committed to the default branch of the repository, the above workflow will run daily at 3 a.m and will sync the AWS source integration with the PostgreSQL destination integration, in parallel.