CLIManaging CloudQueryDeploymentsDocker Offline

Docker - Offline Installation

You can run CloudQuery in a container with integrations pre-installed. This is useful for isolated deployments where you don’t want to download integrations from the internet.

Downloading integrations requires users to be authenticated, normally this means running cloudquery login but that is not doable in a CI environment or inside of a docker build process. The recommended way to handle this is to use an API key. More information on generating an API Key can be found here

To download the integrations based on your configuration file, use the cloudquery plugin install command. Below is an example Dockerfile based on the CloudQuery container. It uses a build.spec.yaml with the minimum configuration required to download the integrations.

# build.spec.yaml kind: source spec:  name: aws  path: cloudquery/aws  registry: cloudquery  version: "v32.47.0"  tables: ["aws_ec2_instances"]  destinations: ["postgresql"] --- kind: destination spec:  name: "postgresql"  path: "cloudquery/postgresql"  registry: "cloudquery"  version: "v8.12.2"  spec:
# Dockerfile FROM ghcr.io/cloudquery/cloudquery:latest AS build WORKDIR /app COPY ./build.spec.yaml /app/build.spec.yaml ARG CLOUDQUERY_API_KEY  RUN /app/cloudquery plugin install build.spec.yaml  FROM ghcr.io/cloudquery/cloudquery:latest  WORKDIR /app  # Copy the .cq directory which contains the integrations COPY --from=build /app/.cq /app/.cq

Build this container as you would normally do:

docker build --build-arg CLOUDQUERY_API_KEY=<your-api-key> ./ -t my-cq-container:latest

Run the Container

Run the container as you would run the default CloudQuery container. Here is an example:

docker run \  # you can mount a different config file that uses the same integrations as in the build.spec  -v <ABSOLUTE_PATH_TO_CONFIG_FILE>:/config.yml \  # set any env variable with -e <ENV_VAR_NAME>=<ENV_VAR_VALUE>  my-cq-container:latest \  sync /config.yml

Troubleshooting

If you encounter the following error when running the cloudquery plugin install command:

tls: failed to verify certificate: x509: certificate signed by unknown authority

You probably need to install certificates in your container image. To identify which certificates are needed, you can run the following command:

openssl s_client -showcerts -connect api.cloudquery.io:443

To extract the certificates to files, you can use the following command. This will create a file for each certificate in the current directory.

openssl s_client -showcerts -connect api.cloudquery.io:443 </dev/null | awk '/BEGIN CERTIFICATE/{n++;fname="cert" n ".pem"} /BEGIN CERTIFICATE/,/END CERTIFICATE/{print > fname}'

Then update your Dockerfile to copy over the certificates to your container and install them. The full Dockerfile should look like this:

# Dockerfile FROM ghcr.io/cloudquery/cloudquery:latest AS build RUN apk add --no-cache ca-certificates  WORKDIR /app COPY ./build.spec.yaml /app/build.spec.yaml COPY ./cert*.pem /usr/local/share/ca-certificates/ ARG CLOUDQUERY_API_KEY  ENV SSL_CERT_DIR=/usr/local/share/ca-certificates/  RUN /app/cloudquery plugin install build.spec.yaml  FROM ghcr.io/cloudquery/cloudquery:latest  WORKDIR /app  # Copy the .cq directory which contains the integrations COPY --from=build /app/.cq /app/.cq

Related docs

Read more about the plugin install command in the CLI Documentation.

DockerGenerate API Key