DEV Community

Cover image for Bypass AWS_ENDPOINT to interact with any S3 compatible service from CockroachDB
Cockroach Labs profile image Fabio Ghirardello
Fabio Ghirardello for Cockroach Labs

Posted on

Bypass AWS_ENDPOINT to interact with any S3 compatible service from CockroachDB

#cockroachdb #minio #s3 #tutorial

In this blog I will demonstrate how you can trick an application or service, in our case CockroachDB, into thinking it's communicating with AWS S3 when in fact another service lays behind the scenes.

CockroachDB is perfectly capable to interact with any S3 compatible services. Following the instructions of the AWS SDK, you can add parameter AWS_ENDPOINT to your URI to tell CockroachDB what server to hit instead of the default AWS servers. This is clearly documented here.

However, this blog comes as a follow up from a customer interaction where their requirement was to NOT use AWS_ENDPOINT in their URI, yet still using their own private S3 compatible service.

Setup

Create a t2.micro or similar instance on your favorite cloud provider.
For this exercise, the instance has the below details:

OS : Ubuntu 20.04 LTS Public IP : 54.83.105.95 Private IP : 10.10.81.189
Enter fullscreen mode Exit fullscreen mode

Install and Configure MinIO

MinIO is a S3 compatible service and it is my favorite choice for when I want to play locally with the S3 API. We will therefore use the MinIO Server as our custom endpoint instead of AWS S3.

Below the steps to run MinIO, also available in the official docs.

# download the binary wget https://dl.min.io/server/minio/release/linux-amd64/minio chmod +x minio ./minio server data
Enter fullscreen mode Exit fullscreen mode

At this point MinIO is running in insecure mode.
It will also have created the configuration folders at ${HOME}/.minio/certs.
Stop the server with Ctrl+C, then configure secure access with TLS.

# allow minio to start on port 443 sudo setcap cap_net_bind_service=+ep minio # create the private key cd .minio/certs openssl genrsa -out private.key 2048 chmod 400 private.key
Enter fullscreen mode Exit fullscreen mode

Create a file named openssl.conf with the content below. Check the alt_names section carefully: in this context, fabio is the name of the s3 bucket.

[req] distinguished_name = req_distinguished_name x509_extensions = v3_req prompt = no [req_distinguished_name] C = US ST = VA L = Somewhere O = MyOrg OU = MyOU CN = MyServerName [v3_req] subjectAltName = @alt_names [alt_names] IP.1 = 127.0.0.1 IP.2 = 54.83.105.95 IP.3 = 10.10.81.189 DNS.1 = localhost DNS.2 = s3.amazonaws.com DNS.3 = fabio.s3.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Create the public cert

openssl req -new -x509 -nodes -days 730 -key private.key -out public.crt -config openssl.conf
Enter fullscreen mode Exit fullscreen mode

Setup MinIO Server to use Virtual Hosted-Style, as per here.
This is required to match AWS S3 style.

export MINIO_DOMAIN=s3.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Finally, start the MinIO Server

$ ~/minio server --console-address ":9001" --address ":443" ~/data API: https://10.10.82.1 https://127.0.0.1 RootUser: minioadmin RootPass: minioadmin Console: https://10.10.82.1:9001 https://127.0.0.1:9001 RootUser: minioadmin RootPass: minioadmin Command-line: https://docs.min.io/docs/minio-client-quickstart-guide $ mc alias set myminio https://10.10.82.1 minioadmin minioadmin Documentation: https://docs.min.io WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
Enter fullscreen mode Exit fullscreen mode

Point your browser at https://54.83.105.95:9001 to access the MinIO Console. As the self-signed certificate is invalid, the browser might not allow you to continue. In Brave Browser, and other Chrome based browsers, type thisisunsafe anywhere on the screen (not on the address toolbar) to continue and disregard the warning.

Login with minioadmin/minioadmin, which are the default login details. Once logged in, you'll see the Dashboard page, below

minio-dashboard

Using the menu on the left, create a bucket called fabio.

bucket

Then create a User fabio with any password you want. Make sure assign the consoleAdmin, readwrite and diagnostic policies to the user, so you don't run into any permission issue. Later on, you can refine these permissions.

Alt Text

Then, click on the user you just created and create a Service Account, and make sure you save the keys. In my example, the keys are:

Access Key: I6SX7TO71RZY79FXDK1T Secret Key: 7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm
Enter fullscreen mode Exit fullscreen mode

Alt Text

Finally, go to Settings and change the region to us-east-1.

Alt Text

This will prompt you to RESTART the server. Do so by stopping the server, and restarting it.

Perfect, you're all set! MinIO Server is up and running in secure mode, and you've a service account to interact with it.

Next, we configure the OS.

Linux OS configuration

For CockroachDB to trust the certificate, we must add it to the system cert pool. We also need to redirect the traffic aimed at AWS servers to MinIO's.

Allow self-signed certs

In Ubuntu, this is how you go about, as per here

sudo mkdir /usr/local/share/ca-certificates/extra sudo cp ~/.minio/certs/public.crt /usr/local/share/ca-certificates/extra/ sudo update-ca-certificates
Enter fullscreen mode Exit fullscreen mode

Routing

Edit file /etc/hosts and add these entries to route all traffic directed to Amazon AWS servers to the MinIO Server instead.
Again, fabio is the name of the bucket.

54.83.105.95 s3.amazonaws.com 54.83.105.95 s3.us-east-1.amazonaws.com 54.83.105.95 fabio.s3.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Good, we're all set from OS point of view. Next, we test with CockroachDB.

CockroachDB

Install and start CockroachDB. We will use the single-node feature as we're only trying to prove it works from a functionality perspective.

As always, the offical installation docs are here.

curl https://binaries.cockroachdb.com/cockroach-v21.1.10.linux-amd64.tgz | tar -xz && sudo cp -i cockroach-v21.1.10.linux-amd64/cockroach /usr/local/bin/ cockroach start-single-node --insecure --background
Enter fullscreen mode Exit fullscreen mode

Connect to the SQL prompt as root:

cockroach sql --insecure
Enter fullscreen mode Exit fullscreen mode

Create a user - doesn't have to be an admin -, then logout and log back in as that user

CREATE USER fabio WITH CREATEDB;
Enter fullscreen mode Exit fullscreen mode 
cockroach sql --insecure -u fabio
Enter fullscreen mode Exit fullscreen mode

As user fabio, create a simple test table and export it. Make sure you update the keys with your values!

CREATE DATABASE fabio; USE fabio; CREATE TABLE test AS SELECT a FROM generate_series(1,100) AS a; -- notice that we don't use AWS_ENDPOINT in the URI EXPORT INTO CSV 's3://fabio?AWS_ACCESS_KEY_ID=I6SX7TO71RZY79FXDK1T&AWS_SECRET_ACCESS_KEY=7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm' FROM TABLE test;
Enter fullscreen mode Exit fullscreen mode 
 filename | rows | bytes -------------------------------------------------------------------+------+-------- export16ad4c7f75f4fe180000000000000001-n701156393427501057.0.csv | 100 | 292 (1 row) Time: 38ms total (execution 38ms / network 0ms)
Enter fullscreen mode Exit fullscreen mode

As root, take a full cluster backup

BACKUP INTO 's3://fabio?AWS_ACCESS_KEY_ID=I6SX7TO71RZY79FXDK1T&AWS_SECRET_ACCESS_KEY=7TxSqDyrvOXCV+EWkRvZKSOXETriFik5LKNWmkQm';
Enter fullscreen mode Exit fullscreen mode 
 job_id | status | fraction_completed | rows | index_entries | bytes ---------------------+-----------+--------------------+------+---------------+-------- 701160820164395009 | succeeded | 1 | 126 | 14 | 5920 (1 row)
Enter fullscreen mode Exit fullscreen mode

Refresh the MinIO Console to see your files

Alt Text

Awesome! The CockroachDB end-user has no clue that the engine behind the scenes is MinIO instead of AWS, and CockroachDB is happy to IMPORT, EXPORT, BACKUP and RESTORE from S3, be it AWS' or Minio's.

Reference

Top comments (0)