DEV Community

InterSystems profile image Mikhail Khomenko
Mikhail Khomenko for InterSystems

Posted on

Automating GKE creation on CircleCI builds

#devops #kubernetes #intersystems #circleci

Creating GKE cluster manually (or through gcloud) is easy, but the modern Infrastructure-as-Code (IaC) approach advises that the description of the Kubernetes cluster should be stored in the repository as code as well. How to write this code is determined by the tool that’s used for IaC.

In the case of Google Cloud, there are several options, among them Deployment Manager and Terraform. Opinions are divided as to which is better: if you want to learn more, read this Reddit thread Opinions on Terraform vs. Deployment Manager? and the Medium article Comparing GCP Deployment Manager and Terraform.

For this article we’ll choose Terraform, since it’s less tied to a specific vendor and you can use your IaC with different cloud providers.

We’ll assume you already have a Google account, and that you’ve created a project named, for instance, "Development". In this article, its ID is shown as <PROJECT_ID>. In the examples below, change it to the ID of your own project.

Keep in mind that Google isn’t free, although it has a free tier. Be sure to control your expenses.

You should fork the original repository. We’ll call this fork “my-objectscript-rest-docker-template” and refer to its root directory as "<root_repo_dir>" throughout this article.

All code samples are stored in this repo to simplify copying and pasting.

The following diagram depicts the whole deployment process in one picture:

Alt Text

So, let's install the latest version of Terraform at the time of initial writing:

$ terraform version Terraform v0.12.17
Enter fullscreen mode Exit fullscreen mode

The version is important here, because many examples on the Internet use earlier versions, and 0.12 brought many changes. Update: 0.13 brought yet more changes.

We want Terraform to perform certain actions (use certain APIs) in our GCP account. To enable this, create a Service Account with the name 'terraform', and enable the Kubernetes Engine API. Don’t worry about how we’re going to achieve this — just read further and your questions will be addressed.

Let's try an example with the gcloud utility, although we could also use the Web Console.

We're going to use a couple different commands in the following examples. See the following documentation topics for more details on these commands and features:

Now let's walk through the example.

$ gcloud init
Enter fullscreen mode Exit fullscreen mode

We won’t discuss all of the setup details here. You can read a little more in this article. For this example, run the following commands:

$ cd <root_repo_dir> $ mkdir terraform; cd terraform $ gcloud iam service-accounts create terraform --description "Terraform" --display-name "terraform"
Enter fullscreen mode Exit fullscreen mode

Now let's add a few roles to the terraform service account besides “Kubernetes Engine Admin” (container.admin). These roles will be useful to us in the future.

$ gcloud projects add-iam-policy-binding <PROJECT_ID> \ --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \ --role roles/container.admin $ gcloud projects add-iam-policy-binding <PROJECT_ID> \ --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \ --role roles/iam.serviceAccountUser $ gcloud projects add-iam-policy-binding <PROJECT_ID> \ --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \ --role roles/compute.viewer $ gcloud projects add-iam-policy-binding <PROJECT_ID> \ --member serviceAccount:terraform@<PROJECT_ID>.iam.gserviceaccount.com \ --role roles/storage.admin $ gcloud iam service-accounts keys create account.json \ --iam-account terraform@<PROJECT_ID>.iam.gserviceaccount.com
Enter fullscreen mode Exit fullscreen mode

Note that the last entry creates your account.json file. Be sure to keep this file secret.

$ gcloud projects list $ gcloud config set project <PROJECT_ID> $ gcloud services list --available | grep 'Kubernetes Engine' $ gcloud services enable container.googleapis.com $ gcloud services list --enabled | grep 'Kubernetes Engine' container.googleapis.com Kubernetes Engine API
Enter fullscreen mode Exit fullscreen mode

Next, let’s describe the GKE cluster in Terraform’s HCL language. Note that we use several placeholders here; replace them with your values:

Placeholder Meaning Example
<PROJECT_ID> GCP project ID possible-symbol-254507
<BUCKET_NAME> Storage for Terraform state - should be unique circleci-gke-terraform-demo
<REGION> Region where resources will be created europe-west1
<LOCATION> Zone where resources will be created europe-west1-b
<CLUSTER_NAME> GKE cluster name dev-cluster
<NODES_POOL_NAME> GKE worker nodes pool name dev-cluster-node-pool

Here’s the HCL configuration for the cluster in practice:

$ cat main.tf terraform { required_version = "~> 0.12" backend "gcs" { bucket = "<BUCKET_NAME>" prefix = "terraform/state" credentials = "account.json" } } provider "google" { credentials = file("account.json") project = "<PROJECT_ID>" region = "<REGION>" } resource "google_container_cluster" "gke-cluster" { name = "<CLUSTER_NAME>" location = "<LOCATION>" remove_default_node_pool = true # In regional cluster (location is region, not zone) # this is a number of nodes per zone initial_node_count = 1 } resource "google_container_node_pool" "preemptible_node_pool" { name = "<NODES_POOL_NAME>" location = "<LOCATION>" cluster = google_container_cluster.gke-cluster.name # In regional cluster (location is region, not zone) # this is a number of nodes per zone node_count = 1 node_config { preemptible = true machine_type = "n1-standard-1" oauth_scopes = [ "storage-ro", "logging-write", "monitoring" ] } }
Enter fullscreen mode Exit fullscreen mode

To make sure the HCL code is in the proper format, Terraform provides a handy formatting command you can use:

$ terraform fmt
Enter fullscreen mode Exit fullscreen mode

The code snippet shown above indicates that the created resources will be provided by Google, and the resources themselves are google_container_cluster and google_container_node_pool, which we designate preemptible for costs savings. We also choose to create our own pool instead of using the default.

Let’s focus briefly on the following setting:

terraform { required_version = "~> 0.12" backend "gcs" { Bucket = "<BUCKET_NAME>" Prefix = "terraform/state" credentials = "account.json" } }
Enter fullscreen mode Exit fullscreen mode

Terraform writes everything it's done into the status file and then uses this file for other work. For convenient sharing, it’s better to store this file somewhere in a remote place. A typical place is a Google Bucket.

Let's create this bucket. Use the name of your bucket instead of the placeholder <BUCKET_NAME>. Before bucket creation let’s check if <BUCKET_NAME> is available as it has to be unique across all GCP:

$ gsutil acl get gs://<BUCKET_NAME>
Enter fullscreen mode Exit fullscreen mode

Good answer:

BucketNotFoundException: 404 gs://<BUCKET_NAME> bucket does not exist
Enter fullscreen mode Exit fullscreen mode

"Busy" answer means you have to choose another name:

AccessDeniedException: 403 <YOUR_ACCOUNT> does not have storage.buckets.get access to <BUCKET_NAME>
Enter fullscreen mode Exit fullscreen mode

Let’s also enable versioning, as Terraform recommends.

$ gsutil mb -l EU gs://<BUCKET_NAME> $ gsutil versioning get gs://<BUCKET_NAME> gs://<BUCKET_NAME>: Suspended $ gsutil versioning set on gs://<BUCKET_NAME> $ gsutil versioning get gs://<BUCKET_NAME> gs://<BUCKET_NAME>: Enabled
Enter fullscreen mode Exit fullscreen mode

Terraform is modular and needs to add a Google provider plugin to create something in GCP. We use the following command to do this:

$ terraform init
Enter fullscreen mode Exit fullscreen mode

Let's look at what Terraform is going to do to create a GKE cluster:

$ terraform plan -out dev-cluster.plan
Enter fullscreen mode Exit fullscreen mode

The command output includes details of the plan. If you have no objections, let's implement this plan:

$ terraform apply dev-cluster.plan
Enter fullscreen mode Exit fullscreen mode

By the way, to delete the resources created by Terraform, run this command from the <root_repo_dir>/terraform/ directory:

$ terraform destroy -auto-approve
Enter fullscreen mode Exit fullscreen mode

Let’s leave the cluster as is for a while and move on. But first note that we don’t want to push everything into the repository, so we’ll add several files to the exceptions:

$ cat <root_repo_dir>/.gitignore .DS_Store terraform/.terraform/ terraform/*.plan terraform/*.json
Enter fullscreen mode Exit fullscreen mode

Using Helm

As described in this article, we could store Kubernetes manifests as yaml files in the <root_repo_dir>/k8s/ directory, which we then sent to the cluster using the "kubectl apply" command.

This time we'll try a different approach: using the Kubernetes package manager Helm, which currently has version 3. Please, use version 3 or later because version 2 had Kubernetes-side security issues (see Running Helm in production: Security best practices for details). First, we’ll pack the Kubernetes manifests from our k8s/ directory into a Helm package, which is known as a chart. A Helm chart installed in Kubernetes is called a release. In a minimal configuration, a chart will consist of several files:

$ mkdir <root_repo_dir>/helm; cd <root_repo_dir>/helm $ tree <root_repo_dir>/helm/ helm/ ├── Chart.yaml ├── templates │ ├── deployment.yaml │ ├── _helpers.tpl │ └── service.yaml └── values.yaml
Enter fullscreen mode Exit fullscreen mode

Their purpose is well-described on the official site. The best practices for creating your own charts are described in the The Chart Best Practices Guide in the Helm documentation.

Here’s what the contents of our files look like:

$ cat Chart.yaml apiVersion: v2 name: iris-rest version: 0.1.0 appVersion: 1.0.3 description: Helm for ObjectScript-REST-Docker-template application sources: - https://github.com/intersystems-community/objectscript-rest-docker-template - https://github.com/intersystems-community/gke-terraform-circleci-objectscript-rest-docker-template
Enter fullscreen mode Exit fullscreen mode 
$ cat templates/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: {{ template "iris-rest.name" . }} labels: app: {{ template "iris-rest.name" . }} chart: {{ template "iris-rest.chart" . }} release: {{ .Release.Name }} heritage: {{ .Release.Service }} spec: replicas: {{ .Values.replicaCount }} strategy: {{- .Values.strategy | nindent 4 }} selector: matchLabels: app: {{ template "iris-rest.name" . }} release: {{ .Release.Name }} template: metadata: labels: app: {{ template "iris-rest.name" . }} release: {{ .Release.Name }} spec: containers: - image: {{ .Values.image.repository }}:{{ .Values.image.tag }} name: {{ template "iris-rest.name" . }} ports: - containerPort: {{ .Values.webPort.value }} name: {{ .Values.webPort.name }}
Enter fullscreen mode Exit fullscreen mode 
$ cat templates/service.yaml {{- if .Values.service.enabled }} apiVersion: v1 kind: Service metadata: name: {{ .Values.service.name }} labels: app: {{ template "iris-rest.name" . }} chart: {{ template "iris-rest.chart" . }} release: {{ .Release.Name }} heritage: {{ .Release.Service }} spec: selector: app: {{ template "iris-rest.name" . }} release: {{ .Release.Name }} ports: {{- range $key, $value := .Values.service.ports }} - name: {{ $key }} {{ toYaml $value | indent 6 }} {{- end }} type: {{ .Values.service.type }} {{- if ne .Values.service.loadBalancerIP "" }} loadBalancerIP: {{ .Values.service.loadBalancerIP }} {{- end }} {{- end }}
Enter fullscreen mode Exit fullscreen mode 
$ cat templates/_helpers.tpl {{/* vim: set filetype=mustache: */}} {{/* Expand the name of the chart. */}} {{- define "iris-rest.name" -}} {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} {{- end -}} {{/* Create chart name and version as used by the chart label. */}} {{- define "iris-rest.chart" -}} {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} {{- end -}}
Enter fullscreen mode Exit fullscreen mode 
$ cat values.yaml namespaceOverride: iris-rest replicaCount: 1 strategy: | type: Recreate image: repository: eu.gcr.io/iris-rest tag: v1 webPort: name: web value: 52773 service: enabled: true name: iris-rest type: LoadBalancer loadBalancerIP: "" ports: web: port: 52773 targetPort: 52773 protocol: TCP
Enter fullscreen mode Exit fullscreen mode

To create the Helm charts, install the Helm client and the kubectl command-line utility.

$ helm version version.BuildInfo{Version:"v3.0.1", GitCommit:"7c22ef9ce89e0ebeb7125ba2ebf7d421f3e82ffa", GitTreeState:"clean", GoVersion:"go1.13.4"}
Enter fullscreen mode Exit fullscreen mode

Create a namespace called "iris". It would be nice if this was created during the deployment, but initially it was not the case. Update: looks like, --create-namespace flag is currently supported.

First, add credentials for the cluster created by Terraform to kube-config:

$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <LOCATION> --project <PROJECT_ID> $ kubectl create ns iris
Enter fullscreen mode Exit fullscreen mode

Confirm (without kicking off a real deploy) that Helm is going to create the following in Kubernetes:

$ cd <root_repo_dir>/helm $ helm upgrade iris-rest \ --install \ . \ --namespace iris \ --debug \ --dry-run
Enter fullscreen mode Exit fullscreen mode

The output - the Kubernetes manifests - has been omitted for space here. If everything looks good, let’s deploy:

$ helm upgrade iris-rest --install . --namespace iris $ helm list -n iris --all iris-rest iris 1 2019-12-14 15:24:19.292227564 +0200 EET deployed iris-rest-0.1.0 1.0.3
Enter fullscreen mode Exit fullscreen mode

We see that Helm has deployed our application, but since we haven’t created the Docker image eu.gcr.io/iris-rest:v1 yet, Kubernetes can’t pull it (ImagePullBackOff):

$ kubectl -n iris get po NAME READY STATUS RESTARTS AGE iris-rest-59b748c577-6cnrt 0/1 ImagePullBackOff 0 10m
Enter fullscreen mode Exit fullscreen mode

Let’s finish with it for now:

$ helm delete iris-rest -n iris
Enter fullscreen mode Exit fullscreen mode

The CircleCI Side

Now that we’ve tried out Terraform and the Helm client, let’s put them to use during the deployment process on the CircleCI side.

$ cat <root_repo_dir>/.circleci/config.yml version: 2.1 orbs: gcp-gcr: circleci/gcp-gcr@0.6.1 jobs: terraform: docker: # Terraform image version should be the same as when # you run terraform before from the local machine - image: hashicorp/terraform:0.12.17 steps: - checkout - run: name: Create Service Account key file from environment variable working_directory: terraform command: echo ${TF_SERVICE_ACCOUNT_KEY} > account.json - run: name: Show Terraform version command: terraform version - run: name: Download required Terraform plugins working_directory: terraform command: terraform init - run: name: Validate Terraform configuration working_directory: terraform command: terraform validate - run: name: Create Terraform plan working_directory: terraform command: terraform plan -out /tmp/tf.plan - run: name: Run Terraform plan working_directory: terraform command: terraform apply /tmp/tf.plan k8s_deploy: docker: - image: kiwigrid/gcloud-kubectl-helm:3.0.1-272.0.0-218 steps: - checkout - run: name: Authorize gcloud on GKE working_directory: helm command: | echo ${GCLOUD_SERVICE_KEY} > gcloud-service-key.json gcloud auth activate-service-account --key-file=gcloud-service-key.json gcloud container clusters get-credentials ${GKE_CLUSTER_NAME} --zone ${GOOGLE_COMPUTE_ZONE} --project ${GOOGLE_PROJECT_ID} - run: name: Wait a little until k8s worker nodes up command: sleep 30 # It’s a place for improvement - run: name: Create IRIS namespace if it doesn't exist command: kubectl get ns iris || kubectl create ns iris - run: name: Run Helm release deployment working_directory: helm command: | helm upgrade iris-rest \ --install \ . \ --namespace iris \ --wait \ --timeout 300s \ --atomic \ --set image.repository=eu.gcr.io/${GOOGLE_PROJECT_ID}/iris-rest \ --set image.tag=${CIRCLE_SHA1} - run: name: Check Helm release status command: helm list --all-namespaces --all - run: name: Check Kubernetes resources status command: | kubectl -n iris get pods echo kubectl -n iris get services workflows: main: jobs: - terraform - gcp-gcr/build-and-push-image: dockerfile: Dockerfile gcloud-service-key: GCLOUD_SERVICE_KEY google-compute-zone: GOOGLE_COMPUTE_ZONE google-project-id: GOOGLE_PROJECT_ID registry-url: eu.gcr.io image: iris-rest path: . tag: ${CIRCLE_SHA1} - k8s_deploy: requires: - terraform - gcp-gcr/build-and-push-image
Enter fullscreen mode Exit fullscreen mode

You’ll need to add several environment variables to your project on CircleCI side:

Alt Text

The GCLOUD_SERVICE_KEY is the CircleCI service account key, and TF_SERVICE_ACCOUNT_KEY is the Terraform service account key. Recall that the service account key is the whole content of account.json file.

Next, let’s push our changes to a repository:

$ cd <root_repo_dir> $ git add .circleci/ helm/ terraform/ .gitignore $ git commit -m "Add Terraform and Helm" $ git push
Enter fullscreen mode Exit fullscreen mode

The CircleCI UI dashboard should show that everything is ok:

Alt Text

Terraform is an idempotent tool and if the GKE cluster is present, the "terraform" job won’t do anything. If the cluster doesn’t exist, it will be created before Kubernetes deployment.
Finally, let’s check IRIS availability:

$ gcloud container clusters get-credentials <CLUSTER_NAME> --zone <LOCATION> --project <PROJECT_ID> $ kubectl -n iris get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE iris-rest LoadBalancer 10.23.249.42 34.76.130.11 52773:31603/TCP 53s $ curl -XPOST -H "Content-Type: application/json" -u _system:SYS 34.76.130.11:52773/person/ -d '{"Name":"John Dou"}' $ curl -XGET -u _system:SYS 34.76.130.11:52773/person/all [{"Name":"John Dou"},]
Enter fullscreen mode Exit fullscreen mode

Conclusion

Terraform and Helm are standard DevOps tools and should be fine integrated with IRIS deployment.

They do require some learning, but after some practice, they can really save you time and effort.

Top comments (0)