When we talk about Infrastructure as Code or IaC, the first tool that comes to mind is Terraform. Terraform, created by HashiCorp, has become the standard for documentation and infrastructure management, but its declarative language, HCL (HashiCorp Configuration Language), has some limitations. The main limitation is not being a programming language but a configuration one.
Some alternatives have been emerging to fulfill these needs, such as:
AWS Cloud Development Kit, Amazon's solution that allows us to use TypeScript, Python, and Java to program the infrastructure using the cloud provider's solutions;
Pulumi, which allows us to use TypeScript, JavaScript, Python, Go, and C# to program infrastructures using solutions from AWS, Microsoft Azure, Google Cloud, and Kubernetes installations.
I will introduce Pulumi, using the Go language to create some infrastructure examples on AWS.
Installation
To make use of Pulumi, we first need to install its command-line application. Following the documentation, I installed it on my macOS using the command:
brew install pulumi On the website, you can see how to install it on Windows and Linux.
Configure AWS Account Access
Since I will use AWS in this example, the next necessary step is to configure the credentials. For that, I got my access key and secret from the AWS dashboard and set the required environment variables:
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID> export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY> Creating the project
With the initial dependencies configured, we can now create the project:
mkdir post-pulumi cd post-pulumi pulumi new aws-go One of the creation steps requires setting up an account on the Pulumi website. For that, the command-line application opens the browser for this step to be completed. So I logged in with my Github account, completed the registration, returned to the terminal, and continued the project creation without any problems.
You can see the result of running the command can at this link. In addition, at the end of the process, it installs all the necessary dependencies for creating the project in Go.
Files created
Looking at the directory contents, we can see that some configuration files and a main.go were created.
Pulumi.yaml
name: post-pulumi runtime: go description: A minimal AWS Go Pulumi program Pulumi.dev.yaml
config: aws:region: us-east-1 main.go
package main import ( "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) func main() { pulumi.Run(func(ctx *pulumi.Context) error { // Create an AWS resource (S3 Bucket) bucket, err := s3.NewBucket(ctx, "my-bucket", nil) if err != nil { return err } // Export the name of the bucket ctx.Export("bucketName", bucket.ID()) return nil }) } When running
pulumi up The bucket was created in S3, as the code indicates.
And the command:
pulumi destroy Destroy all the resources, in this case, the S3 bucket.
First example - creating a static page in S3
Now let's do some more complex examples.
The first step is to create a static page, which we are going to deploy:
mkdir static Inside this directory, I created the file:
static/index.html
<html> <body> <h1>Hello, Pulumi!</h1> </body> </html> I changed main.go to reflect the new structure:
package main import ( "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) func main() { pulumi.Run(func(ctx *pulumi.Context) error { // Create an AWS resource (S3 Bucket) bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{ Website: s3.BucketWebsiteArgs{ IndexDocument: pulumi.String("index.html"), }, }) if err != nil { return err } // Export the name of the bucket ctx.Export("bucketName", bucket.ID()) _, err = s3.NewBucketObject(ctx, "index.html", &s3.BucketObjectArgs{ Acl: pulumi.String("public-read"), ContentType: pulumi.String("text/html"), Bucket: bucket.ID(), Source: pulumi.NewFileAsset("static/index.html"), }) if err != nil { return err } ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint)) return nil }) } To update run:
pulumi up And confirm the change.
The code snippet:
ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint)) Generate as output the address to access index.html:
Outputs: + bucketEndpoint: "http://my-bucket-357877e.s3-website-us-east-1.amazonaws.com" The case above is a straightforward example, but it already demonstrates the power of the tool. So let's make things a little more complex and fun now.
Second example - a site inside a container
Let's create a Dockerfile with a web server to host our static content:
static/Dockerfile
FROM golang ADD . /go/src/foo WORKDIR /go/src/foo RUN go build -o /go/bin/main ENTRYPOINT /go/bin/main EXPOSE 80 Let's now create the static/main.go file, which will be our web server:
package main import ( "log" "net/http" ) func main() { r := http.NewServeMux() fileServer := http.FileServer(http.Dir("./")) r.Handle("/", http.StripPrefix("/", fileServer)) s := &http.Server{ Addr: ":80", Handler: r, } log.Fatal(s.ListenAndServe()) } Let's change main.go to include the infrastructure of an ECS cluster and everything else needed to run our container:
package main import ( "encoding/base64" "fmt" "strings" "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2" "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr" "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs" elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2" "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam" "github.com/pulumi/pulumi-docker/sdk/v3/go/docker" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) func main() { pulumi.Run(func(ctx *pulumi.Context) error { // Read back the default VPC and public subnets, which we will use. t := true vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t}) if err != nil { return err } subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id}) if err != nil { return err } // Create a SecurityGroup that permits HTTP ingress and unrestricted egress. webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{ VpcId: pulumi.String(vpc.Id), Egress: ec2.SecurityGroupEgressArray{ ec2.SecurityGroupEgressArgs{ Protocol: pulumi.String("-1"), FromPort: pulumi.Int(0), ToPort: pulumi.Int(0), CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")}, }, }, Ingress: ec2.SecurityGroupIngressArray{ ec2.SecurityGroupIngressArgs{ Protocol: pulumi.String("tcp"), FromPort: pulumi.Int(80), ToPort: pulumi.Int(80), CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")}, }, }, }) if err != nil { return err } // Create an ECS cluster to run a container-based service. cluster, err := ecs.NewCluster(ctx, "app-cluster", nil) if err != nil { return err } // Create an IAM role that can be used by our service's task. taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{ AssumeRolePolicy: pulumi.String(`{ "Version": "2008-10-17", "Statement": [{ "Sid": "", "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" }] }`), }) if err != nil { return err } _, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{ Role: taskExecRole.Name, PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"), }) if err != nil { return err } // Create a load balancer to listen for HTTP traffic on port 80. webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{ Subnets: toPulumiStringArray(subnet.Ids), SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()}, }) if err != nil { return err } webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{ Port: pulumi.Int(80), Protocol: pulumi.String("HTTP"), TargetType: pulumi.String("ip"), VpcId: pulumi.String(vpc.Id), }) if err != nil { return err } webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{ LoadBalancerArn: webLb.Arn, Port: pulumi.Int(80), DefaultActions: elb.ListenerDefaultActionArray{ elb.ListenerDefaultActionArgs{ Type: pulumi.String("forward"), TargetGroupArn: webTg.Arn, }, }, }) if err != nil { return err } //create a new ECR repository repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{}) if err != nil { return err } repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) { creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{ RegistryId: rid, }) if err != nil { return nil, err } data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken) if err != nil { fmt.Println("error:", err) return nil, err } return strings.Split(string(data), ":"), nil }).(pulumi.StringArrayOutput) repoUser := repoCreds.Index(pulumi.Int(0)) repoPass := repoCreds.Index(pulumi.Int(1)) //build the image image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{ Build: docker.DockerBuildArgs{ Context: pulumi.String("./static"), }, ImageName: repo.RepositoryUrl, Registry: docker.ImageRegistryArgs{ Server: repo.RepositoryUrl, Username: repoUser, Password: repoPass, }, }) if err != nil { return err } containerDef := image.ImageName.ApplyT(func(name string) (string, error) { fmtstr := `[{ "name": "my-app", "image": %q, "portMappings": [{ "containerPort": 80, "hostPort": 80, "protocol": "tcp" }] }]` return fmt.Sprintf(fmtstr, name), nil }).(pulumi.StringOutput) // Spin up a load balanced service running NGINX. appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{ Family: pulumi.String("fargate-task-definition"), Cpu: pulumi.String("256"), Memory: pulumi.String("512"), NetworkMode: pulumi.String("awsvpc"), RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")}, ExecutionRoleArn: taskExecRole.Arn, ContainerDefinitions: containerDef, }) if err != nil { return err } _, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{ Cluster: cluster.Arn, DesiredCount: pulumi.Int(5), LaunchType: pulumi.String("FARGATE"), TaskDefinition: appTask.Arn, NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{ AssignPublicIp: pulumi.Bool(true), Subnets: toPulumiStringArray(subnet.Ids), SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()}, }, LoadBalancers: ecs.ServiceLoadBalancerArray{ ecs.ServiceLoadBalancerArgs{ TargetGroupArn: webTg.Arn, ContainerName: pulumi.String("my-app"), ContainerPort: pulumi.Int(80), }, }, }, pulumi.DependsOn([]pulumi.Resource{webListener})) if err != nil { return err } // Export the resulting web address. ctx.Export("url", webLb.DnsName) return nil }) } func toPulumiStringArray(a []string) pulumi.StringArrayInput { var res []pulumi.StringInput for _, s := range a { res = append(res, pulumi.String(s)) } return pulumi.StringArray(res) } Complex? Yes, but this complexity is inherent to AWS features and not Pulumi. We would have similar complexity if we were using Terraform or CDK.
Before running our code, we need to download the new dependencies:
go get github.com/pulumi/pulumi-docker go get github.com/pulumi/pulumi-docker/sdk/v3/go/docker Now just run the command:
pulumi up The execution output will generate the URL of the load balancer, which we will use to access the contents of our container in execution.
Reorganizing the code
Now we can start making use of the advantages of a complete programming language like Go. For example, we could use language features like functions, concurrency, conditionals, etc. In this example, we are going to organize our code better. For this, I created the iac directory and the iac/fargate.go file. After that, I moved most of the logic from main.go to the new file:
package iac import ( "encoding/base64" "fmt" "strings" "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2" "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr" "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs" elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2" "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam" "github.com/pulumi/pulumi-docker/sdk/v3/go/docker" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) func FargateRun(ctx *pulumi.Context) error { // Read back the default VPC and public subnets, which we will use. t := true vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t}) if err != nil { return err } subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id}) if err != nil { return err } // Create a SecurityGroup that permits HTTP ingress and unrestricted egress. webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{ VpcId: pulumi.String(vpc.Id), Egress: ec2.SecurityGroupEgressArray{ ec2.SecurityGroupEgressArgs{ Protocol: pulumi.String("-1"), FromPort: pulumi.Int(0), ToPort: pulumi.Int(0), CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")}, }, }, Ingress: ec2.SecurityGroupIngressArray{ ec2.SecurityGroupIngressArgs{ Protocol: pulumi.String("tcp"), FromPort: pulumi.Int(80), ToPort: pulumi.Int(80), CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")}, }, }, }) if err != nil { return err } // Create an ECS cluster to run a container-based service. cluster, err := ecs.NewCluster(ctx, "app-cluster", nil) if err != nil { return err } // Create an IAM role that can be used by our service's task. taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{ AssumeRolePolicy: pulumi.String(`{ "Version": "2008-10-17", "Statement": [{ "Sid": "", "Effect": "Allow", "Principal": { "Service": "ecs-tasks.amazonaws.com" }, "Action": "sts:AssumeRole" }] }`), }) if err != nil { return err } _, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{ Role: taskExecRole.Name, PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"), }) if err != nil { return err } // Create a load balancer to listen for HTTP traffic on port 80. webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{ Subnets: toPulumiStringArray(subnet.Ids), SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()}, }) if err != nil { return err } webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{ Port: pulumi.Int(80), Protocol: pulumi.String("HTTP"), TargetType: pulumi.String("ip"), VpcId: pulumi.String(vpc.Id), }) if err != nil { return err } webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{ LoadBalancerArn: webLb.Arn, Port: pulumi.Int(80), DefaultActions: elb.ListenerDefaultActionArray{ elb.ListenerDefaultActionArgs{ Type: pulumi.String("forward"), TargetGroupArn: webTg.Arn, }, }, }) if err != nil { return err } repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{}) if err != nil { return err } repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) { creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{ RegistryId: rid, }) if err != nil { return nil, err } data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken) if err != nil { fmt.Println("error:", err) return nil, err } return strings.Split(string(data), ":"), nil }).(pulumi.StringArrayOutput) repoUser := repoCreds.Index(pulumi.Int(0)) repoPass := repoCreds.Index(pulumi.Int(1)) image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{ Build: docker.DockerBuildArgs{ Context: pulumi.String("./static"), }, ImageName: repo.RepositoryUrl, Registry: docker.ImageRegistryArgs{ Server: repo.RepositoryUrl, Username: repoUser, Password: repoPass, }, }) if err != nil { return err } containerDef := image.ImageName.ApplyT(func(name string) (string, error) { fmtstr := `[{ "name": "my-app", "image": %q, "portMappings": [{ "containerPort": 80, "hostPort": 80, "protocol": "tcp" }] }]` return fmt.Sprintf(fmtstr, name), nil }).(pulumi.StringOutput) // Spin up a load balanced service running NGINX. appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{ Family: pulumi.String("fargate-task-definition"), Cpu: pulumi.String("256"), Memory: pulumi.String("512"), NetworkMode: pulumi.String("awsvpc"), RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")}, ExecutionRoleArn: taskExecRole.Arn, ContainerDefinitions: containerDef, }) if err != nil { return err } _, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{ Cluster: cluster.Arn, DesiredCount: pulumi.Int(5), LaunchType: pulumi.String("FARGATE"), TaskDefinition: appTask.Arn, NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{ AssignPublicIp: pulumi.Bool(true), Subnets: toPulumiStringArray(subnet.Ids), SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()}, }, LoadBalancers: ecs.ServiceLoadBalancerArray{ ecs.ServiceLoadBalancerArgs{ TargetGroupArn: webTg.Arn, ContainerName: pulumi.String("my-app"), ContainerPort: pulumi.Int(80), }, }, }, pulumi.DependsOn([]pulumi.Resource{webListener})) if err != nil { return err } // Export the resulting web address. ctx.Export("url", webLb.DnsName) return nil } func toPulumiStringArray(a []string) pulumi.StringArrayInput { var res []pulumi.StringInput for _, s := range a { res = append(res, pulumi.String(s)) } return pulumi.StringArray(res) } The next step was to configure the iac directory to be a Go language module:
cd iac go mod init github.com/eminetto/post-pulumi/iac cd .. go mod edit -replace github.com/eminetto/post-pulumi/iac=./iac go mod tidy Our main.go can now be simplified:
package main import ( "github.com/eminetto/post-pulumi/iac" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) func main() { pulumi.Run(func(ctx *pulumi.Context) error { return iac.FargateRun(ctx) }) } That way, we can better manage the structure of the code that will handle AWS resources. We can reuse this code in other projects, use environment variables, write tests, or whatever else our imagination allows.
Conclusion
Using a tool like Pulumi significantly increases the range of options that we can use in building a project's infrastructure while maintaining readability, code reuse and organization.
Top comments (4)
Nice post, thanks.
At the beginning you miss to mention that the AWS CDK as well as the terraform CDK also support GO!
See some examples
go-on-aws.com/infrastructure-as-go...
Nice! I will test and write another post :)
Thanks!
As a gopher and cdk user I wrote an intro myself: go-on-aws.com/infrastructure-as-go...
thanks for sharing