In larger organizations a lot of things are in place. Think of things like deployment pipelines for your applications. But how do you set those up using the services AWS provides? In this blog post I will guide you to how I do it for my pet projects.
I am a big fan of CloudFormation so the examples I will show are CloudFormation snippets.
The git repository
I am using AWS CodeCommit as a source repository. The reason for this choice is that it integrates with AWS CodePipeline. We also want the ability to test our infrastructure. So I usually use feature branches to do that. Here the first challange appears. We only need 1 repository and a pipeline to deploy our infrastructure. But we also need a pipeline that deploys our feature branch. In CloudFormation you can do this using conditions.
Parameters: FeatureGitBranch: Type: String Default: "" ProjectName: Type: String Default: "" Conditions: IsMainBranchPipeline: !Equals [!Ref FeatureGitBranch, ""] IsFeatureBranchPipeline: !Not [Condition: IsMainBranchPipeline] By providing a FeatureGitBranch parameter we can now make some choices. For example we could only create the repository in the production pipeline.
Resources: PipelineRepo: Condition: IsMainBranchPipeline Type: AWS::CodeCommit::Repository Properties: RepositoryName: !Ref ProjectName RepositoryDescription: <repository-description> The pipeline requirements
The pipeline uses a couple of resources. You can share these resources across your pipelines. So they are in a separate template and I deploy these resources once per region that I am using.
First you will need a S3 bucket to store the artifacts created by each stage.
ArtifactsBucket: Type: AWS::S3::Bucket DeletionPolicy: Retain UpdateReplacePolicy: Retain Properties: BucketName: !Sub codepipeline-artifacts-${AWS::AccountId}-${AWS::Region} VersioningConfiguration: Status: Enabled BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: SSEAlgorithm: AES256 I am using a predictable bucket name so I can use it in my pipeline. Because bucket names need to have a globally unique name I also include the account id and region.
You will also need an IAM role. You will use this role in your pipeline to deploy your CloudFormation template.
CloudFormationServiceRole: Type: AWS::IAM::Role Properties: RoleName: !Sub cloudformation-execution-role-${AWS::Region} AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Action: sts:AssumeRole Effect: Allow Principal: { Service: cloudformation.amazonaws.com } ManagedPolicyArns: - arn:aws:iam::aws:policy/AdministratorAccess Again I use a predictable name and I included the region. Since role names are unique in your account you can deploy this in every region. For the sake of simplicity I used the AdministratorAccess managed policy. You might want to limit this for your own scope.
The pipeline
The pipeline itself also needs a role. This role can pass the CloudFormation role and use the buckets created in the previous step.
CodePipelineExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Action: "sts:AssumeRole" Effect: Allow Principal: Service: - codepipeline.amazonaws.com Policies: - PolicyName: CodePipelineAccess PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: "iam:PassRole" Resource: !Sub cloudformation-execution-role-${AWS::Region} - PolicyName: CodePipelineCodeAndS3Bucket PolicyDocument: Version: "2012-10-17" Statement: - Action: - s3:GetBucketAcl - s3:GetBucketLocation Effect: Allow Resource: - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${AWS::AccountId}-eu-west-1 - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${AWS::AccountId}-eu-central-1 - Action: - "s3:GetObject" - "s3:GetObjectVersion" - "s3:PutObject" Effect: Allow Resource: - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${AWS::AccountId}-eu-west-1/* - !Sub arn:${AWS::Partition}:s3:::codepipeline-artifacts-${AWS::AccountId}-eu-central-1/* - PolicyName: CodeCommitAccess PolicyDocument: Version: "2012-10-17" Statement: - Action: - codecommit:GitPull - codecommit:GetBranch - codecommit:GetCommit - codecommit:UploadArchive - codecommit:GetUploadArchiveStatus Effect: Allow Resource: - !Sub arn:${AWS::Partition}:codecommit:${AWS::Region}:${AWS::AccountId}:${ProjectName} - !Sub arn:${AWS::Partition}:codecommit:${AWS::Region}:${AWS::AccountId}:${ProjectName}/* - PolicyName: CodePipelineCodeBuildAndCloudformationAccess PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - "codebuild:StartBuild" - "codebuild:BatchGetBuilds" Resource: - !GetAtt CodeBuildProjectBuildAndPackage.Arn - Effect: Allow Action: - "cloudformation:DescribeStacks" Resource: !Sub "arn:${AWS::Partition}:cloudformation:eu-*:${AWS::AccountId}:stack/*" - Effect: Allow Action: - "cloudformation:CreateStack" - "cloudformation:DeleteStack" - "cloudformation:UpdateStack" - "cloudformation:CreateChangeSet" - "cloudformation:ExecuteChangeSet" - "cloudformation:DeleteChangeSet" - "cloudformation:DescribeChangeSet" - "cloudformation:SetStackPolicy" - "cloudformation:SetStackPolicy" - "cloudformation:ValidateTemplate" Resource: - !Sub "arn:${AWS::Partition}:cloudformation:eu-*:${AWS::AccountId}:stack/${ProjectName}-*/*" As you can see in the snippet the role is also allowed to start a CodeBuild project. And it can deploy a CloudFormation template.
You might have noticed that I used 2 regions in this role. This is because in this particular example I have a pipeline that uses 2 regions.
Because we use 2 regions we also need 2 artifact stores.
Pipeline: Type: AWS::CodePipeline::Pipeline Properties: ArtifactStores: - Region: eu-west-1 ArtifactStore: Location: !Sub codepipeline-artifacts-${AWS::AccountId}-eu-west-1 Type: S3 - Region: eu-central-1 ArtifactStore: Location: !Sub codepipeline-artifacts-${AWS::AccountId}-eu-central-1 Type: S3 RoleArn: !GetAtt CodePipelineExecutionRole.Arn RestartExecutionOnUpdate: true Stages: - Name: Source Actions: - Name: SourceCodeRepo ActionTypeId: Category: Source Owner: AWS Version: 1 Provider: CodeCommit Configuration: PollForSourceChanges: False RepositoryName: !Ref ProjectName BranchName: !If [IsFeatureBranchPipeline, !Ref FeatureGitBranch, "main"] OutputArtifacts: - Name: SourceCodeAsZip RunOrder: 1 Depending on the IsFeatureBranchPipeline condition we select the correct branch for this pipeline. Depending on your project you might need a build step. I use CodeBuild to run unit tests and build and package any resources needed.
- Name: BuildAndPackage Actions: - Name: CodeBuild ActionTypeId: Category: Build Owner: AWS Provider: CodeBuild Version: "1" Configuration: ProjectName: !Ref CodeBuildProjectBuildAndPackage InputArtifacts: - Name: SourceCodeAsZip OutputArtifacts: - Name: BuildArtifactAsZip I did not include the CodeBuild project and IAM role snippet for CodeBuild. I will do a blog post on that in the future.
So lets go over to the fun part. The next stage is the actual deployment of the CloudFormation template. I am using 2 regions and I only want to deploy to these 2 regions for my production environment. I also have different parameters for my production environment. Again, I am using conditions to control this behaviour.
- !If - IsMainBranchPipeline - Name: Production Actions: - Name: Ireland-CreateChangeSet RunOrder: 1 Region: eu-west-1 InputArtifacts: - Name: BuildArtifactAsZip ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: "1" Configuration: ActionMode: CHANGE_SET_REPLACE RoleArn: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role StackName: !Sub ${ProjectName}-production ChangeSetName: !Sub ${ProjectName}-production-ChangeSet TemplatePath: BuildArtifactAsZip::packaged-template.yaml Capabilities: CAPABILITY_NAMED_IAM ParameterOverrides: |- { "EnvType": "production" } - Name: Frankfurt-CreateChangeSet RunOrder: 1 Region: eu-central-1 InputArtifacts: - Name: BuildArtifactAsZip ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: "1" Configuration: ActionMode: CHANGE_SET_REPLACE RoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/cloudformation-execution-role StackName: !Sub ${ProjectName}-production ChangeSetName: !Sub ${ProjectName}-production-ChangeSet TemplatePath: BuildArtifactAsZip::packaged-template.yaml Capabilities: CAPABILITY_NAMED_IAM ParameterOverrides: | { "EnvType": "production" } - Name: Ireland-ExecuteChangeSet RunOrder: 2 Region: eu-west-1 ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: "1" Configuration: ActionMode: CHANGE_SET_EXECUTE RoleArn: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role StackName: !Sub ${ProjectName}-production ChangeSetName: !Sub ${ProjectName}-production-ChangeSet - Name: Frankfurt-ExecuteChangeSet RunOrder: 2 Region: eu-central-1 ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: "1" Configuration: ActionMode: CHANGE_SET_EXECUTE RoleArn: !Sub arn:aws:iam::${AWS::AccountId}:role/cloudformation-execution-role StackName: !Sub ${ProjectName}-production ChangeSetName: !Sub ${ProjectName}-production-ChangeSet - !Ref AWS::NoValue In the 1 action I create a change set. In the second action I execute it. Please note the RoleArn used here. You could also specify a role from a different account. This allows you to use a separate account for different environments. For example you could host the pipeline in a deployment account. And have separate development, testing, acceptance and production accounts.
The feature branch
So we now have a working pipeline for the production environment. How about the feature branch? For this we use again the conditions.
- !If - IsFeatureBranchPipeline - Name: FeatureDevelopment Actions: - Name: Ireland-CreateChangeSet RunOrder: 1 Region: eu-west-1 InputArtifacts: - Name: BuildArtifactAsZip ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: "1" Configuration: ActionMode: CHANGE_SET_REPLACE RoleArn: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role StackName: !Sub ${ProjectName}-${FeatureName} ChangeSetName: !Sub ${ProjectName}-feature-ChangeSet TemplatePath: BuildArtifactAsZip::packaged-template.yaml Capabilities: CAPABILITY_NAMED_IAM ParameterOverrides: | { "EnvType": "development" } - Name: Ireland-ExecuteChangeSet RunOrder: 2 Region: eu-west-1 ActionTypeId: Category: Deploy Owner: AWS Provider: CloudFormation Version: "1" Configuration: ActionMode: CHANGE_SET_EXECUTE RoleArn: !Sub arn:${AWS::Partition}:iam::${AWS::AccountId}:role/cloudformation-execution-role StackName: !Sub ${ProjectName}-${FeatureName} ChangeSetName: !Sub ${ProjectName}-feature-ChangeSet - !Ref AWS::NoValue As you can see, the feature branch is only deployed to the eu-west-1 region. And it has set the EnvType parameter to development.
Pickup new commits
When you push your commits to the CodeCommit repository. You can use an event rule to trigger a new pipeline execution. To make this happen we need an IAM role and the rule itself.
CloudWatchEventRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: { Service: events.amazonaws.com } Action: sts:AssumeRole Policies: - PolicyName: root PolicyDocument: Version: '2012-10-17' Statement: - Action: codepipeline:StartPipelineExecution Resource: !Sub arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline} Effect: Allow CloudWatchEventRule: Type: AWS::Events::Rule Properties: Description: Check CodeCommit Repo Changes EventPattern: detail-type: - "CodeCommit Repository State Change" source: - aws.codecommit resources: - !Sub arn:${AWS::Partition}:codecommit:${AWS::Region}:${AWS::AccountId}:${ProjectName} detail: event: anything-but: - referenceDeleted referenceType: - branch referenceName: - !If [IsFeatureBranchPipeline, !Ref FeatureGitBranch, "main"] Targets: - Id: codepipeline Arn: !Sub arn:${AWS::Partition}:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline} RoleArn: !Sub ${CloudWatchEventRole.Arn} We are triggering on every event except when the branch removal.
Conclusion
You can store the template for the pipeline in your CodeCommit repository. When you create a feature branch you can deploy that template specifying the branch name. The pipeline can then deploy the project CloudFormation template. And with every commit that you push your feature environment will receive an update.
With this setup you have everything in a single repository. (Except for the shared S3 bucket and IAM role.) Having everything in a single repository makes it easier to maintain.
Top comments (0)