Create a CodePipeline with CloudFormation
In this article we’ll take a look at AWS CodePipeline, a continuos deployment and integration platform, to deploy updates to an existing Lambda function. We will use CloudFormation to create each step of our CodePipeline, source, build, and deploy. For our source stage, we will use a zip file in S3. Our build stage will use AWS CodeBuild, another managed service of AWS to allow us to build code in the cloud, and finally an existing CloudFormation to deploy our code updates.
AWS CodePipeline
With AWS CodePipeline we can implement software development best practices such as automated builds, tests, and continuos code deployment all within a uniform platform. And because CodePipeline is a managed AWS service, there aren’t any servers to setup, and CodePipeline can natively interact with many other AWS services to release changes to applications hosted in AWS.
CodePipelines generally consist of a source, a build stage, and a deployment stage. As with most AWS services we can set this service up using the web console, CLI tools, or CloudFormation. In this article we are going to use CloudFormation scripts. Once our pipeline is setup we can use the AWS web console to have a visual representation of how our code is being deployed in each step.
Prerequisites
An AWS account with sufficient permissions, along with CLI access. Additionally, CodePipeline requires additional setup before we can use CloudFormation to deploy a pipeline:
Deploying to Lambda
In this article we will be using a CodePipeline to deploy updates to an existing Lambda function. In a previous article, Create a Lambda Function with CloudFormation, we created a new Lambda function with CloudFormation. In this article we are going to use CodePipelines to deploy updated code to this function. For CodePipeline to be able to deploy code to a Lambda function, the function needs to be part of a CloudFormation stack, and the name of this stack will be required to create a CodePipeline.
Building Lambda with CodeBuild
During the build phase of CodePipeline we will use AWS CodeBuild to run NPM install on our NodeJS Lambda application. While this step is not required to use CodePipelines, it is helpful as most modern web applications will need to be built in some fashion.
To use AWS CodeBuild, your Lambda source must have a buildspec.yml file included in the .zip file uploaded to S3. For more information about working with Lambda and CodeBuild see this article. Prepare a NodeJS Lambda Function for CodePipeline.
CodePipeline IAM Role
The IAM role CodePipeline will use needs to be pre-defined before we can create a pipeline with CloudFormation. This can be an existing CodePipeline role, or a AWS generated role. More details can be found here: Create an IAM Role to use with CodePipeline and from AWS documentation. You will need the ARN of this role to create a CodePipeline with CloudFormation.
CloudFormation IAM Role
Because we are using CloudFormation to manage updates to our NodeJS application, we will need an IAM role with permissions to allow CloudFormation to create and manage AWS resources. CloudFormation can use your existing user’s IAM role to manage the stack, but best security practices dictate that we should use roles that only have access to the specific resources the role needs. For more information about creating a role for our CloudFormation stack, see the article Create an IAM Role for CloudFormation or AWS Documentation.
CodeBuild IAM Role
The CodePipeline we are setting up with use CodeBuild as a build step when deploying our application, and the CodeBuild project will need an IAM user. This user will need permissions to perform CodeBuild tasks, write to S3, and write to CloudWatch logs. For more information about creating a role for CodeBuild, see this article: Create a Reusable IAM Role for AWS CodeBuild or CodeBuild Access Control AWS Documentation.
S3 Structure
An S3 bucket is required for CodePipeline to build source files into, which are called artifacts. We also use this bucket as the source for our new Lambda code, in a zip file. I’ve created a build folder to store our Lambda function source code. CodePipeline will create another folder store artifacts automatically.
Also note, S3 buckets will need versioning enabled in order to work with CodePipeline.
CloudFormation Template - CodePipeline.
We’ll start out creating our CloudFormation template with defining our CodePipeline resource type AWS::CodePipeline::Pipeline. I’m calling my resource LambdaCodePipeline, and saving this file as CFTemplate.json:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"LambdaCodePipeline": {
"Type" : "AWS::CodePipeline::Pipeline",
"Properties" : {
"ArtifactStore" : {
"Type": "S3",
"Location": "cm-lambda-code-deploy"
},
"Name" : "HelloWorldCodePipeline",
"RoleArn": "arn:aws:iam::account-id:role/CodePipelineRole",
"Stages" : []
}
}
}
}
Important properties of our AWS::CodePipeline::Pipeline resource include the following:
- ArtifactStore: The type and location of where CodePipeline will store our build results. In this case an S3 bucket.
- Name: The name of the CodePipeline. Not required but useful.
- RoleArn: The IAM role that will be executing the pipeline. NOTE: You’ll need to replace your account ID here.
- Stages: An array of Stage objects defined in our pipeline, source, build, and deploy. Will define late in our template.
Now that we have our template started, we can create each Stage object for our CodePipeline deployment
CodePipeline Stages - Source
The first stage we need to define is where to get our updated source code. In this example we will use a .zip file in S3. Another common source is a GIT repository (Github, Bitbucket, etc). The bucket I’m using is cm-lambda-code-deploy and the .zip file is located in builds/hello-word.zip:
{
"Name": "Source",
"Actions": [
{
"Name": "Source",
"ActionTypeId": {
"Category": "Source",
"Owner": "AWS",
"Provider": "S3",
"Version": "1"
},
"Configuration": {
"PollForSourceChanges": "false",
"S3Bucket": "cm-lambda-code-deploy",
"S3ObjectKey": "builds/hello-world.zip"
},
"OutputArtifacts": [
{
"Name": "SourceArtifact"
}
]
}
]
}
Actions and actionTypeId Each CodePipeline stage must consist of actions. We’ve defined our first action as “name”:“Source”. Each action must be of a predefined category, defined in actionTypeId, in our case, Source.
Each action type also has its own configuration items specific to each type. In this example, since we are using S3 for our source. I’ve set the configuration to use a specified S3 bucket and key, in this case a .zip file for our source which is located in the builds directory.
CodePipeline Stages - Build
Next section of our CodePipeline is the build stage. This stage is used to mainly used to install my NPM packages, so I do not have to commit them to GIT. CodeBuild allows us to perform such tasks in a serverless manner.
Add the following Stage object to your Stages array in the CloudFormation template:
{
"Name": "Build",
"Actions": [
{
"Name": "Build",
"ActionTypeId": {
"Category": "Build",
"Owner": "AWS",
"Provider": "CodeBuild",
"Version": "1"
},
"RunOrder": 1,
"Configuration": {
"BatchEnabled": "false",
"ProjectName": { "Fn::GetAtt" : [ "HelloWorldCodeBuild" ,"Arn" ] }
},
"OutputArtifacts": [
{
"Name": "BuildArtifact"
}
],
"InputArtifacts": [
{
"Name": "SourceArtifact"
}
],
"Namespace": "BuildVariables"
}
]
},
We now have a build step that is dependent on the CodeBuild project HelloWorldCodeBuild, we’ll need to add that resource to our CloudFormation template. This is defined in the Build section using the CloudFormation Ref function.
CodeBuild Resource
The CodeBuild resource has to exist before we can create our CodePipeline, so we’ve used the Ref function to tell CloudFormation to create this resource first. Add the following CodeBuild resource to your CloudFormation template:
"HelloWorldCodeBuild": {
"Type" : "AWS::CodeBuild::Project",
"Properties" : {
"Name": "NodeJSBuild",
"Source": {
"Type": "CODEPIPELINE"
},
"Artifacts": {
"Type": "CODEPIPELINE",
"Name": "HelloWorldCodePipeline"
},
"Environment": {
"Type": "LINUX_CONTAINER",
"Image": "aws/codebuild/standard:4.0",
"ComputeType": "BUILD_GENERAL1_SMALL",
"ImagePullCredentialsType": "CODEBUILD"
},
"ServiceRole": "arn:aws:iam::account-id:role/CodeBuildRole",
"TimeoutInMinutes": 60,
"QueuedTimeoutInMinutes": 480,
"Tags": [],
"LogsConfig": {
"CloudWatchLogs": {
"Status": "ENABLED"
},
"S3Logs": {
"Status": "DISABLED",
"EncryptionDisabled": false
}
}
}
}
Note: You’ll need to replace your account ID in the ServiceRole section.
With our build section complete now we can fill out our last stage of the CodePipeline, deploy.
CodePipeline Stages - Deploy
The last step in the pipeline is to deploy our code. As mentioned we will use a predefined CloudFormation template that contains our Lambda. Using this template we can create and execute a changeset in CloudFormation using two CodePipeline actions. The first action will be to create a changeset, and then the second action will be to execute the changeset:
{
"Name": "Deploy",
"Actions": [
{
"Name": "create-changeset",
"ActionTypeId": {
"Category": "Deploy",
"Owner": "AWS",
"Provider": "CloudFormation",
"Version": "1"
},
"RunOrder": 1,
"Configuration": {
"ActionMode": "CHANGE_SET_REPLACE",
"Capabilities": "CAPABILITY_NAMED_IAM,CAPABILITY_AUTO_EXPAND",
"ChangeSetName": "HelloWorld-changeset",
"RoleArn": "arn:aws:iam::account-id:role/HelloWorldCloudFormation",
"StackName": "HelloWorld",
"TemplatePath": "BuildArtifact::outputtemplate.yml"
},
"OutputArtifacts": [],
"InputArtifacts": [
{
"Name": "BuildArtifact"
}
],
"Namespace": "DeployVariables"
},
{
"Name": "execute-changeset",
"ActionTypeId": {
"Category": "Deploy",
"Owner": "AWS",
"Provider": "CloudFormation",
"Version": "1"
},
"RunOrder": 2,
"Configuration": {
"ActionMode": "CHANGE_SET_EXECUTE",
"ChangeSetName": "HelloWorld-changeset",
"StackName": "HelloWorld"
},
"OutputArtifacts": [],
"InputArtifacts": [
{
"Name": "BuildArtifact"
}
]
}
]
}
Create CodePipeline
We now have enough of our CloudFormation template to build create our stack and CodePipeline. Here is our completed template. Save this file as CFTemplate.json:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"LambdaCodePipeline": {
"Type": "AWS::CodePipeline::Pipeline",
"Properties": {
"ArtifactStore": {
"Type": "S3",
"Location": "cm-lambda-code-deploy"
},
"Name": "HelloWorldCodePipeline",
"RoleArn": "arn:aws:iam::account-id:role/CodePipelineRole",
"Stages": [
{
"Name": "Source",
"Actions": [
{
"Name": "Source",
"ActionTypeId": {
"Category": "Source",
"Owner": "AWS",
"Provider": "S3",
"Version": "1"
},
"Configuration": {
"PollForSourceChanges": "false",
"S3Bucket": "cm-lambda-code-deploy",
"S3ObjectKey": "builds/hello-world.zip"
},
"OutputArtifacts": [
{
"Name": "SourceArtifact"
}
]
}
]
},
{
"Name": "Build",
"Actions": [
{
"Name": "Build",
"ActionTypeId": {
"Category": "Build",
"Owner": "AWS",
"Provider": "CodeBuild",
"Version": "1"
},
"RunOrder": 1,
"Configuration": {
"BatchEnabled": "false",
"ProjectName": { "Fn::GetAtt" : [ "HelloWorldCodeBuild" ,"Arn" ] }
},
"OutputArtifacts": [
{
"Name": "BuildArtifact"
}
],
"InputArtifacts": [
{
"Name": "SourceArtifact"
}
],
"Namespace": "BuildVariables"
}
]
},
{
"Name": "Deploy",
"Actions": [
{
"Name": "create-changeset",
"ActionTypeId": {
"Category": "Deploy",
"Owner": "AWS",
"Provider": "CloudFormation",
"Version": "1"
},
"RunOrder": 1,
"Configuration": {
"ActionMode": "CHANGE_SET_REPLACE",
"Capabilities": "CAPABILITY_NAMED_IAM,CAPABILITY_AUTO_EXPAND",
"ChangeSetName": "HelloWorld-changeset",
"RoleArn": "arn:aws:iam::account-id:role/HelloWorldCloudFormation",
"StackName": "HelloWorld",
"TemplatePath": "BuildArtifact::outputtemplate.yml"
},
"OutputArtifacts": [],
"InputArtifacts": [
{
"Name": "BuildArtifact"
}
],
"Namespace": "DeployVariables"
},
{
"Name": "execute-changeset",
"ActionTypeId": {
"Category": "Deploy",
"Owner": "AWS",
"Provider": "CloudFormation",
"Version": "1"
},
"RunOrder": 2,
"Configuration": {
"ActionMode": "CHANGE_SET_EXECUTE",
"ChangeSetName": "HelloWorld-changeset",
"StackName": "HelloWorld"
},
"OutputArtifacts": [],
"InputArtifacts": [
{
"Name": "BuildArtifact"
}
]
}
]
}
]
}
},
"HelloWorldCodeBuild": {
"Type" : "AWS::CodeBuild::Project",
"Properties" : {
"Name": "NodeJSBuild",
"Source": {
"Type": "CODEPIPELINE"
},
"Artifacts": {
"Type": "CODEPIPELINE",
"Name": "HelloWorldCodePipeline"
},
"Environment": {
"Type": "LINUX_CONTAINER",
"Image": "aws/codebuild/standard:4.0",
"ComputeType": "BUILD_GENERAL1_SMALL",
"ImagePullCredentialsType": "CODEBUILD"
},
"ServiceRole": "arn:aws:iam::account-id:role/CodeBuildRole",
"TimeoutInMinutes": 60,
"QueuedTimeoutInMinutes": 480,
"Tags": [],
"LogsConfig": {
"CloudWatchLogs": {
"Status": "ENABLED"
},
"S3Logs": {
"Status": "DISABLED",
"EncryptionDisabled": false
}
}
}
}
}
}
Create our Pipeline
Now we can run the command with AWS CLI:
aws cloudformation create-stack --stack-name HelloWorldCodePipeline --template-body file://CFTemplate.json
If the call was successful you will get back a StackId with an arn of your newly created CloudFormation stack.
CloudFormation Errors
If your stack fails to create any of the resources defined in your CloudFormation template then the stack will end up in an error state. For more information about working with CloudFormation and debugging read our CloudFormation Quick Guide.
Don’t forget about account ID If you are seeing permissions errors ensure that you’ve replaced account ID in all sections.
Conclusion
After creating each required user and resource, and setting up each section of our CodePipeline, including source, build, and deploy, we can use our pipeline to deploy code to Lambda. If you are running into any issues creating your pipelines with Cloudformation, drop a comment below I can help troubleshoot.