akhil mittalAWS CloudFormation is a service that allows you to define and manage your AWS infrastructure as code. With CloudFormation, you can create, update, and delete AWS resources in a predictable and repeatable manner using templates written in JSON or YAML format.
Here's a detailed explanation of CloudFormation, how to use it, and a breakdown of a well-structured CloudFormation template.
What is AWS CloudFormation?
Overview
AWS CloudFormation enables you to automate and manage AWS resources, including EC2 instances, RDS databases, VPCs, IAM roles, and many other services. CloudFormation uses templates that describe the desired state of your infrastructure, and then AWS takes care of provisioning and configuring those resources.
Key Benefits
Infrastructure as Code (IaC): Define your entire AWS infrastructure in code, making it easy to version, replicate, and maintain.
Automation: Automates the creation and management of AWS resources, reducing the need for manual intervention.
Consistency: Ensures consistent setups across environments, such as dev, test, and production.
Rollback Capabilities: Automatically rolls back to the last known good state if stack creation or updates fail.
How to Use CloudFormation
Step 1: Create a CloudFormation Template
Create a template in JSON or YAML that describes the AWS resources you want to create. This file acts as the blueprint for your infrastructure.
Step 2: Upload the Template to AWS CloudFormation
You can create a stack using the AWS Management Console, AWS CLI, or AWS SDKs. A stack is a collection of AWS resources you create and manage as a single unit.
Step 3: CloudFormation Provisions Resources
CloudFormation provisions and configures your AWS resources based on the template.
Step 4: Monitor and Manage the Stack
You can monitor the stack's progress through the AWS Console, review events, and check the status of the resources being provisioned.
Step 5: Update or Delete the Stack
If you need to make changes, update the template and use CloudFormation to apply the changes. When resources are no longer needed, you can delete the stack to remove all associated resources.
Detailed Walkthrough of a Complex CloudFormation Template
Below is a detailed CloudFormation template example with explanations of each section. This template sets up a highly available web application stack, including VPC, subnets, an Auto Scaling group with an Application Load Balancer, and an RDS database.
AWSTemplateFormatVersion: '2010-09-09'
Description: >
Complex CloudFormation template for a highly available web application stack with VPC, ALB, Auto Scaling, and RDS.
Parameters:
EnvironmentName:
Type: String
Default: production
Description: The environment name (e.g., dev, staging, production).
VpcCIDR:
Type: String
Default: 10.0.0.0/16
Description: CIDR block for the VPC.
PublicSubnet1CIDR:
Type: String
Default: 10.0.1.0/24
Description: CIDR block for the first public subnet.
PublicSubnet2CIDR:
Type: String
Default: 10.0.2.0/24
Description: CIDR block for the second public subnet.
DBInstanceType:
Type: String
Default: db.t3.micro
AllowedValues:
- db.t2.micro
- db.t3.micro
- db.t3.small
- db.t3.medium
Description: RDS DB instance type.
KeyName:
Description: Name of an existing EC2 KeyPair to enable SSH access.
Type: AWS::EC2::KeyPair::KeyName
DBUsername:
NoEcho: true
Description: The database admin account username.
Type: String
MinLength: 1
MaxLength: 16
AllowedPattern: "[a-zA-Z][a-zA-Z0-9]*"
ConstraintDescription: must begin with a letter and contain only alphanumeric characters.
DBPassword:
NoEcho: true
Description: The database admin account password.
Type: String
MinLength: 8
MaxLength: 41
AllowedPattern: "[a-zA-Z0-9]*"
ConstraintDescription: must contain only alphanumeric characters.
Mappings:
RegionMap:
us-east-1:
AMI: ami-0c55b159cbfafe1f0
us-west-2:
AMI: ami-0bdb828fd58c52235
Conditions:
CreateProdResources: !Equals [ !Ref EnvironmentName, "production" ]
Resources:
VPC:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcCIDR
EnableDnsSupport: true
EnableDnsHostnames: true
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-VPC"
PublicSubnet1:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
CidrBlock: !Ref PublicSubnet1CIDR
MapPublicIpOnLaunch: true
AvailabilityZone: !Select [ 0, !GetAZs "" ]
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-PublicSubnet1"
PublicSubnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref VPC
CidrBlock: !Ref PublicSubnet2CIDR
MapPublicIpOnLaunch: true
AvailabilityZone: !Select [ 1, !GetAZs "" ]
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-PublicSubnet2"
InternetGateway:
Type: AWS::EC2::InternetGateway
Properties:
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-IGW"
VPCGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
VpcId: !Ref VPC
InternetGatewayId: !Ref InternetGateway
PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-PublicRT"
PublicRoute:
Type: AWS::EC2::Route
DependsOn: VPCGatewayAttachment
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
SubnetRouteTableAssociation1:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet1
RouteTableId: !Ref PublicRouteTable
SubnetRouteTableAssociation2:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PublicSubnet2
RouteTableId: !Ref PublicRouteTable
ALB:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Name: !Sub "${EnvironmentName}-ALB"
Subnets:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
Scheme: internet-facing
SecurityGroups:
- !Ref ALBSecurityGroup
LoadBalancerAttributes:
- Key: idle_timeout.timeout_seconds
Value: '60'
ALBSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allow HTTP and HTTPS traffic to the ALB
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0
- IpProtocol: tcp
FromPort: 443
ToPort: 443
CidrIp: 0.0.0.0/0
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-ALB-SG"
LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: !Sub "${EnvironmentName}-LaunchTemplate"
LaunchTemplateData:
InstanceType: t3.micro
KeyName: !Ref KeyName
SecurityGroupIds:
- !Ref InstanceSecurityGroup
ImageId: !FindInMap [ RegionMap, !Ref "AWS::Region", AMI ]
AutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
VPCZoneIdentifier:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
LaunchTemplate:
LaunchTemplateId: !Ref LaunchTemplate
Version: !GetAtt LaunchTemplate.LatestVersionNumber
MinSize: '1'
MaxSize: '3'
DesiredCapacity: '2'
TargetGroupARNs:
- !Ref ALBTargetGroup
ALBTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
VpcId: !Ref VPC
Port: 80
Protocol: HTTP
TargetType: instance
HealthCheckPath: /
HealthCheckIntervalSeconds: 30
HealthCheckTimeoutSeconds: 5
HealthyThresholdCount: 3
UnhealthyThresholdCount: 2
InstanceSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Enable SSH access and HTTP from ALB
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0
- IpProtocol: tcp
FromPort: 80
ToPort: 80
SourceSecurityGroupId: !Ref ALBSecurityGroup
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-Instance-SG"
RDSInstance:
Type: AWS::RDS::DBInstance
Properties:
DBInstanceClass: !Ref DBInstanceType
Engine: mysql
EngineVersion: '8.0'
MasterUsername: !Ref DBUsername
MasterUserPassword: !Ref DBPassword
AllocatedStorage: '20'
BackupRetentionPeriod: 7
VPCSecurityGroups:
- !Ref RDSSecurityGroup
DBSubnetGroupName: !Ref DBSubnetGroup
MultiAZ: !If [ CreateProdResources, true, false ]
StorageEncrypted: true
RDSSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Allow MySQL access from app instances
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 3306
ToPort: 3306
SourceSecurityGroupId: !Ref InstanceSecurityGroup
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-RDS-SG"
DBSubnetGroup:
Type: AWS::RDS::DBSubnetGroup
Properties:
DBSubnetGroupDescription: Subnets available for the RDS DB Instance
SubnetIds:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
Tags:
- Key: Name
Value: !Sub "${EnvironmentName}-DBSubnetGroup"
Outputs:
ALBEndpoint:
Description: The endpoint URL of the Application Load Balancer.
Value: !GetAtt ALB.DNSName
DBEndpoint:
Description: The endpoint address of the RDS instance.
Value: !GetAtt RDSInstance.Endpoint.Address
Detailed Explanation of the Template Sections
1) AWSTemplateFormatVersion: Specifies the template version. The format version 2010-09-09 is commonly used and supports all modern features.
2) Description: Provides a description of what the template does. This is useful for documentation purposes.
3) Metadata: Defines the UI experience when the template is used in the AWS Management Console. It can include information like grouping parameters for easier input.
4) Parameters: Defines inputs for the template, allowing it to be reusable. For example, VpcCIDR allows users to specify a different IP range for the VPC.
5) Mappings: Defines static values for different environments or regions. Here, RegionMap provides AMI IDs for different AWS regions.
6) Resources: The core of the template where all AWS resources are defined. In this example:
A VPC, subnet, internet gateway, and route table are created.
An EC2 instance is launched with SSH access enabled through a security group.
7) Outputs: Defines values to be returned after the stack is created, such as the instance ID and public IP address, which can be useful for further automation or integration.
Components of a Complex CloudFormation Template
A complex CloudFormation template is typically used to provision sophisticated and large-scale infrastructure, which may include:
1) Multiple VPCs and Subnets: Setting up multiple Virtual Private Clouds with associated subnets for isolated network setups.
2) Application Load Balancers and Auto Scaling: Configuring load balancers and auto-scaling groups to manage incoming traffic and scale resources automatically.
3) RDS Databases: Setting up relational databases with backups, encryption, and multi-AZ deployments.
4) IAM Roles, Policies, and Security Groups: Managing access and permissions with Identity and Access Management (IAM) and defining security groups for resource access control.
5) Nested Stacks: Using nested stacks to break down the main stack into manageable and reusable sub-stacks.
6) Custom Resources and Lambda Functions: Extending CloudFormation capabilities with custom logic executed through AWS Lambda.Custom resources in AWS CloudFormation allow you to extend the capabilities of CloudFormation beyond its built-in resource types. This is particularly useful when you need to manage resources not directly supported by CloudFormation, or when you need to perform custom actions during the stack's lifecycle, such as configuration management, integrations, or provisioning third-party services.
Example Template:
AWSTemplateFormatVersion: '2010-09-09'
Description: CloudFormation template with a custom resource for creating a DNS record.
Parameters:
DomainName:
Type: String
Description: The domain name to create a DNS record for.
RecordValue:
Type: String
Description: The IP address or DNS value for the record.
Resources:
# Lambda Execution Role
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: DNSUpdatePolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource: "*"
- Effect: Allow
Action:
- route53:ChangeResourceRecordSets
- route53:ListHostedZones
Resource: "*"
# Lambda Function to Handle Custom Resource
CustomDNSHandlerFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Role: !GetAtt LambdaExecutionRole.Arn
Code:
ZipFile: |
import json
import boto3
import urllib3
import cfnresponse
def handler(event, context):
response_data = {}
try:
print("Received event: " + json.dumps(event, indent=2))
request_type = event['RequestType']
domain_name = event['ResourceProperties']['DomainName']
record_value = event['ResourceProperties']['RecordValue']
# Example of processing Create, Update, Delete
if request_type == 'Create':
# Code to create the DNS record
print(f"Creating DNS record for {domain_name} with value {record_value}")
elif request_type == 'Update':
# Code to update the DNS record
print(f"Updating DNS record for {domain_name} with value {record_value}")
elif request_type == 'Delete':
# Code to delete the DNS record
print(f"Deleting DNS record for {domain_name}")
# Sending success response
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
except Exception as e:
print(f"Error: {str(e)}")
cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
Runtime: python3.9
Timeout: 60
# Custom Resource
CustomDNSRecord:
Type: Custom::DNSRecord
Properties:
ServiceToken: !GetAtt CustomDNSHandlerFunction.Arn
DomainName: !Ref DomainName
RecordValue: !Ref RecordValue
Outputs:
DNSRecordStatus:
Description: Status of the DNS record creation.
Value: !GetAtt CustomDNSRecord.Status
7) Conditions, Mappings, and Dynamic References: Controlling resource creation with conditions and dynamically setting resource properties based on input parameters or other AWS data.
Dynamic referencing in AWS CloudFormation allows you to securely and dynamically retrieve values from AWS services, such as AWS Systems Manager Parameter Store, AWS Secrets Manager, or any other resources during stack creation or updates. This approach helps keep sensitive information secure and minimizes the need for hardcoding values in your CloudFormation templates.
Example:
AWSTemplateFormatVersion: '2010-09-09'
Description: Example of dynamic referencing SSM parameters in CloudFormation.
Parameters:
AppEnvironment:
Type: String
Default: dev
Description: The environment for the application (dev, staging, prod).
Resources:
MyEC2Instance:
Type: AWS::EC2::Instance
Properties:
InstanceType: t2.micro
ImageId: !Ref AmiId
KeyName: !Sub "{{resolve:ssm:/my-app/${AppEnvironment}/key-name}}"
Tags:
- Key: Name
Value: !Sub "EC2-${AppEnvironment}"
Outputs:
InstanceId:
Description: The ID of the created EC2 instance.
Value: !Ref MyEC2Instance
Best Practices for Complex CloudFormation Templates
Use Parameters for Reusability: Use parameters to make the template configurable for different environments (e.g., development, testing, production).
Implement Conditions: Use conditions to control resource creation based on environment type or input values, optimizing costs and performance.
Modularize with Nested Stacks: Break down complex templates into nested stacks to manage different parts of the infrastructure separately, improving maintainability and reusability.
Nested stacks in AWS CloudFormation are stacks created as part of other stacks. They enable you to manage complex cloud architectures by breaking down large, monolithic templates into smaller, reusable, and more manageable templates. Nested stacks make CloudFormation templates more modular, easier to maintain, and reusable across different environments or projects.
Example Template:
AWSTemplateFormatVersion: '2010-09-09'
Description: Main template that sets up nested stacks for a web application.
Parameters:
EnvironmentName:
Type: String
Description: Name of the environment (e.g., dev, prod).
Resources:
# Network Stack
NetworkStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://s3.amazonaws.com/my-bucket/network-stack.yaml
Parameters:
EnvironmentName: !Ref EnvironmentName
# Security Stack
SecurityStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://s3.amazonaws.com/my-bucket/security-stack.yaml
Parameters:
VPCId: !GetAtt NetworkStack.Outputs.VPCId
PublicSubnetIds: !GetAtt NetworkStack.Outputs.PublicSubnets
# Application Stack
ApplicationStack:
Type: AWS::CloudFormation::Stack
Properties:
TemplateURL: https://s3.amazonaws.com/my-bucket/application-stack.yaml
Parameters:
VPCId: !GetAtt NetworkStack.Outputs.VPCId
SecurityGroupId: !GetAtt SecurityStack.Outputs.AppSecurityGroupId
PublicSubnetIds: !GetAtt NetworkStack.Outputs.PublicSubnets
EnvironmentName: !Ref EnvironmentName
Outputs:
ApplicationURL:
Description: URL of the application.
Value: !GetAtt ApplicationStack.Outputs.AppURL
Use Mappings for Region-specific Data: Use mappings to handle region-specific configurations such as AMI IDs or instance types.
Implement Security Best Practices: Define least privilege security groups and IAM roles, and use NoEcho for sensitive parameters like passwords.
Outputs for Useful Information: Use outputs to provide useful information about the stack, such as endpoints, ARNs, and resource IDs.
This detailed walkthrough covers key aspects of creating a robust CloudFormation template for complex, highly available applications in AWS.
