When we talk about Infrastructure as Code or IaC, the first tool that comes to mind is Terraform. Terraform, created by HashiCorp, has become the standard for documentation and infrastructure management, but its declarative language, HCL (HashiCorp Configuration Language), has some limitations. The main limitation is not being a programming language but a configuration one.
Some alternatives have been emerging to fulfill these needs, such as:
AWS Cloud Development Kit, Amazon's solution that allows us to use TypeScript, Python, and Java to program the infrastructure using the cloud provider's solutions;
Pulumi, which allows us to use TypeScript, JavaScript, Python, Go, and C# to program infrastructures using solutions from AWS, Microsoft Azure, Google Cloud, and Kubernetes installations.
I will introduce Pulumi, using the Go language to create some infrastructure examples on AWS.
Installation
To make use of Pulumi, we first need to install its command-line application. Following the documentation, I installed it on my macOS using the command:
brew install pulumi
On the website, you can see how to install it on Windows and Linux.
Configure AWS Account Access
Since I will use AWS in this example, the next necessary step is to configure the credentials. For that, I got my access key and secret from the AWS dashboard and set the required environment variables:
export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>
Creating the project
With the initial dependencies configured, we can now create the project:
mkdir post-pulumi
cd post-pulumi
pulumi new aws-go
One of the creation steps requires setting up an account on the Pulumi website. For that, the command-line application opens the browser for this step to be completed. So I logged in with my Github account, completed the registration, returned to the terminal, and continued the project creation without any problems.
You can see the result of running the command can at this link. In addition, at the end of the process, it installs all the necessary dependencies for creating the project in Go.
Files created
Looking at the directory contents, we can see that some configuration files and a main.go
were created.
Pulumi.yaml
name: post-pulumi
runtime: go
description: A minimal AWS Go Pulumi program
Pulumi.dev.yaml
config:
aws:region: us-east-1
main.go
package main
import (
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Create an AWS resource (S3 Bucket)
bucket, err := s3.NewBucket(ctx, "my-bucket", nil)
if err != nil {
return err
}
// Export the name of the bucket
ctx.Export("bucketName", bucket.ID())
return nil
})
}
When running
pulumi up
The bucket was created in S3, as the code indicates.
And the command:
pulumi destroy
Destroy all the resources, in this case, the S3 bucket.
First example - creating a static page in S3
Now let's do some more complex examples.
The first step is to create a static page, which we are going to deploy:
mkdir static
Inside this directory, I created the file:
static/index.html
<html>
<body>
<h1>Hello, Pulumi!</h1>
</body>
</html>
I changed main.go to reflect the new structure:
package main
import (
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Create an AWS resource (S3 Bucket)
bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
Website: s3.BucketWebsiteArgs{
IndexDocument: pulumi.String("index.html"),
},
})
if err != nil {
return err
}
// Export the name of the bucket
ctx.Export("bucketName", bucket.ID())
_, err = s3.NewBucketObject(ctx, "index.html", &s3.BucketObjectArgs{
Acl: pulumi.String("public-read"),
ContentType: pulumi.String("text/html"),
Bucket: bucket.ID(),
Source: pulumi.NewFileAsset("static/index.html"),
})
if err != nil {
return err
}
ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint))
return nil
})
}
To update run:
pulumi up
And confirm the change.
The code snippet:
ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint))
Generate as output the address to access index.html:
Outputs:
+ bucketEndpoint: "http://my-bucket-357877e.s3-website-us-east-1.amazonaws.com"
The case above is a straightforward example, but it already demonstrates the power of the tool. So let's make things a little more complex and fun now.
Second example - a site inside a container
Let's create a Dockerfile with a web server to host our static content:
static/Dockerfile
FROM golang
ADD . /go/src/foo
WORKDIR /go/src/foo
RUN go build -o /go/bin/main
ENTRYPOINT /go/bin/main
EXPOSE 80
Let's now create the static/main.go file, which will be our web server:
package main
import (
"log"
"net/http"
)
func main() {
r := http.NewServeMux()
fileServer := http.FileServer(http.Dir("./"))
r.Handle("/", http.StripPrefix("/", fileServer))
s := &http.Server{
Addr: ":80",
Handler: r,
}
log.Fatal(s.ListenAndServe())
}
Let's change main.go to include the infrastructure of an ECS cluster and everything else needed to run our container:
package main
import (
"encoding/base64"
"fmt"
"strings"
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2"
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr"
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs"
elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2"
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam"
"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Read back the default VPC and public subnets, which we will use.
t := true
vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t})
if err != nil {
return err
}
subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id})
if err != nil {
return err
}
// Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{
VpcId: pulumi.String(vpc.Id),
Egress: ec2.SecurityGroupEgressArray{
ec2.SecurityGroupEgressArgs{
Protocol: pulumi.String("-1"),
FromPort: pulumi.Int(0),
ToPort: pulumi.Int(0),
CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
},
},
Ingress: ec2.SecurityGroupIngressArray{
ec2.SecurityGroupIngressArgs{
Protocol: pulumi.String("tcp"),
FromPort: pulumi.Int(80),
ToPort: pulumi.Int(80),
CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
},
},
})
if err != nil {
return err
}
// Create an ECS cluster to run a container-based service.
cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
if err != nil {
return err
}
// Create an IAM role that can be used by our service's task.
taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{
AssumeRolePolicy: pulumi.String(`{
"Version": "2008-10-17",
"Statement": [{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}]
}`),
})
if err != nil {
return err
}
_, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{
Role: taskExecRole.Name,
PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
})
if err != nil {
return err
}
// Create a load balancer to listen for HTTP traffic on port 80.
webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{
Subnets: toPulumiStringArray(subnet.Ids),
SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
})
if err != nil {
return err
}
webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{
Port: pulumi.Int(80),
Protocol: pulumi.String("HTTP"),
TargetType: pulumi.String("ip"),
VpcId: pulumi.String(vpc.Id),
})
if err != nil {
return err
}
webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{
LoadBalancerArn: webLb.Arn,
Port: pulumi.Int(80),
DefaultActions: elb.ListenerDefaultActionArray{
elb.ListenerDefaultActionArgs{
Type: pulumi.String("forward"),
TargetGroupArn: webTg.Arn,
},
},
})
if err != nil {
return err
}
//create a new ECR repository
repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{})
if err != nil {
return err
}
repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) {
creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{
RegistryId: rid,
})
if err != nil {
return nil, err
}
data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
if err != nil {
fmt.Println("error:", err)
return nil, err
}
return strings.Split(string(data), ":"), nil
}).(pulumi.StringArrayOutput)
repoUser := repoCreds.Index(pulumi.Int(0))
repoPass := repoCreds.Index(pulumi.Int(1))
//build the image
image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
Build: docker.DockerBuildArgs{
Context: pulumi.String("./static"),
},
ImageName: repo.RepositoryUrl,
Registry: docker.ImageRegistryArgs{
Server: repo.RepositoryUrl,
Username: repoUser,
Password: repoPass,
},
})
if err != nil {
return err
}
containerDef := image.ImageName.ApplyT(func(name string) (string, error) {
fmtstr := `[{
"name": "my-app",
"image": %q,
"portMappings": [{
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp"
}]
}]`
return fmt.Sprintf(fmtstr, name), nil
}).(pulumi.StringOutput)
// Spin up a load balanced service running NGINX.
appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{
Family: pulumi.String("fargate-task-definition"),
Cpu: pulumi.String("256"),
Memory: pulumi.String("512"),
NetworkMode: pulumi.String("awsvpc"),
RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")},
ExecutionRoleArn: taskExecRole.Arn,
ContainerDefinitions: containerDef,
})
if err != nil {
return err
}
_, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{
Cluster: cluster.Arn,
DesiredCount: pulumi.Int(5),
LaunchType: pulumi.String("FARGATE"),
TaskDefinition: appTask.Arn,
NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{
AssignPublicIp: pulumi.Bool(true),
Subnets: toPulumiStringArray(subnet.Ids),
SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
},
LoadBalancers: ecs.ServiceLoadBalancerArray{
ecs.ServiceLoadBalancerArgs{
TargetGroupArn: webTg.Arn,
ContainerName: pulumi.String("my-app"),
ContainerPort: pulumi.Int(80),
},
},
}, pulumi.DependsOn([]pulumi.Resource{webListener}))
if err != nil {
return err
}
// Export the resulting web address.
ctx.Export("url", webLb.DnsName)
return nil
})
}
func toPulumiStringArray(a []string) pulumi.StringArrayInput {
var res []pulumi.StringInput
for _, s := range a {
res = append(res, pulumi.String(s))
}
return pulumi.StringArray(res)
}
Complex? Yes, but this complexity is inherent to AWS features and not Pulumi. We would have similar complexity if we were using Terraform or CDK.
Before running our code, we need to download the new dependencies:
go get github.com/pulumi/pulumi-docker
go get github.com/pulumi/pulumi-docker/sdk/v3/go/docker
Now just run the command:
pulumi up
The execution output will generate the URL of the load balancer, which we will use to access the contents of our container in execution.
Reorganizing the code
Now we can start making use of the advantages of a complete programming language like Go. For example, we could use language features like functions, concurrency, conditionals, etc. In this example, we are going to organize our code better. For this, I created the iac directory and the iac/fargate.go file. After that, I moved most of the logic from main.go
to the new file:
package iac
import (
"encoding/base64"
"fmt"
"strings"
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2"
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr"
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs"
elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2"
"github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam"
"github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func FargateRun(ctx *pulumi.Context) error {
// Read back the default VPC and public subnets, which we will use.
t := true
vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t})
if err != nil {
return err
}
subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id})
if err != nil {
return err
}
// Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{
VpcId: pulumi.String(vpc.Id),
Egress: ec2.SecurityGroupEgressArray{
ec2.SecurityGroupEgressArgs{
Protocol: pulumi.String("-1"),
FromPort: pulumi.Int(0),
ToPort: pulumi.Int(0),
CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
},
},
Ingress: ec2.SecurityGroupIngressArray{
ec2.SecurityGroupIngressArgs{
Protocol: pulumi.String("tcp"),
FromPort: pulumi.Int(80),
ToPort: pulumi.Int(80),
CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
},
},
})
if err != nil {
return err
}
// Create an ECS cluster to run a container-based service.
cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
if err != nil {
return err
}
// Create an IAM role that can be used by our service's task.
taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{
AssumeRolePolicy: pulumi.String(`{
"Version": "2008-10-17",
"Statement": [{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}]
}`),
})
if err != nil {
return err
}
_, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{
Role: taskExecRole.Name,
PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
})
if err != nil {
return err
}
// Create a load balancer to listen for HTTP traffic on port 80.
webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{
Subnets: toPulumiStringArray(subnet.Ids),
SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
})
if err != nil {
return err
}
webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{
Port: pulumi.Int(80),
Protocol: pulumi.String("HTTP"),
TargetType: pulumi.String("ip"),
VpcId: pulumi.String(vpc.Id),
})
if err != nil {
return err
}
webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{
LoadBalancerArn: webLb.Arn,
Port: pulumi.Int(80),
DefaultActions: elb.ListenerDefaultActionArray{
elb.ListenerDefaultActionArgs{
Type: pulumi.String("forward"),
TargetGroupArn: webTg.Arn,
},
},
})
if err != nil {
return err
}
repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{})
if err != nil {
return err
}
repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) {
creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{
RegistryId: rid,
})
if err != nil {
return nil, err
}
data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
if err != nil {
fmt.Println("error:", err)
return nil, err
}
return strings.Split(string(data), ":"), nil
}).(pulumi.StringArrayOutput)
repoUser := repoCreds.Index(pulumi.Int(0))
repoPass := repoCreds.Index(pulumi.Int(1))
image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
Build: docker.DockerBuildArgs{
Context: pulumi.String("./static"),
},
ImageName: repo.RepositoryUrl,
Registry: docker.ImageRegistryArgs{
Server: repo.RepositoryUrl,
Username: repoUser,
Password: repoPass,
},
})
if err != nil {
return err
}
containerDef := image.ImageName.ApplyT(func(name string) (string, error) {
fmtstr := `[{
"name": "my-app",
"image": %q,
"portMappings": [{
"containerPort": 80,
"hostPort": 80,
"protocol": "tcp"
}]
}]`
return fmt.Sprintf(fmtstr, name), nil
}).(pulumi.StringOutput)
// Spin up a load balanced service running NGINX.
appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{
Family: pulumi.String("fargate-task-definition"),
Cpu: pulumi.String("256"),
Memory: pulumi.String("512"),
NetworkMode: pulumi.String("awsvpc"),
RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")},
ExecutionRoleArn: taskExecRole.Arn,
ContainerDefinitions: containerDef,
})
if err != nil {
return err
}
_, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{
Cluster: cluster.Arn,
DesiredCount: pulumi.Int(5),
LaunchType: pulumi.String("FARGATE"),
TaskDefinition: appTask.Arn,
NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{
AssignPublicIp: pulumi.Bool(true),
Subnets: toPulumiStringArray(subnet.Ids),
SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
},
LoadBalancers: ecs.ServiceLoadBalancerArray{
ecs.ServiceLoadBalancerArgs{
TargetGroupArn: webTg.Arn,
ContainerName: pulumi.String("my-app"),
ContainerPort: pulumi.Int(80),
},
},
}, pulumi.DependsOn([]pulumi.Resource{webListener}))
if err != nil {
return err
}
// Export the resulting web address.
ctx.Export("url", webLb.DnsName)
return nil
}
func toPulumiStringArray(a []string) pulumi.StringArrayInput {
var res []pulumi.StringInput
for _, s := range a {
res = append(res, pulumi.String(s))
}
return pulumi.StringArray(res)
}
The next step was to configure the iac
directory to be a Go language module:
cd iac
go mod init github.com/eminetto/post-pulumi/iac
cd ..
go mod edit -replace github.com/eminetto/post-pulumi/iac=./iac
go mod tidy
Our main.go
can now be simplified:
package main
import (
"github.com/eminetto/post-pulumi/iac"
"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
return iac.FargateRun(ctx)
})
}
That way, we can better manage the structure of the code that will handle AWS resources. We can reuse this code in other projects, use environment variables, write tests, or whatever else our imagination allows.
Conclusion
Using a tool like Pulumi significantly increases the range of options that we can use in building a project's infrastructure while maintaining readability, code reuse and organization.