How to Create Reproducible AWS Infrastructure with Terraform Modules

Kyle Galbraith - Jul 13 '20 - - Dev Community

At this point, I spend a large part of my week inside of the Amazon Web Services ecosystem. If I had to make a guess I would say 85% of the day is creating, updating, or destroying AWS infrastructure.

But, I spend less than 1% of my week inside of the AWS Console.

That is to say that I don't touch any AWS infrastructure via the console. All the infrastructure I interact with is in code, Terraform HCL to be specific.

When infrastructure is in a declarative language like HCL, provisioning, updating, and deleting is easier, less error-prone, and reproducible. It is also transparent. Every change to the infrastructure goes through a pull request process that is reviewed by peers.

For all the positives with infrastructure as code, particularly HCL, there are downsides and negatives.

  1. HCL is yet another language to learn. As we will see in a minute, it's not like the language you are likely creating your application in.
  2. It's limited in the set of features it offers.
  3. Modularity can be difficult to get started with if you're not familiar with it.

Point (3) is the one we are going to focus on here. With Terraform modules we can create infrastructure packages that are reusable.

Working from an example

Let's take a basic example, creating an IAM user with the necessary policy to access Amazon Elastic Container Registry (ECR). Here are the three resources we need in Terraform.

# IAM User + Policy + Key to access ECR
resource "aws_iam_user" "some-user" {
  name = "some-user"
}

resource "aws_iam_user_policy" "some-user" {
  name = "some-user"
  user = aws_iam_user.some-user.name
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = ["ecr:GetAuthorizationToken"],
        Resource = "*"
      },
      {
        Effect = "Allow",
        Action = [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:DescribeImages",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage"
        ],
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_access_key" "some-user" {
  user    = aws_iam_user.some-user.name
}
Enter fullscreen mode Exit fullscreen mode

To ground ourselves in reality, let's put these into a real Terraform project. Run the following commands in the Terminal. (note: you will need to install Terraform and have the aws cli installed).

$ mkdir terraform-example 
$ cd terraform-example
$ touch main.tf
Enter fullscreen mode Exit fullscreen mode

Now let's add the following block to main.tf.

terraform {
  required_version = ">=0.12, <0.13"
}

provider "aws" {
  version = "~> 2.58"
  region  = "us-west-2"
}
Enter fullscreen mode Exit fullscreen mode

This requires that we have a terraform version greater than or equal to 0.12, but less than 0.13. The second bit is where we are declaring that we are going to use the AWS provider. This is the provider that will allow us to access AWS resources inside of our account.

Below the provider, let's add in the IAM resources from earlier. We should now have something like this in main.tf.

terraform {
  required_version = ">=0.12, <0.13"
}

provider "aws" {
  version = "~> 2.58"
  region  = "us-west-2"
}

# IAM User + Policy + Key to access ECR
resource "aws_iam_user" "some-user" {
  name = "some-user"
}

resource "aws_iam_user_policy" "some-user" {
  name = "some-user"
  user = aws_iam_user.some-user.name
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = ["ecr:GetAuthorizationToken"],
        Resource = "*"
      },
      {
        Effect = "Allow",
        Action = [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:DescribeImages",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage"
        ],
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_access_key" "some-user" {
  user = aws_iam_user.some-user.name
}
Enter fullscreen mode Exit fullscreen mode

Let's run this template and confirm that a new IAM user gets created. From the terraform-example folder run the following commands.

$ terraform init
$ terraform apply
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_iam_access_key.some-user will be created
  + resource "aws_iam_access_key" "some-user" {
      + encrypted_secret     = (known after apply)
      + id                   = (known after apply)
      + key_fingerprint      = (known after apply)
      + secret               = (sensitive value)
      + ses_smtp_password    = (sensitive value)
      + ses_smtp_password_v4 = (sensitive value)
      + status               = (known after apply)
      + user                 = "some-user"
    }

  # aws_iam_user.some-user will be created
  + resource "aws_iam_user" "some-user" {
      + arn           = (known after apply)
      + force_destroy = false
      + id            = (known after apply)
      + name          = "some-user"
      + path          = "/"
      + unique_id     = (known after apply)
    }

  # aws_iam_user_policy.some-user will be created
  + resource "aws_iam_user_policy" "some-user" {
      + id     = (known after apply)
      + name   = "some-user"
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "ecr:GetAuthorizationToken",
                        ]
                      + Effect   = "Allow"
                      + Resource = "*"
                    },
                  + {
                      + Action   = [
                          + "ecr:GetAuthorizationToken",
                          + "ecr:BatchCheckLayerAvailability",
                          + "ecr:GetDownloadUrlForLayer",
                          + "ecr:GetRepositoryPolicy",
                          + "ecr:DescribeRepositories",
                          + "ecr:ListImages",
                          + "ecr:DescribeImages",
                          + "ecr:BatchGetImage",
                          + "ecr:InitiateLayerUpload",
                          + "ecr:UploadLayerPart",
                          + "ecr:CompleteLayerUpload",
                          + "ecr:PutImage",
                        ]
                      + Effect   = "Allow"
                      + Resource = "*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + user   = "some-user"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:
Enter fullscreen mode Exit fullscreen mode

The apply step has stopped us to confirm these are all the resources we want to create. Go ahead and type yes in the prompt. We should see the following after the apply.

aws_iam_user.some-user: Creating...
aws_iam_user.some-user: Creation complete after 1s [id=some-user]
aws_iam_user_policy.some-user: Creating...
aws_iam_access_key.some-user: Creating...
aws_iam_access_key.some-user: Creation complete after 0s [id=<some-id>]
aws_iam_user_policy.some-user: Creation complete after 1s [id=some-user:some-user]
Enter fullscreen mode Exit fullscreen mode

Awesome, we now have an IAM user, some-user, that has a policy that allows them to access ECR. If we only ever had to create one user, were set, nothing more to do here.

But, we often need more than one user. It also tends to work out that we need to give each user the same permissions.

Using modules

We could have more than one IAM user with the same policy by copying and pasting the resources. So if we had a second user, our main.tf could look like this.

terraform {
  required_version = ">=0.12, <0.13"
}

provider "aws" {
  version = "~> 2.58"
  region  = "us-west-2"
}

# IAM User + Policy + Key to access ECR
resource "aws_iam_user" "some-user" {
  name = "some-user"
}

resource "aws_iam_user_policy" "some-user" {
  name = "some-user"
  user = aws_iam_user.some-user.name
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = ["ecr:GetAuthorizationToken"],
        Resource = "*"
      },
      {
        Effect = "Allow",
        Action = [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:DescribeImages",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage"
        ],
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_access_key" "some-user" {
  user = aws_iam_user.some-user.name
}

# Second user
resource "aws_iam_user" "another-user" {
  name = "another-user"
}

resource "aws_iam_user_policy" "another-user" {
  name = "another-user"
  user = aws_iam_user.another-user.name
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = ["ecr:GetAuthorizationToken"],
        Resource = "*"
      },
      {
        Effect = "Allow",
        Action = [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:DescribeImages",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage"
        ],
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_access_key" "another-user" {
  user = aws_iam_user.another-user.name
}
Enter fullscreen mode Exit fullscreen mode

We can look at the another-user resources and see that we just copied and pasted the resources. This works and is OK in a pinch. But imagine if you had to do this for 10 users 😭

This is where Terraform modules come in to save the day. A module is a collection of resources that belong together. They can be called from other modules which adds the resources from the module.

In fact, we are already using a module in our example. We are using the root module. The root module is always the entry main.tf defined in our project.

We are going to convert the three resources, IAM user, IAM policy, and IAM access key into a module that can be reused.

To get started, create a new folder at the root of terraform-example for our module to live in.

$ mkdir modules
$ cd modules
$ touch iam.tf
Enter fullscreen mode Exit fullscreen mode

Now inside of iam.tf let's add our three resources from earlier.

# IAM User + Policy + Key to access ECR
resource "aws_iam_user" "some-user" {
  name = "some-user"
}

resource "aws_iam_user_policy" "some-user" {
  name = "some-user"
  user = aws_iam_user.some-user.name
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = ["ecr:GetAuthorizationToken"],
        Resource = "*"
      },
      {
        Effect = "Allow",
        Action = [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:DescribeImages",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage"
        ],
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_access_key" "some-user" {
  user = aws_iam_user.some-user.name
}
Enter fullscreen mode Exit fullscreen mode

Right now this IAM module is not parameterized. Let's go ahead and fix that by adding a username variable at the top iam.tf.

# IAM User + Policy + Key to access ECR
variable "username" {
  type = string
}

resource "aws_iam_user" "some-user" {
  name = var.username
}

resource "aws_iam_user_policy" "some-user" {
  name = "some-user"
  user = aws_iam_user.some-user.name
  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = ["ecr:GetAuthorizationToken"],
        Resource = "*"
      },
      {
        Effect = "Allow",
        Action = [
          "ecr:GetAuthorizationToken",
          "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetRepositoryPolicy",
          "ecr:DescribeRepositories",
          "ecr:ListImages",
          "ecr:DescribeImages",
          "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload",
          "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload",
          "ecr:PutImage"
        ],
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_access_key" "some-user" {
  user = aws_iam_user.some-user.name
}
Enter fullscreen mode Exit fullscreen mode

Now our entire IAM module takes in a username variable. It uses that to create the IAM user, IAM policy, and access key for that username.

Reusing modules

Now that we have an IAM module that takes in a username we can go back and edit our main.tf. We no longer have to copy and paste resources to add a new user. Instead, we can call the module with the username we want to create.

Here is an example where we are creating three users.

terraform {
  required_version = ">=0.12, <0.13"
}

provider "aws" {
  version = "~> 2.58"
  region  = "us-west-2"
}

module "user-1" {
  source   = "./modules"
  username = "some-user-2"
}

module "user-2" {
  source   = "./modules"
  username = "another-user"
}

module "user-3" {
  source   = "./modules"
  username = "yet-another-user"
}
Enter fullscreen mode Exit fullscreen mode

Instead of copying and pasting resources around, we use the module we created. This module has all the resources that need to be created. From the outside, we give the module the username we want the IAM user to have.

Let's do a plan on this and see what happens. We have to run terraform init again because added a new module.

$ terraform init
$ terraform plan
Enter fullscreen mode Exit fullscreen mode

We see that nine resources are going to be created which makes sense because we are creating three new users. If we look at the plan we see that resources are now coming from the module.

# module.user-3.aws_iam_access_key.some-user will be created
  + resource "aws_iam_access_key" "some-user" {
      + encrypted_secret     = (known after apply)
      + id                   = (known after apply)
      + key_fingerprint      = (known after apply)
      + secret               = (sensitive value)
      + ses_smtp_password    = (sensitive value)
      + ses_smtp_password_v4 = (sensitive value)
      + status               = (known after apply)
      + user                 = "yet-another-user"
    }

  # module.user-3.aws_iam_user.some-user will be created
  + resource "aws_iam_user" "some-user" {
      + arn           = (known after apply)
      + force_destroy = false
      + id            = (known after apply)
      + name          = "yet-another-user"
      + path          = "/"
      + unique_id     = (known after apply)
    }

  # module.user-3.aws_iam_user_policy.some-user will be created
  + resource "aws_iam_user_policy" "some-user" {
      + id     = (known after apply)
      + name   = "some-user"
      + policy = jsonencode(
            {
              + Statement = [
                  + {
                      + Action   = [
                          + "ecr:GetAuthorizationToken",
                        ]
                      + Effect   = "Allow"
                      + Resource = "*"
                    },
                  + {
                      + Action   = [
                          + "ecr:GetAuthorizationToken",
                          + "ecr:BatchCheckLayerAvailability",
                          + "ecr:GetDownloadUrlForLayer",
                          + "ecr:GetRepositoryPolicy",
                          + "ecr:DescribeRepositories",
                          + "ecr:ListImages",
                          + "ecr:DescribeImages",
                          + "ecr:BatchGetImage",
                          + "ecr:InitiateLayerUpload",
                          + "ecr:UploadLayerPart",
                          + "ecr:CompleteLayerUpload",
                          + "ecr:PutImage",
                        ]
                      + Effect   = "Allow"
                      + Resource = "*"
                    },
                ]
              + Version   = "2012-10-17"
            }
        )
      + user   = "yet-another-user"
    }
Enter fullscreen mode Exit fullscreen mode

Now if we go and apply this we should see that three new users get created. (note: I use the -auto-approve flag to tell Terraform to auto-approve the apply without prompting me).

$ terraform apply -auto-approve
module.user-1.aws_iam_user_policy.some-user: Creation complete after 0s [id=some-user-2:some-user]
module.user-1.aws_iam_access_key.some-user: Creation complete after 0s [id=<some-id>]

Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
Enter fullscreen mode Exit fullscreen mode

Now we see that nine resources got added. That is because each module for our IAM user has three resources, the user, the IAM policy, and the IAM access key.

With that, we now have an IAM user Terraform module that we can reuse to create new IAM users. But it's actually even better than that, we can also leverage this to update users.

Because each user is created via the module, any change we make to the module also gets applied to every user. This is handy, for example, if we wanted to have a common IAM policy that each user had associated with them.

Conclusion

Infrastructure as code is a powerful tool within a cloud environment. It allows your infrastructure to live next to your everyday code. It encourages knowledge sharing and pull request processes on infrastructure changes. With the right tools, it allows you to share common infrastructure ideas across teams.

What we saw here was one example of a tool, Terraform. We took a look at how we can use Terraform modules to create packages of infrastructure that are reusable.

In this example, we focused on something straight forward, an IAM user. But modules can be created for any number of things. Perhaps you want to create a module that represents every piece of infrastructure for your application. Or maybe you want each team to use a common VPC module that has security at the forefront. Whatever the case, modules allow you to create reusable blocks of resources.

Want to check out my other projects?

I am a huge fan of the DEV community. If you have any questions or want to chat about different ideas relating to refactoring, reach out on Twitter or drop a comment below.

Outside of blogging, I created a Learn AWS By Using It course. In the course, we focus on learning Amazon Web Services by actually using it to host, secure, and deliver static websites. It's a simple problem, with many solutions, but it's perfect for ramping up your understanding of AWS. I recently added two new bonus chapters to the course that focus on Infrastructure as Code and Continuous Deployment.

Image
7 Secrets About Key Repair Near Me That Nobody Can Tell You
Image
10 Unexpected Bioethanol Freestanding Stove Tips
Image
It's The Adult ADHD Diagnosis Case Study You'll Never Forget
Image
Database Setup, JWT Implementation & API

postgres, jwt, authentication
Image
Buy A Harmonica: A Guide to Choosing the Best Instrument
Image
Buy A Harmonica: A Guide to Choosing the Best Instrument
Image
Why You Should Forget About Making Improvements To Your Cost Of ADHD Assessment
Image
What's The Most Common How To Get A ADHD Assessment UK Debate Doesn't Have To Be As Black And White As You May Think
Image
https://www.facebook.com/share/p/raeu1ZhVEFbU4yDj/
Image
7 Small Changes That Will Make A Big Difference With Your Chesterfields Sofas
Image
Zanurz się w Komfort: Meble do Sypialni, Które Odkryją Nowy Wymiar Relaksu
Image
Cat Flap Fitters Near Me
Image
Cat Flap Installers
Image
15 Things You've Never Known About 4 Seater Chesterfield Sofa
Image
15 Reasons Why You Shouldn't Ignore Bi Fold Repairs
Image
Diatonic vs Chromatic Harmonica: A Best Comparison
Image
Buzzwords De-Buzzed: 10 Alternative Ways To Deliver Double Chaise Couch
Image
10 Basics About Bifold Door Repair You Didn't Learn At School
Image
10 Facts About Bifold Door Repair That Will Instantly Set You In A Positive Mood
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player