Why should you automate your architecture on AWS?

The answer to this question in not obvious. I remember explaining how to automate an application architecture on AWS to a colleague a few years ago, and his reaction was : I don't understand anything!

Maybe I didn't explain this clearly, but when someone tells me he doesn't understand something, it often means he doesn't see any benefit for him.
That’s why I wanted to list the benefits I discovered while automating my architectures on AWS for many years now.

N.B. : This article is AWS (Amazon Web Services) oriented with AWS examples because this is the cloud computing platform I work with everyday, nevertheless the benefits listed above, would be the same on any other cloud provider enabling automation.

First let’s define what is an architecture for me.

What is an architecture on AWS?

Defining architecture is hard, but let's say that an architecture is a virtual structure that will host our algorithms and data. Here is a very simple but typical layered architecture : a client, a server and a database (both inside a network).

N.B. : This is a high level of abstraction, and there are many ways to implement this pattern on AWS.

So, let's imagine that you want to build that structure for your application using basic AWS services, then you would:

• create a network (a VPC,...)
• create a server (an EC2 instance) and configure it with the required libraries
• create a database (e.g. another EC2 instance again) and configure it with a database engine (e.g. MySQL)

To build the previously described structure can be very easy nowadays by clicking in the web pages of the Amazon Web Services console.

However, it exists another option: writing a file where you will describe your architecture and use an "architecture engine" that will automatically create all the components. Well known examples of these tools are AWS CloudFormation or Hashicorp Terraform.

But, I must be honest, this will take more time to write, execute and test that way. Maybe 2 or 3 times more, compared to clicking into the AWS console. Even if you have the skills.

So, again, why should I automate my architecture on AWS?

Automate to repeat your architecture

Unfortunately, we do not need our architecture at only one place. We often need to have our application deployed in different environments. A development environment, a production environment and sometimes a staging environment. All these environments have different purposes.

A production environment has an obvious purpose. This is where you application will run to serve your real users (those who want the application to always be available).

The development environment is where you want to experiment things, without disturbing the production environment.

Staging environment is, for example, where you want to load test your application in an architecture which is identical to production.

By automating your architecture, you can rebuild it anywhere from scratch. And the time you thought you "lost" when building the first template becomes largely profitable because the "architecture engine" will "click" much faster than you do!

Automate to update your architecture with confidence

In my example at the beginning of this post, I do not show all the details of my architecture for the sake of simplicity. But in the real world you may have to deal with many architecture details.

You can have network details (e.g. subnets, route tables, …), security details (e.g. firewall rules, role permissions, …), storage details (e.g. volume size, volume type, …), availability details (e.g. load balancer, scaling policy, …) etc.

So, if you had to manually update these in multiple environments, it's easy to make mistakes, to forget something, to configure things slightly differently, etc. That can have consequences on your application stability, reducing your confidence in changing things.

By automating your architecture, you avoid human errors and you feel more confident. Machines do not only click faster than you do, they are also more reliable!

Automate to reuse & share architectural models

The good news is that you almost never have to build a complete architecture from scratch. Template models exist on the web and you can reuse and adapt them to your needs.

Moreover, you can create and share your own architecture templates within the company. Or simply restart from one you previously deployed when you have a similar architecture to work on (you can imagine building your own "architecture catalog").

With specific tools like AWS CDK (Cloud Development Kit), some models are completely embedded with best practices (like the AWS Well Architected Framework practices). You can create a complete well architected Kubernetes cluster with just a few lines of code!

Do not Repeat Yourself (DRY), is now a development technique that you can use in application architecture!

Automate to evolve your architecture

I do not know your architecture today, but I can tell you one thing: within 3 years it will not be the same!

So, imagine you want to deploy a new version of your architecture (e.g. you want to use containers or add serverless technologies). You could simply update it directly, but this can incur some downtime of your application. Even worse, the new version may not work right away and the application may be unavailable for a long time.

With automation, you can build a partial or completely new architecture without disturbing the one in place.

Then you test the new version. If it's working ok, you shift (progressively) the traffic to the new version. If not, you shift the traffic back to the previous version. When you are happy, you can then destroy the old architecture.

Today's architecture evolves more often, automating will help evolving your architecture incrementally while limiting risks. Do you think successful startups have the right architecture on day one and keep it forever?

Automate to postpone your architectural decisions

Architects are suppose to take "hard/costly to change decisions" like choosing between a relational (e.g. MySQL) or a non relation database (e.g. MongoDB) because if you discover that you made the wrong choice when you start running your application in production this will have a big impact!

We also often think that the architect should have the answer at the beginning of the project. Which is certainly the worst moment because this is when he has the least information to make a decision.

I remember working on an architecture where I made very complicated decisions at the virtual network layer. The reason was that my understanding of the problem to solve was not mature enough.

If I had not automated this part I guess I would have kept this complex virtual network architecture (which would not have been good for the project).

When I had a better understanding, I simplified this virtual network layer in my templates and rebuilt the complete architecture on top of it. Within a few hours of work, it was done thanks to automation.

Automate to document your architecture

Code is the only source of truth. If you document your architecture in Microsoft Word or Google Docs documents, you will rapidly have a drift (and I personally never use those for describing an architecture, but only a diagram with the most important information).

Modern architectures constantly evolve. At the detail level, updates can be frequent (New security rules, new permissions, ...), but you may also want to add or replace components (e.g. using AWS EFS, Elastic File System instead of an EBS Volume).

This is really hard to maintain in a word processing document. The only source of truth should be your template!

Automate to lower the cost of your architecture

I may not need to always leave my development architecture up 24/7. With automation, you can rapidly build an architecture, but you can also destroy its components automatically and easily.

By destroying unused components (a virtual instance, for example), you stop paying for it.

So, if you destroy your architecture in your development environment on a Friday night and you rebuild it on Monday morning you will save about 25% on your bill. And you can go further if you destroy/rebuild every evening/morning during the week.

Even more interesting, you can build a "cheap" architecture to make experiments. For example, if your application needs an expensive EC2 instance (tens of dollars per hour) to run in production, you can still test your architecture with a small instances (lower than 1 dollar per hour) to explore and validate new ideas without paying too much.

When your new shiny architecture is ready for staging or production you just have to change the instance size parameter of your template to make it work.

Automate to keep your architecture clean

Is this architectural component still used?

This is the kind of question you can ask yourself when you discover an existing architecture built by someone else. But this can also be the case for your own architecture after just a few days of clicks ;)

When you automate the creation and deletion of your architecture's components, you are not bothered with useless components, so you ask yourself less questions and you go faster.

Conclusion

I could carry on listing other benefits, but this blog post would have been twice as big! So I just kept what I thought was the most important things from my point of view.

In my opinion, there are only benefits with architecture automation.

You may argue it is quicker to click into the console. And you are right if you think of the initial investment you have to do for automating things or if you think your architecture will never evolve.

But, if you carry on doing things manually you take the risk to lose all the automation benefits and be slower and slower over time.

Useful Links:

AWS CloudFormation
AWS CloudFormation Workshop

AWS CDK (Cloud Development Kit)
AWS CDK Workshop