Let's Begin

Hi! You can read this tutorial series in its original format over at my blog, J-bytes. This is not a short series; it clocks in at over 11,000 words and is a detailed tutorial on how to get a good understanding of several technologies. Many of the articles are long, and require you to actually have an AWS console screen open in front of you. This might be something best to work through when you have a bit of time. Additionally, you can contact me at rob@<blog-domain-name>.com if you have any problems / something cool you wrote / things to chat about.

We are going to make a Twitter and email campaign 'serverless' backend for a site using entirely AWS cloud services (mainly Lambda and DynamoDB) to save a ton of money over classic infrastructure implementations. Again, this is a massive tutorial, so set aside a few hours to complete it. We'll be doing everything from encryption and key management to creating a static site behind AWS Cloudfront.

Super-cool diagram:

If you don't care about the how and why and just want to get to the project, just head on over to the next post.

If you find yourself in need of a way to register lots of twitter users / emails for a single purpose such as a sign-up campaign, then this is the tutorial for you. You can also make some easy modifications, such as RDS over DynamoDB, if you intend to send emails/tweets more than once.

We will be making two stages (staging and production) in this project to demonstrate stage handling at each level of the project.

This tutorial assumes that the site is static pages hosted in an S3 account, but you can simply adjust if that isn't the case. We will be making a placeholder page to demonstrate proof of concept.

We will allow users to register their email and then send them a confirmation message. Alternatively, we will allow them to register by twitter and decide if they want to follow us. When I asked twitter about their automated follow policy they assured me that we could have users follow us in this way, but only if they explicitly consented (i.e. use a checkbox).

We will encrypt and store their emails and twitter info so that later we can send them emails/send them direct messages/do cool things.

I assume you have an AWS account with sufficient permissions and are not worried about minor charges throughout the project. I assume that you have a domain name of some kind or can easily get one. If you want to send emails you may need it, otherwise don't worry about it. Most of this is free tier, but I cannot guarantee you won't be charged anything. I also assume that you can use Google, and are patient.

Other assumed knowledge:

• You know how to install packages on your specific machine (this tutorial assumes Mac or Linux, Windows may work but I don't know)
• Very basic Javascript is nice but not necessary (node.js). You can also use Python or Java for the Lambda code, but you're on your own if you do.
• Some Ruby is also nice but not necessary (you know what gems are).
• Git, because always use version control. Always.
• You are somewhat familiar with the command line (mkdir, cd, touch, etc.)
• Basic Mysql (optional, but needed for the cool parts at the end where we send tweets)
• You know what JSON is (even if you can't write it by scratch, you understand the concept)
• You have at least heard of the following (if not, read a bit on the ones you don't know): AWS Lambda, API Gateway, Data Pipeline, S3, CloudWatch, DynamoDB, CloudFront, CLI, KMS, WAF, IAM, the mail service Mailgun, the tool Postman

This implementation is for you if:

• You want to remove most hardware bottlenecks from your website and let DB read-write capacity be the main determining factor. Please note that if your request rate rapidly increased over 300 PUT/LIST/DELETE or 800 GET per second, S3 can bottleneck (source). If it goes over 600 combined "lambdas per second" (twitter registrations, callbacks, and email registrations), you will need to request an increase in your lambda limit(source)
• You don't want to have to deal with waiting for auto-scaling, server crashes, etc.
• You want to save a truckload of money over using ec2 servers (first 1M lambda requests are free)
• You want to seem totally cool using a bunch of cloud technologies to make completely fail-safe, robust back ends. The only way it fails is if your AWS region goes out.
• You are a masochist and can deal with two languages (three if you count SQL), myriad technologies, and the buggy monstrosity that is the AWS Console + DynamoDB's occasionally finicky I/O times all in the same project.

Notes:

If you find this confusing, don't worry. This took me a lot of time to finish and I needed to consult with other people more than once. It took weeks of refinement to get it to a decent place. Also, because of the layout of this guide, some of our decisions aren't going to make sense until later on. For example, why would we build a lambda that always throws errors? (Answer: Only way to redirect with API gateway)(source)

The part with making multiple lambda versions for different stages absolutely sucks. I know. There is nothing that can be done until/if AWS refines their service, unless you skip lambda versioning and staging all together.

Disclaimers:

1) I learned Javascript in 4 hours for this. If you have easy-to-read but clearly better ways to write the lambda code, please let me know and I will publish an update. When I do, you will receive credit at the end of the edited file (by name, email, username, whichever you wish). First-come first-credited.

2) I would love to help you troubleshoot your code, but I unfortunately don't have the time to address issues one-on-one. If you leave me a detailed message about the error you are experiencing with some evidence that it occurs despite directly following the guide I will certainly revisit the post, investigate, and credit you with the changes, but Google + forums are your friend here.

3) This project is all tied together. Cherry picking “how to make an API” or something might leave you disappointed because we make design decisions that are great for our project but are absolutely terrible in others (for example, most API schemes only returning 301 instead of 200).

Why I Built It:

I built this project because my tech lead and I had a brief conversation that went:
Tech Lead: “You built the API in Rails?”
Me: “Yeah, took a few hours because of some weirdness and my juvenile understanding of some keywords, but it's done.”
Lead: “Great. Do it again from scratch in AWS Lambda. I've never done this, but it should be possible. Tell me how it goes.”
I like working with new tech, so I agreed. I looked around and could not find a single how-to guide so I thought that this might actually be useful technology to learn and then share with the world.

Let's begin!

By the way, MIT License. Just check the repo.