This is the second blog post of this two-part series that uses a practical application to demonstrate how to integrate Redis with AWS Lambda. The first part was about the solution overview, deployment and hopefully you were able to try it out end to end. As promised, the second part (this one) will cover the Infrastructure aspects (IaaC to be specific) which is comprised of three (CDK) Stacks (in the context of a single CDK App).

I will provide a walk through of the CDK code which is written in Go, thanks to the CDK Go support (which is Developer Preview at the time of writing).

CDK code walk through

Let's take it one stack at a time

Please note that some of the code has been redacted/omitted for brevity - you can always refer to complete code in the GitHub repo

Start with the infrastructure stack

stack := awscdk.NewStack(scope, &id, &sprops)
vpc = awsec2.NewVpc(stack, jsii.String("demo-vpc"), nil)

authInfo := map[string]interface{}{"Type": "password", "Passwords": []string{memorydbPassword}}
user = awsmemorydb.NewCfnUser(stack, jsii.String("demo-memorydb-user"), &awsmemorydb.CfnUserProps{UserName: jsii.String("demo-user"), AccessString: jsii.String(accessString), AuthenticationMode: authInfo})
acl := awsmemorydb.NewCfnACL(stack, jsii.String("demo-memorydb-acl"), &awsmemorydb.CfnACLProps{AclName: jsii.String("demo-memorydb-acl"), UserNames: &[]*string{user.UserName()}})

//snip .....

subnetGroup := awsmemorydb.NewCfnSubnetGroup(stack, jsii.String("demo-memorydb-subnetgroup"), &awsmemorydb.CfnSubnetGroupProps{SubnetGroupName: jsii.String("demo-memorydb-subnetgroup"), SubnetIds: &subnetIDsForSubnetGroup})

memorydbSecurityGroup = awsec2.NewSecurityGroup(stack, jsii.String("memorydb-demo-sg"), &awsec2.SecurityGroupProps{Vpc: vpc, SecurityGroupName: jsii.String("memorydb-demo-sg"), AllowAllOutbound: jsii.Bool(true)})

memorydbCluster = awsmemorydb.NewCfnCluster(//... details omitted)

//...snip

twitterIngestFunctionSecurityGroup = awsec2.NewSecurityGroup(//... details omitted)
twitterLeaderboardFunctionSecurityGroup = awsec2.NewSecurityGroup(//... details omitted)

memorydbSecurityGroup.AddIngressRule(//... details omitted)
memorydbSecurityGroup.AddIngressRule(//... details omitted)

To summarise:

A single line of code to create VPC and related components!
We create ACL, User, Subnet group for MemoryDB cluster and refer to them when during cluster creation with awsmemorydb.NewCfnCluster
We also create required security groups - their main role is to allow Lambda functions to access MemoryDB (we specify explicit Inbound rules to make that possible)
- One for MemoryDB cluster
- One each for both the Lambda functions

The next stack deploys the tweets ingestion Lambda Function

//....

memoryDBEndpointURL := fmt.Sprintf("%s:%s", *memorydbCluster.AttrClusterEndpointAddress(), strconv.Itoa(int(*memorydbCluster.Port())))

lambdaEnvVars := &map[string]*string{"MEMORYDB_ENDPOINT": jsii.String(memoryDBEndpointURL), "MEMORYDB_USER": user.UserName(), "MEMORYDB_PASSWORD": jsii.String(getMemorydbPassword()), "TWITTER_API_KEY": jsii.String(getTwitterAPIKey()), "TWITTER_API_SECRET": jsii.String(getTwitterAPISecret()), "TWITTER_ACCESS_TOKEN": jsii.String(getTwitterAccessToken()), "TWITTER_ACCESS_TOKEN_SECRET": jsii.String(getTwitterAccessTokenSecret())}

awslambda.NewDockerImageFunction(stack, jsii.String("lambda-memorydb-func"), &awslambda.DockerImageFunctionProps{FunctionName: jsii.String(tweetIngestionFunctionName), Environment: lambdaEnvVars, Timeout: awscdk.Duration_Seconds(jsii.Number(20)), Code: awslambda.DockerImageCode_FromImageAsset(jsii.String(tweetIngestionFunctionPath), nil), Vpc: vpc, VpcSubnets: &awsec2.SubnetSelection{Subnets: vpc.PrivateSubnets()}, SecurityGroups: &[]awsec2.ISecurityGroup{twitterIngestFunctionSecurityGroup}})

//....

It's quite simple compared to the previous stack. We define the environment variables required by our Lambda function (including Twitter API credentials) and deploy it as a Docker image.

For the function to be packaged as a Docker image, I used the Go:1.x base image. But, you can explore other options as well. During deployment, the Docker image is built locally, pushed to a private ECR registry and finally the Lambda function is created - all this, with a few lines of code!

Notice that the MemoryDB cluster and security group are automatically referred/looked-up from the previous stack (not re-created!).

Finally, the third stack takes care of the leaderboard Lambda function

It's quite similar to the previous one, except for the addition of the Lambda Function URL (awslambda.NewFunctionUrl) which we use the output for the stack:

//....
memoryDBEndpointURL := fmt.Sprintf("%s:%s", *memorydbCluster.AttrClusterEndpointAddress(), strconv.Itoa(int(*memorydbCluster.Port())))

lambdaEnvVars := &map[string]*string{"MEMORYDB_ENDPOINT": jsii.String(memoryDBEndpointURL), "MEMORYDB_USERNAME": user.UserName(), "MEMORYDB_PASSWORD": jsii.String(getMemorydbPassword())}

function := awslambda.NewDockerImageFunction(stack, jsii.String("twitter-hashtag-leaderboard"), &awslambda.DockerImageFunctionProps{FunctionName: jsii.String(hashtagLeaderboardFunctionName), Environment: lambdaEnvVars, Code: awslambda.DockerImageCode_FromImageAsset(jsii.String(hashtagLeaderboardFunctionPath), nil), Timeout: awscdk.Duration_Seconds(jsii.Number(5)), Vpc: vpc, VpcSubnets: &awsec2.SubnetSelection{Subnets: vpc.PrivateSubnets()}, SecurityGroups: &[]awsec2.ISecurityGroup{twitterLeaderboardFunctionSecurityGroup}})

funcURL := awslambda.NewFunctionUrl(stack, jsii.String("func-url"), &awslambda.FunctionUrlProps{AuthType: awslambda.FunctionUrlAuthType_NONE, Function: function})

awscdk.NewCfnOutput(stack, jsii.String("Function URL"), &awscdk.CfnOutputProps{Value: funcURL.Url()})

That's all for this blog post. Closing off with links to AWS Go CDK v2 references:

For MemoryDB - https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awsmemorydb
For Lambda - https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awslambda
For VPC etc. - https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2/awsec2
CDK V2 https://pkg.go.dev/github.com/aws/aws-cdk-go/awscdk/v2

This concludes the two-part series. Stay tuned for more and as always, Happy Coding!

Twitter Leaderboard app with Redis and AWS Lambda (part 2)

CDK code walk through