In this VERY opinionated post, I will share my thoughts about AWS Lambda extensions, the good and the bad, and when you should or should not use them.

This blog post was originally published on my website, “Ran The Builder.”

Lambda Extensions Introduction

Lambda extensions are an interesting mechanism. AWS recommends using them to enhance your functions with black box capabilities developed by AWS or external providers.

…use Lambda extensions to integrate functions with your preferred monitoring, observability, security, and governance tools — AWS

You can think of them as black boxes that you can attach to your Lambda, which functions as a Lambda layer and enjoy new features.

Here’s a list of classic uses that come to mind:

1. Fetch configuration, secrets, and parameters and store them in the cache. Fetch them automatically once every several minutes. Think of fetching secrets from Secrets Manager or Parameter Store.

2. Send logs, traces, and metrics to an external observability provider such as Lumigo, DataDog, and others.

3. Monitor the Lambda function’s CPU, memory, disk usage, and other interesting machine parameters and send them to a monitoring system, such as the Lambda insights extension.

You can find more extension use cases in this repository. Some are AWS-backed, and some are third-party.

To use a Lambda extension, you only need to attach it as a Lambda layer. Check out my post to learn about Lambda layers and when to use them. I find them particularly useful for deployment optimization.

The Case for Lambda Extensions

On paper, extensions sound like a fantastic mechanism — you add a Lambda layer and a few environment variables, and boom, you are now integrated with an observability third-party provider and get new functionality you didn’t develop.

It gets even better — extension developers can develop extensions in Rust, a compiled language that provides blazing-fast performance, a minor cold start hit, is environment agnostic, and can run on Lambda functions that use a different runtime.

The advantage of an executable is that it is compiled to native code, and is ready to run. The extension is environment agnostic, so it can run alongside with a Lambda function written in another runtime. — Optimizing AWS Lambda extensions in C# and Rust

However, Lambda extensions often come at a cost that outweighs these advantages.

Let’s review my reasons and the use cases for which I currently use extensions.

The Case Against Lambda Extensions

Lambda extensions are a powerful tool that inherit most of its problems from the mechanism it is built upon — Lambda layers. Let’s review the cases against Lambda extensions.

We can divide the cases into several categories:

1. Security

2. Developer experience

3. Performance & cost

Security

Extensions are similar to any open-source SDK library. However, there’s an added twist.

Extensions can expose you to a security risk as you are never sure what is bundled in the layer (some extension providers document and do a good job, but many don’t).

In addition, according to the AWS documentation:

Extensions have access to the same resources as functions. Because extensions are executed within the same environment as the function — AWS documentation

If you use a compromised version of an extension (think of hackers gaining access and publishing their version of the layer in the layer publisher’s official AWS account), they can access whatever resource your function has in their black box running process. So far, this is quite similar to a comprised open-source SDK.

But there’s a plot twist.

Secure CI/CD pipelines include a vulnerability scanner, such as Synk, that scans your Python dependencies files (poetry.toml), checks whether there’s any comprised version, and fails your pipeline before a compromised version is deployed to production.

That’s not the case with Lambda layers and extensions. Their code is added to the Lambda during invocation, where tools like Amazon Inspector can scan the Lambda’s layers, code, and dependencies and find compromised code.

However, that’s too late. Your compromised extension is already running in production. So, there’s an added security risk.

Developer Experience

Setting up a local developer environment with extensions is hard. You need to figure out what external dependencies the extension’s layer brings and install them locally to test your code and debug in the IDE. This can prove challenging, especially in cases where there are version conflicts. A discrepancy between the local developer and Lambda function environments can lead to crashes and bugs that only happen on production and not in local environments.

In addition, some Lambda extensions require your Lambda function’s code to interact with it (most likely via localhost network calls), making it almost impossible to test your code in the IDE. If you have no idea what I’m talking about, check out my AWS re:invent 2023 session and my Lambda testing series, where I show how you can locally test your functions. Extensions break this experience.

Another critical developer experience is related to maintenance and upgrades.

Lambda extensions are enabled via Lambda layers.

Lambda layers have inherent problems:

1. Versioning — Layers are versioned, and your function always consumes a specific version that changes between regions, making your life even more complicated (in one region, it’s version 55, in another, 43). The version is part of the layer ARN.

2. Updates — You need to be aware of a new version (somehow) and manually change it to the latest version. Lambda layers don’t have a package manager like Python and other languages, so updates become a manual endeavor.

3. The total unzipped size of the function and all layers cannot exceed the 250 MB unzipped deployment package size quota. Your extension takes a chunk out of this quota.

Performance & Cost

There are no free meals. Extensions share function resources such as CPU, memory, and storage, and you may see an increased duration of billed function.

In addition, each extension must complete its initialization before Lambda invokes the function. Therefore, an extension that consumes significant initialization time can increase the function’s cold start duration. It may be worth it for you, but it’s a fact you need to be aware of.

When to Use Lambda Extensions

Let’s cover several use cases see if they fit Lambda extensions or not.

Fetch Configuration

I recently read a blog proving that an extension can fetch configuration faster than the Node Lambda function that uses it. That’s probably because the extension is written in Rust, which is faster than Node. However, once the secret is fetched, it is stored in a cache for several minutes. Powertools for AWS, an amazing open-source library, provides the same features without an extension. An extension saves you a few dozen milliseconds in the first call (and for the first call every time the cache expires), but it’s all the same once the cache is not empty. Unless you fetch a ridiculously high volume of secrets in the function, it’s probably not worth adding the complexity and issues mentioned above to save a dozen milliseconds every few minutes.

TL;DR — don’t use an extension.

Send Logs to a 3rd Party Observability Provider

This is one of the most common use cases for using the Lambda extension. It works, and it works well, and you probably don’t need your code to interact with it directly in the IDE. It seems fitting for most companies.

However, coming from a security enterprise, we chose a different route. I wrote a blog post about why it’s better to use a centralized service to send all logs from your AWS account to a 3rd party observability provider instead of having all Lambda functions send them individually. It’s simple when you have few services, but it gets painful at scale, especially when you want to have the same configuration and log filtering governance across hundreds of services. You are better off writing the data to CloudWatch and using a centralized mechanism to send your data to any provider you wish and filter out data you don’t want to send. It’s more secure and performs better, but you pay more (but it’s worth it!). Check my post here for detailed pros and cons.

TL;DR — use an extension but you might want to reconsider at scale.

Monitoring Lambda’s Container Metrics

This is a great use case, and I can’t find any issues with it other than it should come out of the box in Lambda. I don’t want to think about it or attach extensions. I want to enable it via IaC configuration and see the statistics in CloudWatch.

TL;DR — use an extension

Chaos Engineering

Another unique use case with extensions in chaos engineering is where Lambda extensions are a practical tool.

Koby Aharon covered serverless chaos engineering concepts in his two fantastic guest posts on my website and provided implementation details. He conducted his chaos experiment using a Lambda extension. You can review his introduction post here and his experiment details here.

He “installed” the extension at the beginning of the experiment, leaving his deployment code and function code clean of extensions and layers, and removed it once the experiment was done. It is super clean, and he does it only during a chaos experiment, which is done in a separate account, so it does not affect production. In this example, maintenance is less of an issue, and developers don’t need to be aware of it during development. Zero cons, and you get all the extension pros.

TL;DR — use an extension

Now, I haven’t covered all the extensions in the world. Still, these examples cover a large percentage of the widespread use cases.

Summary

In this post, I have reviewed several common Lambda extension use cases.

I have listed the pros and cons of lambda extensions and suggested the most fitting use cases.

Bottom line, if your code actively interacts with the extension, you are doing something wrong, and you should replace the extension with regular Lambda function code or, in most cases, with an open-source library that does the same work, sometimes even with more features. Just because you can *do it with an extension does not mean you *should.

Observability and chaos engineering, on the other hand, are fine examples of proper extension usage.