The problem

How can code running in the managed AWS Lambda environment call services that use private certificates for HTTPS?

The majority of enterprises moving to AWS or other cloud platforms have existing on-premises applications, and there is often a need for the new cloud based applications to talk back to services on-prem. Typically this done with a hybrid network where the corporate network and AWS VPC(s) are connected using a VPN or Direct Connect.

Let's assume that the network has been set-up and the on-prem service is either using public DNS or a solution like Route 53 resolver has been configured and it's possible to establish a connection to the on-premises service.

If the service is using a public certificate (issued by a public certificate authority that the Lambda environment is aware of and has a copy of the root certificate public key) then there will be no problems, but if the enterprise is using their own private Certificate Authority then there will be an error making HTTPS calls.

Enterprises may have their own CA for services that were only ever expected to be used internally. It gives greater control over issuing and revoking certificates. Typically the CA certificate will be installed onto desktops and services at the enterprise to ensure that connections to these services are trusted in browsers and between services.

If this were an EC2 instance calling back to on-prem then you would probably install the certificate as part of the instance set-up, but we have less control in the transient Lambda environment.

It's possible to write code to handle the TLS failure, or even to disable certificate validation entirely (please don't do this), but what if there was a better way to have the lambda function transparently trust the private certificate?

The solution

Lambda Layers to the rescue!

Many TLS/SSL libraries or application frameworks have mechanisms to add additional root certificates to the "trust store". Once this is configured, any connections using certificates that were signed by the private root certificate will be automatically trusted.

On Linux (and macOS) .NET Core uses OpenSSL for cryptography and OpenSSL allows you to add additional root certificates from a file (in PEM format) using the SSL_CERT_FILE environment variable. The root certificate doesn't need to be "installed" into the environment.

Setting environment variables is easy for Lambda functions, but where should this certificate file be, and how do we get it into the Lambda environment?

The contents of the Lambda zip file are extracted to /var/task, and it's possible to include the certificate file here and point SSL_CERT_FILE to a location in this directory, however this has the drawback that every lambda zip file needs to contain the certificate file and you either need to include it into every repository, or include it as part of your CI/CD process.

Lambda Layers have two main use-cases: sharing dependencies (typically code or libraries, but can be configuration like this) or creating a custom runtime. The contents of the layer are extracted to /opt.

If we build a layer containing our certificate file it can be re-used across as many lambda functions as we like.

Building and sharing the layer

This SAM template will build a Certificate Lambda Layer including files in the certs sub-directory in the layer zip file.

The ARN for the layer is placed in a Parameter Store value that can be referenced by the templates for Lambda functions.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: SAM Template for certificate layer

Resources:
  CertificateLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      CompatibleRuntimes:
        - dotnetcore2.1
        - dotnetcore3.1
      ContentUri: ./certs/
      Description: Layer containing additional certificates
      RetentionPolicy: Retain

  LayerParameter:
    Type: AWS::SSM::Parameter
    Properties:
      Description: ARN for latest Certificate Layer
      Name: /Lambda/Layers/Certificate
      Type: String
      Value: !Ref CertificateLayer

Within the certs sub-directory there should be a file containing one or more root certificates in PEM format. Typically this file will have a .crt extension, so let's call it additional-certificates.crt.

If we need to add additional root certificates (or remove expired ones), the layer can be updated, which will create a new version. Existing functions will continue to use the old version, but the Parameter Store key will now point to the ARN of the new version so any new deployments will pick up the change.

Why not a CloudFormation export? CloudFormation exports are good for things that never change, but you can't update the value of an export that's in use by another stack.

In the parameters of your CloudFormation stacks containing your Lambda functions, include a parameter to pull in the value from Parameter store and then use it with the function, also setting the SSL_CERT_FILE environment variable:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Lambda Function

Parameters:
  CertificateLayer:
    Default: /Lambda/Layers/Certificate
    Description: Certificate Layer ARN
    Type: AWS::SSM::Parameter::Value<String>

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          SSL_CERT_FILE: /opt/additional-certificates.crt
      Layers:
        - !Ref CertificateLayer
      # Plus all the other properties

Once again we're using SAM here, although there is unfortunately an issue with the SAM CLI build command where it does not like the use of parameter store values for layers. There are some work arounds described in the GitHub issue.

Other runtimes

I've used this approach with .NET Core lambda functions, but this layer may be useful for other lambda runtimes.

For example, the popular Python requests library also allows you to configure certificates using the REQUESTS_CA_BUNDLE environment variable.