Hide Your AWS Lambda Environment Variables From Prying Eyes

There is a lot you can write about configuration in the age of cloud-native architectures. And then there are simple steps which are easy to miss and yet they’re crucial: How do you handle sensitive environment variables in AWS Lambda? 🧐

I’ve recently read a solid article about serverless environment variables from Adam DeLong. The author went in quite some detail on how to approach the configuration, especially in the context of serverless framework, and how to use different services AWS offers such as AWS Secrets Manager or Parameter Store. I was missing one particular thing in that article though.

If you configure your AWS Lambda function with environment variables as opposed to e.g. calling the Secrets Manager from within the function itself, how do you prevent these variables from exposition in the AWS Console and API? Because they just sit there, visible to your whole account :)

Here is a screenshot from a test AWS Lambda function in the console:

Unencrypted AWS Lambda environment variables in AWS Console

Oh shoot indeed.

Note	serverless framework doesn’t solve this issue for you automatically. If you refer to a Secrets Manager secret like: `supersecret:${ssm:/aws/reference/secretsmanager...}` then the environment variable will contain a decrypted secret. Their docs mention you have to address it yourself.

The "Encryption configuration" dropdown from the previous picture comes to the rescue:

AWS Lambda encryption configuration in AWS Console

But what does it mean?

Encryption at rest

First of all, AWS Lambda encrypts your functions at rest with a default key aws/lambda which is provided and managed by AWS Key Management Service (KMS). That’s useful, but you can see the environment variables as they are. Can we somehow hide our environment variables from the world?

Changing policy for the default key

You can find the aws/lambda key and its key policy at Key Management Service (KMS) → AWS Managed keys → aws/lambda in the console. The key’s policy allows access to the key through AWS Lambda for everyone who is authorized to use AWS Lambda. One of the default permissions is kms:Decrypt which enables you to see and manage the environment variables.

That gives us the hint how to hide the environment variables. We need to deny kms:Decrypt permission and they will become inaccessible. There are a few ways how to do it. We can’t override the managed key’s policy, but we can create a separate IAM policy and attach it to whatever IAM entity we want:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": ["kms:Decrypt"],
      "Resource": "arn:aws:kms:us-east-1:YOUR_ACCOUNT_ID:key/abcdefgh-your-aws-lambda-key-identifier"
    }
  ]
}

What happens if I add such a policy to my IAM user? Let’s check my function again:

AWS Lambda environment variables in AWS Console after deyning kms:Decrypt permission

Voi-là, the environment variables is visible no more. The policy itself is not limiting that much. You can still manage your function, just not the environment variable part. What about the other option then?

Encryption in Transit

"Enable helpers for encryption in transit" means:

Let’s encrypt the environment variable value client-side, i.e. on your computer.
Store the encrypted value in the function configuration.
Decrypt the value in your function code by yourself.

The encryption in transit requires a customer managed key (CMK) from KMS. It’s not possible to use the managed aws/lambda key. So let’s create a CMK key first.

Create a customer managed key in KMS

It’s a pretty straightforward process. The console has a wizard for it which has a concept of key administrators and key users, based on which it will generate the key policy.

Who do you select as a key user?

The function execution role so you can use the key inside the code.
Any other IAM entity which manages these sensitive environment variables.

Note	A CMK key costs $1 per month and you pay for usage: $0.03 per 10 000 requests in `us-east-1`. The free tier covers 20 000 requests a month across all regions. By the way, AWS managed keys do not cost anything.

Encryption in Transit with AWS Console

Once you’ve got the key with the proper permissions, you can check the checkbox and select it. The console does the heavy-lifting and your only responsibility is to decrypt the value in the code. The console helps with that too in the form of a "Code" button which shows you a decryption code snippet.

AWS Lambda environment variable in AWS console while using encryption in transit

You can play with IAM policies again, e.g. restricting kms:Decrypt to certain IAM users etc., or you can simply modify the key policy further.

Encryption in Transit with AWS CLI

If you want to do this with AWS CLI instead of the console, you have to everything manually, yay.

Encrypt the variable value with KMS

ENCRYPTED_VALUE=$(aws kms encrypt \
                    --key-id $YOUR_CMK_KEY_ID \
                    --plaintext OhShootImEncrypted \
                    --output text \
                    --query CiphertextBlob

Pass it to the function configuration

aws lambda update-function-configuration \
    --function-name my-function \
    --environment "Variables={dbPassword=${ENCRYPTED_VALUE}}"

Profit, i.e. decrypt in the code

Conclusion

Both solutions have their merits. You should pick one for sure.

Restrict kms:Decrypt for aws/lambda key — always useful, I’d use it for smaller things and ad-hoc lambdas personally, it doesn’t cost a dime
Use encryption in transit — structured midsize to large projects, it’ll cost some money, and you need to plug the encryption part somewhere as a part of your deployment process.

Bonus solutions

Don’t use environment variables for sensitive information, fetch the values from elsewhere:

AWS SSM Parameter Store — cheap but rated
AWS SSM Secrets Manager — "expensive" but "scales"
HashiCorp Vault — roll out your own instance and bear the consequences
S3 configuration file with limited access rights, maybe also encrypted, who the heck knows 🤷🏻‍♂️