Have you ever wanted Lambda functions to be able to save and load files locally without needing to transfer data between an S3 bucket? This article is for you.
By using AWS EFS we can attach a persistent filesystem to your Lambda function!
CDK init & deploy
I won’t cover setting up CDK and bootstrapping the environment. You can find that information here.
Once you have set up CDK, we need to set up the project:
mkdir cdk_docker_lambda && cd cdk_docker_lambdacdk init --language pythonsource .venv/bin/activate-
pip install -r requirements.txt && pip install -r requirements-dev.txtNow deploy empty stack to AWS:
cdk deploy
Stack design
Our CDK stack is going to deploy an EFS filesystem, a Lambda function, and an access point which will allow us to attach the filesystem to the function.
cdk_lambda_efs/cdk_lambda_efs_stack.py
from aws_cdk import Stack from aws_cdk import aws_ec2 as ec2 from aws_cdk import aws_efs as efs from aws_cdk import aws_lambda as _lambda from constructs import Construct from aws_cdk import RemovalPolicy class CdkLambdaEfsStack(Stack): def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: super().__init__(scope, construct_id, **kwargs) # I like to define all my pieces in __init__ self.vpc = None self.access_point = None self.lambda_func = None self.build_infrastructure() def build_infrastructure(self): self.build_vpc() self.build_filesystem() self.build_lambda_func() def build_vpc(self): # Build the VPC where both EFS and Lambda will sit self.vpc = ec2.Vpc(self, "VPC") def build_filesystem(self): file_system = efs.FileSystem( scope=self, id="Efs", vpc=self.vpc, file_system_name="ExampleLambdaAttachedEFS", # Makes sure to delete EFS when stack goes down removal_policy=RemovalPolicy.DESTROY, ) # create a new access point from the filesystem self.access_point = file_system.add_access_point( "AccessPoint", # set /export/lambda as the root of the access point path="/export/lambda", # as /export/lambda does not exist in a new efs filesystem, the efs will create the directory with the following createAcl create_acl=efs.Acl( owner_uid="1001", owner_gid="1001", permissions="750" ), # enforce the POSIX identity so lambda function will access with this identity posix_user=efs.PosixUser(uid="1001", gid="1001"), ) def build_lambda_func(self): # I'm just using the normal CDK lambda function here. See my other articles for additional building methods. _lambda.Function( self, "LambdaWithEFS", runtime=_lambda.Runtime.PYTHON_3_9, # lambda function file name.handler function handler="lambda_EFS.handler", # Points to directory of lambda function code=_lambda.Code.from_asset("cdk_lambda_efs/lambda_EFS"), # Lambda needs to be in same VPC as EFS vpc=self.vpc, filesystem=_lambda.FileSystem.from_efs_access_point( ap=self.access_point, mount_path="/mnt/filesystem" ) if self.access_point else None, ) Lambda function
I will deploy a lambda function without any additional dependencies. If you need dependencies, you will need to use different CDK constructs to do it. Here is an example of using aws_lambda_python_alpha and here is an example of building using Docker.
This lambda function opens a file on the EFS filesystem, writes a string to it, then opens it again and returns the result.
cdk_lambda_efs/lambda_EFS/lambda_EFS.py
from pathlib import Path def handler(event, context): # Writing to a file on the EFS filesystem path = Path("/mnt/filesystem/test.txt") with path.open("w") as f: f.write("Test123") # Now open the file, read the text, return with path.open("r") as f: text = f.read() return f"Hello Lambda! {text}" Test the lambda function with attached filesystem
Navigate to the Lambda console on AWS. First notice the filesystem has been successfully attached to your lambda function
Now go ahead and test it using any kind of event.
I hope this article has helped you to solve your problem of lambda not persisting data. EFS is an easy to use and versatile solution to memory problems.
Make sure to cdk destroy when you're done to avoid any charges.
Top comments (0)