Using powetools as a layer and 'sam local invoke' command

[MrTomerLevi]

Hello,
I'm using SAM and integrated powetools as a lambda layer into my lambda functions.

When runningsam local invoke to test my functions locally, I get a missing module exception at:
from aws_lambda_powertools import Logger, Tracer

This error makes sense since aws_lambda_powertools module is not added to lambda functions locally.
Is there any possible way I can use powertolls layer locally?

** As a workaround, I added aws-lambda-powertools==1.11.0 to my requirements.txt, this enables me to test locally but makes the build process heavier (and my lambda zip file larger)

Thank you all,
Tomer

[heitorlessa]

Hey Tomer - SAM CLI downloads Layers and make them available in the local Docker container, but it sounds like it's not being the case here.

Could you share what version of SAM CLI you're using so we can try reproducing that too?

Pinging @am29d to see if he has the bandwidth to test today, otherwise I'll create a dummy project on Monday to test this out :)

Thanks for reaching out!

[am29d]

I will take a look.

[MrTomerLevi]

Hi @heitorlessa
Thanks for helping.

I'm using sam cli version 1.21.1

[am29d]

Hi Tomer,

thanks for reaching out! I have checked the behaviour on an example project and the correct workflow is to run sam build before sam local invoke. This has nothing to do with the layers or anything. By running the build command sam will create .aws-sam/build/HelloWorldFunction directory with all the dependencies. If you run sam local invoke this build directory will be mounted into the local runtime:

Invoking app.lambda_handler (python3.8)
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.18.2.

Mounting [REDACTED]/sam-example/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 31f4a484-b494-4a23-aa1c-69b862523cb2 Version: $LATEST
[...]
{"statusCode": 200, "body": "{\"hello\": \"world\"}"}

if you run the local invoke command without this folder, the dependencies are missing. So make sure to always build run sam build when you add new dependencies to your requirements.txt or layers in template.yml.

[MrTomerLevi]

Hi Alex,
Thanks for helping.

My SAM template looks like the following:

  AwsLambdaPowertoolsPythonLayer:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer
        SemanticVersion: 1.11.0
  
  AuthorizerLambda:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: task_functions/
      Handler: task_auth.lambda_handler
      Runtime: python3.8
      Environment:
        Variables:
          LOG_LEVEL: INFO
          POWERTOOLS_SERVICE_NAME: AuthorizerLambda
      Layers:
        - !GetAtt AwsLambdaPowertoolsPythonLayer.Outputs.LayerVersionArn

When running:
sam build --use-container

This log message is shown:

2021-03-22 15:16:53 Requested to create CFN template arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer/1.11.0 in serverless application repo.

Same log message appears after I run:
sam local invoke -e events/appsync.json TaskCreateLambda

I couldn't fund aws_lambda_powertools inside my build folder: .aws-sam/build/TaskCreateLambda

[am29d]

Hi Tomer,

I really struggle to reproduce the issue on my machine. Does it also happen without the --use-container flag? I suspect docker does not have access to make API calls to SAR. It would normally create a CFN template and a change set inside internal SAR account (internal S3 account, REDACTED in my logs):

Starting Build inside a container
2021-03-22 16:47:39 {'ResponseMetadata': {'RequestId': 'a02dc369-fc3e-43d6-9430-69ddfb0bcc99', 'HTTPStatusCode': 201, 'HTTPHeaders': {'date': 'Mon, 22 Mar 2021 15:47:39 GMT', 'content-type': 'application/json', 'content-length': '1690', 'connection': 'keep-alive', 'x-amzn-requestid': 'a02dc369-fc3e-43d6-9430-69ddfb0bcc99', 'x-amz-apigw-id': '[REDACTED]', 'x-amzn-trace-id': 'Root=1-6058bc1b-4769f56e28bb911677be9b86'}, 'RetryAttempts': 0}, 'ApplicationId': 'arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer', 'CreationTime': '2021-03-22T15:47:39.413Z', 'ExpirationTime': '2021-03-22T21:47:39.413Z', 'SemanticVersion': '1.11.0', 'Status': 'PREPARING', 'TemplateId': '8e13b173-f9bc-4086-a595-f830193675de', 'TemplateUrl': 'https://awsserverlessrepo-changesets-1xxlnameudh3r.s3.eu-central-1.amazonaws.com/[REDACTED]/arn%3Aaws%3Aserverlessrepo%3Aeu-west-1%3A057560766410%3Aapplications-aws-lambda-powertools-python-layer-versions-1.11.0/964d9809-bf98-4849-af06-32f90da2b7c7.yaml?X-Amz-Security-Token=[REDACTED]%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210322T154739Z&X-Amz-SignedHeaders=host&X-Amz-Expires=21599&X-Amz-Credential=[REDACTED]2%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=[REDACTED]'}
2021-03-22 16:47:40 {'ResponseMetadata': {'RequestId': '2af1da56-c445-46de-96b1-9096cc5005d4', 'HTTPStatusCode': 201, 'HTTPHeaders': {'date': 'Mon, 22 Mar 2021 15:47:40 GMT', 'content-type': 'application/json', 'content-length': '1690', 'connection': 'keep-alive', 'x-amzn-requestid': '2af1da56-c445-46de-96b1-9096cc5005d4', 'x-amz-apigw-id': '[REDACTED]=', 'x-amzn-trace-id': 'Root=1-6058bc1b-7596ce19366d55c006842e02'}, 'RetryAttempts': 0}, 'ApplicationId': 'arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer', 'CreationTime': '2021-03-22T15:47:40.092Z', 'ExpirationTime': '2021-03-22T
...

Can you rule out network connectivity from docker container? 🤔

[am29d]

Followup:

I just turned my wifi off and got the same log message as you, so I highly assume it has something to do with the your docker network connectivity.

[MrTomerLevi]

Same thing without --use-container :)

[am29d]

O_o, this is indeed strange, I will try to reproduce it on a different machine, but it sounds like there is some issue with the SAM CLI in your setup.

[heitorlessa]

Thanks Tomer! I suspect what’s happening here is that SAM CLI cannot resolve that ARN reference because it isn’t deployed yet. I’ll test this hypothesis this afternoon and let you know what I find out — If that doesn’t work at all then we should fall back to provide an opaque Layer version ARN without a SAR App

…

On Mon, 22 Mar 2021 at 14:26, Tomer Levi ***@***.***> wrote: Hi Alex, Thanks for helping. My SAM template looks like the following: AwsLambdaPowertoolsPythonLayer: Type: AWS::Serverless::Application Properties: Location: ApplicationId: arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer SemanticVersion: 1.11.0 AuthorizerLambda: Type: AWS::Serverless::Function Properties: CodeUri: task_functions/ Handler: task_auth.lambda_handler Runtime: python3.8 Environment: Variables: LOG_LEVEL: INFO POWERTOOLS_SERVICE_NAME: AuthorizerLambda Layers: - !GetAtt AwsLambdaPowertoolsPythonLayer.Outputs.LayerVersionArn When running: sam build --use-container This log message is shown: 2021-03-22 15:16:53 Requested to create CFN template arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer/1.11.0 in serverless application repo. Same log message when thing when running: sam local invoke -e events/appsync.json TaskCreateLambda I couldn't fund aws_lambda_powertools inside my build folder: .aws-sam/build/TaskCreateLambda — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#348 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBCVVRW3TY5UYCYEI33TE5ARRANCNFSM4ZLIECQA> .

[MrTomerLevi]

@heitorlessa @am29d thank you for taking the time and helping.
I can open a new discussion/bug in SAM CLI github repo if you think it's irrelevant.

[heitorlessa]

Hey Tomer - It'd be great if you could open an issue within the SAM CLI repo. Also, could you try using this direct Lambda Layer over SAR to double check a theory of mine? arn:aws:lambda:eu-west-1:770693421928:layer:Klayers-python38-aws-lambda-powertools:67 You can change the region to the one you're using. If that works for you, we will prioritize creating a pure Layer as requested here: #329 Thank you!!

…

On Wed, 24 Mar 2021 at 20:50, Tomer Levi ***@***.***> wrote: @heitorlessa <https://github.com/heitorlessa> @am29d <https://github.com/am29d> thank you for taking the time and helping. I can open a new discussion/bug in SAM CLI github repo if you think it's irrelevant. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#348 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBDJ67M3TEPJ3TCYGVDTFI67VANCNFSM4ZLIECQA> .

[MrTomerLevi]

@heitorlessa yes, it works using the direct Lambda layer.
Even using sam build .... --use-container

Thanks!