Testing strategies

[saragerion]

To incrementally, iteratively increase the confidence over our code and mitigate failures in production, I am in favour of implementing automated "safety nets" as part of our CI/CD pipelines.
The idea is not that they will be bullet proof, but they will allow us to capture bugs fast and understand better the failure modes of our business logic.

The scope of this discussion is around (automated) tests, included but not limited to:

• component tests
• integration tests
• E2E tests
• acceptance tests
• smoke tests
• load tests
• stress tests

Additionally, it would be really nice if we can run our CI/CD pipeline tests on environments supported by the Lambda runtime:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
Node.js 14, Node.js 12, Node.js 10
SDK for javascript 2.1001.0
Amazon Linux 2
x86_64, arm64

[flochaz]

Here is a PR with a proposal for e2e tests #326 . It is based on jest and therefore can be easily extended to test all combination we want in a real cloud env.

[flochaz]

We will need to add tests evaluating packaging size and add in e2e tests evaluation of coldstart duration (as easy as grab metrics and assert on value)

[flochaz]

what about combination of modules ? we may want to have tests in global examples

[saragerion]

Agreed E2e scenarios (global examples at the root of the repository):

• 1 example combining all three packages with middleware
• 1 example combining all three packages with decorators
• 1 example combining all three packages with manual approach

[saragerion]

Note: we should keep an eye on this issue to make sure we are able to support the new node version when it's out, as it will be used by members of the AWS community:
aws/aws-lambda-base-images#14

Related twitter thread: https://twitter.com/martindonadieu/status/1479007397498785794?t=20-x8rnpsu5Y3OJck1UYhw&s=19

[fitzgerald1337]

make sure we are able to support the new node version when it's out

by it, do you mean the new node version? because if so, it has been out (by out, we all actually mean in active LTS support, right?) for over a year... that's the entire point of the aws-lambda-base-images GitHub discussion that you linked.

[dreamorosi]

As stated in the project's tenets we optimise for AWS Lambda function environments and supported runtimes only. Utilities might work with web frameworks and non-Lambda environments, though they are not officially supported. We are aware that Node v16 has entered LTS support a couple of months ago but at the moment this version is not yet officially supported as Node.js runtime by AWS Lambda.

As such, like Sara said, for what concerns Powertools for Typescript we'll be keeping an eye on the release schedule and start testing for v16.x once it's available in Lambda.

[simonireilly]

Thanks for the great library, in particular, I look forward to #364, as controversial as I am sure it will become.

I had a chance to take a look at benchmarking the logger and found it to be about an order of magnitude above other perhaps best in class loggers.

simonireilly#1

Results:

benchLogger*10000: 12588ms
benchConsole*10000: 7.372s
benchPino*10000: 497.632ms

Simplified analysis, 12588 / 497 = 25; therefore to use this logger it would cost 25 times more on a lambda bill for the equivalent number of lines of logs.

I don't think this is the place to discuss performance given the library is beta, and the community works towards improvements every day 🥳

I think though, some perf testing to prevent regressions that can run far left in the software delivery lifecycle would be useful for the power tools.

[saragerion]

Thanks for your comment, your kind supportive words, and taking the time to measure our utilities' performance.

I think though, some perf testing to prevent regressions that can run far left in the software delivery lifecycle would be useful for the power tools.

I agree with you 10000%.
Performance is important and does have an impact on the utility of this libraries.
I have created a related issue and referenced this discussion: #398 (comment)