feat(logger): add logger buffer feature

[leandrodamascena]

🚨 Status

Under Construction: This PR is a work in progress

Issue number: #4432

Summary

Changes

This pull request introduces a new logging buffer mechanism to optimize CloudWatch Logs cost and provide better debugging context during exceptions. Implement a smart log buffering system for AWS Lambda that captures debug context only when exceptions occur, reducing CloudWatch Logs costs.

The logging buffer will utilize a ring buffer implementation to manage memory efficiently, with a configurable buffer size that allows developers to control memory allocation and log retention. The mechanism will be designed with minimal performance overhead, ensuring that the log buffering process does not introduce significant computational cost or latency.

Checklist

Log Buffering

• Add new parameter to logger constructor
• Add new method to buffer log entries
• Modify existing log methods to support buffering configuration
• Ensure log line location tracking
• Ensure precise timestamp tracking

Error Handling

• Implement decorator for exceptions
• Validate flush logic based on configuration flags

Testing

• Write functional tests
• Write e2e tests
• Write unit tests

Documentation

• Improve documentation
• Add detailed usage examples
• Add configuration guide
• Add performance considerations

User experience

from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging.buffer import LoggerBufferConfig
from aws_lambda_powertools.utilities.typing import LambdaContext

logger_buffer = LoggerBufferConfig(max_size=10240, minimum_log_level="DEBUG")
logger = Logger(logger_buffer=logger_buffer)

def lambda_handler(event: dict, context: LambdaContext) -> dict:
    logger.debug("This will be buffered")
    return {"ok"}

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[codecov]

Codecov Report

Attention: Patch coverage is 98.88268% with 2 lines in your changes missing coverage. Please review.

Project coverage is 96.29%. Comparing base (d4310c3) to head (9368459).
Report is 5 commits behind head on develop.

Files with missing lines	Patch %	Lines
aws_lambda_powertools/logging/buffer/cache.py	98.11%	0 Missing and 1 partial ⚠️
aws_lambda_powertools/logging/logger.py	98.57%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #6060      +/-   ##
===========================================
+ Coverage    96.25%   96.29%   +0.03%     
===========================================
  Files          236      240       +4     
  Lines        11407    11571     +164     
  Branches       830      858      +28     
===========================================
+ Hits         10980    11142     +162     
  Misses         337      337              
- Partials        90       92       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[leandrodamascena]

The initial logic of buffering, checking the configuration, flushing and preserving all keys is working. Still need to improve things.

from datetime import datetime
import os
from time import sleep
from aws_lambda_powertools import Logger
from aws_lambda_powertools.logging.buffer import LoggerBufferConfig

os.environ["_X_AMZN_TRACE_ID"] = "123"

logger_buffer_config = LoggerBufferConfig(max_size=10240, minimum_log_level="WARNING")
logger = Logger(level="INFO", logger_buffer=logger_buffer_config)

logger.append_keys(my_key="my_key")

logger.info("my log line with info - buffered")
logger.debug("my log line with debug - buffered")
print("Datetime log creation ---> ", datetime.now().strftime("%d/%m/%Y %H:%M:%S"), "<---- logs must have this timestamp")
sleep(5)
print("Datetime log flush [timestamp must be preserved] ---> ", datetime.now().strftime("%d/%m/%Y %H:%M:%S"))
logger.flush_buffer()

(venv) ➜  new-logger-buffer python meulog.py
Datetime log creation --->  13/02/2025 23:13:28 <---- logs must have this timestamp
Datetime log flush [timestamp must be preserved] --->  13/02/2025 23:13:33
{"level":"INFO","location":"<module>:14","message":"my log line with info - buffered","timestamp":"2025-02-13 23:13:28,744+0000","service":"service_undefined","my_key":"my_key","xray_trace_id":"123"}
{"level":"DEBUG","location":"<module>:15","message":"my log line with debug - buffered","timestamp":"2025-02-13 23:13:28,744+0000","service":"service_undefined","my_key":"my_key","xray_trace_id":"123"}
(venv) ➜  new-logger-buffer 

[github-actions]

⚠️Large PR detected⚠️

Please consider breaking into smaller PRs to avoid significant review delays. Ignore if this PR has naturally grown to this size after reviews.

[github-actions]

⚠️Large PR detected⚠️

Please consider breaking into smaller PRs to avoid significant review delays. Ignore if this PR has naturally grown to this size after reviews.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[leandrodamascena]

It's missing only documentation and I think we are good to go.