feat(idempotency): leverage new DynamoDB Failed conditional writes behavior with ReturnValuesOnConditionCheckFailure

[dastra]

Issue number: #3327

Summary

Changes

This feature passes ReturnValuesOnConditionCheckFailure when putting the idempotency record in DynamoDB. If the put fails, then a copy of the item as it was during the failed write attempt is returned in the response. This avoids having to perform a separate get request to retrieve the item.

Detail:

• ReturnValuesOnConditionCheckFailure is only supported in boto3 >=1.26.164. As the default version of boto3 installed within lambda for pythons 3.7-3.11 is 1.26.90, I've made the new functionality conditional on a library version check rather than requiring an upgrade of boto3
• aws_lambda_powertools.utilities.idempotency.persistence.dynamodb._put_record now checks whether boto3's version is >=1.26.164, and if so passes ReturnValuesOnConditionCheckFailure. If the conditional check fails, then it adds the item returned to the IdempotencyItemAlreadyExistsError exception before raising it.
• aws_lambda_powertools.utilities.idempotency.base._process_idempotency extracts the item from the exception. Where it doesn't exist, it calls _get_idempotency_record as before to get the item.
• These changes meant that I had to move the DataRecord class into its utilities/idempotency/persistence/datarecord.py to avoid a circular dependency, as it's now imported into aws_lambda_powertools.utilities.idempotency.exceptions
• I've updated the documentation to reflect that there's no RCU consumed. To keep the docs simple, I've not added the detail that an RCU is consumed when boto3 < 1.26.164. Happy to add this detail if you think it's worth this level of clarity.

User experience

The user experience is identical before and after the change. However as there is no subsequent get when the conditional check fails, there is no associated RCU consumed.

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[boring-cyborg]

Thanks a lot for your first contribution! Please check out our contributing guidelines and don't hesitate to ask whatever you need.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
1 Security Hotspot
3 Code Smells

No Coverage information
0.0% Duplication

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

[heitorlessa]

hey @dastra stellar work - setting expectation that we might be able to review and give you feedback next week only; we're catching up to the re:Invent backlog as the team was out.

[codecov-commenter]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (3c63c24) 95.56% compared to head (85b829c) 95.57%.
Report is 3 commits behind head on develop.

Files	Patch %	Lines
...ws_lambda_powertools/utilities/idempotency/base.py	80.00%	0 Missing and 1 partial ⚠️
...wertools/utilities/idempotency/persistence/base.py	88.88%	0 Missing and 1 partial ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3446      +/-   ##
===========================================
+ Coverage    95.56%   95.57%   +0.01%     
===========================================
  Files          213      214       +1     
  Lines         9961     9993      +32     
  Branches      1824     1827       +3     
===========================================
+ Hits          9519     9551      +32     
  Misses         329      329              
  Partials       113      113              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[leandrodamascena]

Hello @dastra! This is a great job and a great improvement for our customers, after all, we are reducing costs by removing a get_item call and even reducing network latency, as now the data stored in the DDB is returned in a single call.

I made some comments but this PR is super! I'll make sure we have this merged as soon as possible.

[leandrodamascena]

Hello @dastra! Thanks for addressing the last few review comments! I just left a comment about a small change.

I'm still working on our documentation because now we have 3 possible workflows for successful executions, and each of them makes a different number of requests. So we must document it properly to make it clear to our customers.

First execution

PutItem (1 DDB request) -> execute Lambda -> UpdateItem with Lambda response (1 DDB request) = 2 requests total

Second execution (record exists and boto3 supports ReturnValuesOnConditionCheckFailure)

PutItem (1 DDB request) -> The conditional check fails with existing record -> Return Item from exception = 1 request total

Second execution with legacy behavior (record exists and boto3 doesn't support ReturnValuesOnConditionCheckFailure)

PutItem (1 request) - The conditional check fails with the existing record -> GetItem (1 DDB request) = 2 requests total

[leandrodamascena]

Items missing before we merge this:

• Modify our documentation
• Validate this changes with end-2-end tests in different versions of Lambda Runtime/boto3
• Final review

[leandrodamascena]

This is ready for review @rubenfonseca! Please review it.

[rubenfonseca]

Sorry, just one more tiny thing

[sonarqubecloud]

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

3 New issues
0 Security Hotspots
No data about Coverage
0.6% Duplication on New Code

See analysis details on SonarCloud

[leandrodamascena]

Thank you for this stellar work @dastra!

APPROVED!

[boring-cyborg]

Awesome work, congrats on your first merged pull request and thank you for helping improve everyone's experience!