Parsing nested envelopes?

[sk0g]

So we have payloads that are essentially Kinesis Firehouse events containing an SQS event. Is there a way to unwrap this using @event_parser()? If not, what's the next best thing?

[leandrodamascena]

Hi @sk0g! Thank you for opening this discussion.

Indeed this is a great improvement for those who use sqs events wrapped in firehose records. I created a PR (#2294) to add this new Model, but before merging this I would like to know how do you ingest SQS into Firehose.

SQS can't send messages directly to Kinesis Firehose by default, so I assume you have a lambda/eventbridge/pipe/other intermediaries to get that message from SQS and ingest. How are you doing this? Is it one on one? Is it a batch? Could you show me a Kinesis Firehose record, please? It can be in base64 anyway, but please remove any sensitive data.

Thank you.

[sk0g]

That's right @leandrodamascena, we have EventBridge in between.

The resulting JSON is something like this:

{
  ...
  "Records": [
    {
      "body": { "actual_data": "some_val" }
      ...
    }
  ]
}

Current flow is to handle it like this:

for record in event["Records"]:
    PydanticModel.parse_raw(event["body"])

 # OR if no batching
 assert len(event["Records"]) == 1
 PydanticModel.parse_raw(event["Records"][0]["body"])

Depending on the event there might be no batching, so it's more straightforward to assert length is 1, and do something like parse. In other cases Firehose dumps JSON bodies without separation so we have an awkward workaround that's not too biting us right now. That's a more complicated case I'm not expecting to be solved by Lambda Powertools. Unless... 🙏

[rubenfonseca]

It seems to me that the event sources can be combined in any way, and we're struggling to keep up with new combinations of possible pairs (or more) of nested envelopes.

I propose we take some time to consider an alternative solution where we implement a PipelineEnvelope and just add as many basic models as we want. The parser would then validate and unwrap the arbitrary combination of Envelopes, thus avoiding us to continue to create and test specific combinations.

What do you think of this @sk0g ? We can work on the developer experience to see if it makes sense.

[sk0g]

Absolutely, I was thinking about that as this case might be common-ish, but in the grand scheme of things you are going to have a n^2 explosion of serialiser classes if you want to address other combinations as well. That's ignoring that you can probably end up with even more nesting...

@rubenfonseca If this can be dynamic it would be brilliant. There's no accounting for what people create, so not having to create functionality for each combination should both provide a simpler API, as well as less feature/ maintenance burden (I hope).

If I could suggest one thing though, with such parsing, testing the handler does become a bit trickier as you have to provide more conformant events. Maybe things like faker.LambdaContext() for a start would be handy, as would request types like EventBridge, SQS, etc. The custom data can still be placed manually without issues, though it would likely be a parameter for the fake request creator. That's a whole other thing though!

[heitorlessa]

Related as it's a request for non-pydantic customers too: #2678

Converting this discussion as a feature so we don't lose tracking of its importance.

@sk0g on fake events (faker), we deprioritized creating a testing feature in Powertools due to bandwidth and needing a separate package (ops overhead). I think we might be better positioned to resume that next year -- offering as a hypothesis strategy is what I have in mind (e.g., mutation testing, fuzzing, composables)