You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The below breaking changes were made in version 3.0.0. All other functionality is backwards compatible.
9
+
10
+
### Lua support removed
11
+
12
+
Support for Lua transformations has been removed. If you are running a Lua transformation, you can port the logic to [Javascript](/docs/destinations/forwarding-events/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/index.md) or [JQ](/docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/jq.md).
13
+
14
+
## HTTP target: non-JSON data no longer supported
15
+
16
+
We never intended to support non-JSON data, but prior to version 3.0.0, the request body was simply populated with whatever bytes were found in the message data, regardless of whether it is valid JSON.
17
+
18
+
From version 3.0.0 onwards, only valid JSON will work, otherwise the message will be considered invalid and sent to the failure target.
19
+
20
+
## HTTP target: request batching
21
+
22
+
Many HTTP APIs allow sending several events in a single request by putting them into a JSON array. Since version 3.0.0, if the Snowbridge source provides data in batches, the HTTP target will batch events in this way.
23
+
24
+
As a consequence, even when the source provides events in a single event batch, it will now be placed into an array of one element. For example, prior to version 3.0.0, a request body might look like this:
25
+
26
+
```
27
+
{"foo": "bar"}
28
+
```
29
+
30
+
But it will now look like this:
31
+
32
+
```
33
+
[{"foo": "bar"}]
34
+
```
35
+
36
+
As of version 3.0.0, the SQS source provides events in batches of up to ten, and the Kinesis, Kafka, and Pubsub and Stdin sources provide events in single-event batches. This behavior will likely change in a future version.
37
+
38
+
You can preserve the previous behavior and ensure that requests are always single-event non-array objects, even with a batching source. To do so, set `request_max_messages` to 1, and provide this template (as long as your data is valid JSON):
Copy file name to clipboardExpand all lines: docs/destinations/forwarding-events/snowbridge/concepts/failure-model/index.md
+11-5Lines changed: 11 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,11 +18,15 @@ There are several different failures that Snowbridge may hit.
18
18
19
19
### Target failure
20
20
21
-
This is where a request to the destination technology fails or is rejected - for example a http 400 response is received. When Snowbridge hits this failure, it will retry 5 times. If all 5 attempts fail, it will be reported as a 'MsgFailed' for monitoring purposes, and will proceed without acking the failed Messages. As long as the source's acking model allows for it, these will be re-processed through Snowbridge again.
21
+
This is where a request to the destination technology fails or is rejected - for example a HTTP 400 response is received.
22
22
23
-
Note that this means failures on the receiving end (eg. if an endpoint is unavailable), mean Snowbridge will continue to attempt to process the data until the issue is fixed.
23
+
Retry behavior for target failures is determined by the retry configuration. You can find details of this in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/retries/index.md).
24
24
25
-
As of Snowbridge 2.4.2, the kinesis target does not treat kinesis write throughput exceptions as this type of failure. Rather it has an in-built backoff and retry, which will persist until each event in the batch is either successful, or fails for a different reason.
25
+
As of Snowbridge 2.4.2, the Kinesis target does not treat kinesis write throughput exceptions as this type of failure. Rather it has an in-built backoff and retry, which will persist until each event in the batch is either successful, or fails for a different reason.
26
+
27
+
Before version 3.0.0, Snowbridge treated every kind of target failure the same - it would retry 5 times. If all 5 attempts failed, it would proceed without acking the failed Messages. As long as the source's acking model allows for it, these would be re-processed through Snowbridge again.
28
+
29
+
Each target failure attempt will be reported as a 'MsgFailed' for monitoring purposes.
26
30
27
31
### Oversized data
28
32
@@ -34,6 +38,8 @@ Writes of oversized messages to the failure target will be recorded with 'Oversi
34
38
35
39
In the unlikely event that Snowbridge encounters data which is invalid for the target destination (for example empty data is invalid for pubsub), it will create a [generic error failed event](/docs/fundamentals/failed-events/index.md#generic-error), emit it to the failure target, and ack the original message.
36
40
41
+
As of version 3.0.0, the HTTP target may produce 'invalid' type failures. This occurs when: the a POST request body cannot be formed; the templating feature's attempts to template data result in an error; or the response conforms to a response rules configuration which specifies that the failure is to be treated as invalid. You can find more details in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md).
42
+
37
43
Transformation failures are also treated as invalid, as described below.
38
44
39
45
Writes of invalid messages to the failure target will be recorded with 'InvalidMsg' statistics in monitoring. Any failure to write to the failure target will cause a [fatal failure](#fatal-failure).
@@ -54,6 +60,6 @@ Firstly, if it hits an error in retrieving data from the source stream, it will
54
60
55
61
Secondly, as described above, where there are failures it will attempt to reprocess the data if it can, and where failures aren't recoverable it will attempt to handle that via a failure target. Normally, even reaching this point is rare.
56
62
57
-
In the very unlikely event that Snowbridge reaches this point and cannot write to a failure target, the app will crash. Should this happen, and the app is re-deployed, it will begin processing data from the last acked message. Note that the likely impact of this is duplicated sends to the target, but not data loss.
63
+
In the very unlikely event that Snowbridge reaches this point and cannot write to a failure target, the app will crash. Should this happen, and the app is re-deployed, it will begin processing data from the last acked message. Note that the likely impact of this is duplicated sends to the target, but not data loss.
58
64
59
-
Of course, if you experience crashes or other issues that are not explained by the above, please log an issue detailing the bahaviour.
65
+
Of course, if you experience crashes or other issues that are not explained by the above, please log an issue detailing the behavior.
|`target_success`| Events successfully sent to the target. |
37
-
|`target_failed`| Events which failed to reach the target, after 5 retries. Will be retried later. |
37
+
|`target_success`| Events successfully sent to the target. |
38
+
|`target_failed`| Events which failed to reach the target, and will be handled by the retry config. Retries which fail are also counted.|
38
39
|`message_filtered`| Events filtered out via transformation. |
39
-
|`failure_target_success`| Events we could not send to the target, which are not retryable, successfully sent to the failure target. |
40
+
|`failure_target_success`| Events we could not send to the target, which are not retryable, successfully sent to the failure target. |
40
41
|`failure_target_failed`| Events we could not send to the target, which are not retryable, which we failed to send to the failure target. In this scenario, Snowbridge will crash. |
41
42
|`min_processing_latency`| Min time between entering Snowbridge and write to target/failure target. |
42
43
|`max_processing_latency`| Max time between entering Snowbridge and write to target/failure target. |
This feature is in beta status because we may make breaking changes in future versions.
10
+
:::
11
+
12
+
This feature allows you to configure the retry behavior when the target encounters a failure in sending the data. There are two types of failure you can define:
13
+
14
+
A **transient failure** is a failure which we expect to succeed again on retry. For example, some temporary network error, or when we encounter throttling. Typically, you would configure a short backoff for this type of failure. When we encounter a transient failure, we keep processing the rest of the data as normal, under the expectation that everything is operating as normal. The failed data is retried after a backoff.
15
+
16
+
A **setup failure** is one we don't expect to be immediately resolved, for example an incorrect address, or an invalid API Key. Typically, you would configure a long backoff for this type of failure, under the assumption that the issue needs to be fixed with either a configuration change or a change to the target itself (e.g. permissions need to be granted). Setup errors will be retried 5 times before the app crashes.
17
+
18
+
As of version 3.0.0, only the http target can be configured to return setup errors, via the response rules feature - see [the http target configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). For all other targets, all errors returned will be considered transient, and behavior can be configured using the `transient` block of the retry configuration.
19
+
20
+
Retries will be attempted with an exponential backoff. In other words, on each subsequent failure, the backoff time will double. You can configure transient failures to be retried indefinitely by setting `max_attempts` to 0.
0 commit comments