Snowbridge v3

docs/destinations/forwarding-events/snowbridge/3-X-X-upgrade-guide/index.md

@@ -0,0 +1,47 @@
+---
+title: "3.X.X upgrade guide"
+sidebar_position: 400
+---
+
+## Breaking Changes
+
+The below breaking changes were made in version 3.0.0. All other functionality is backwards compatible.
+
+### Lua support removed
+
+Support for Lua transformations has been removed. If you are running a Lua transformation, you can port the logic to [Javascript](/docs/destinations/forwarding-events/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/index.md) or [JQ](/docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/jq.md).
+
+### HTTP Target Changes
+
+Previously, by default the body of HTTP requests came in whatever form it was received. It is now an array. From v3, where no template is configured, the POST request body will contain an array of JSON containing the data for the whole batch. Data must be valid JSON or it will be considered invalid and sent to the failure target.
+
+Note that this is a breaking change to the pre-v3 default behaviour, in two ways:
+
+1. Prior to v3, we sent data one request per message
+
+This means that where no template is provided, request bodies will now be arrays of JSON rather than individual JSON objects. 
+
+For example, pre-v3, a request body might look like this:
+
+```
+{"foo": "bar"}
+```
+
+But it will now look like this:
+
+```
+[{"foo": "bar"}]
+```
+
+If you need to preserve the previous behaviour (as long as your data is valid JSON), you can set `request_max_messages` to 1, and provide this template:
+
+```go reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/targets/http-template-unwrap-example.file
+```
+
+2. Non-JSON data is not supported
+
+While the intention was never to support non-JSON data, prior to v3, the request body was simply populated with whatever bytes were found in the message data, regardless of whether it was valid JSON.
+
+From v3 on, only valid JSON will work, otherwise the message will be considered invalid and sent to the failure target.
+

docs/destinations/forwarding-events/snowbridge/concepts/failure-model/index.md

@@ -18,12 +18,16 @@ There are several different failures that Snowbridge may hit.
 
 ### Target failure
 
-This is where a request to the destination technology fails or is rejected - for example a http 400 response is received. When Snowbridge hits this failure, it will retry 5 times. If all 5 attempts fail, it will be reported as a 'MsgFailed' for monitoring purposes, and will proceed without acking the failed Messages. As long as the source's acking model allows for it, these will be re-processed through Snowbridge again.
+This is where a request to the destination technology fails or is rejected - for example a http 400 response is received. 
 
-Note that this means failures on the receiving end (eg. if an endpoint is unavailable), mean Snowbridge will continue to attempt to process the data until the issue is fixed.
+Retry behaviour for target failures is determined by the retry configuration. You can find details of this in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/retries/index.md).
 
 As of Snowbridge 2.4.2, the kinesis target does not treat kinesis write throughput exceptions as this type of failure. Rather it has an in-built backoff and retry, which will persist until each event in the batch is either successful, or fails for a different reason.
 
+Before version 3.0.0, Snowbridge treats every kind of target failure the same -  it will retry 5 times. If all 5 attempts fail, it will proceed without acking the failed Messages. As long as the source's acking model allows for it, these will be re-processed through Snowbridge again.
+
+Each target failure attempt will be reported as a 'MsgFailed' for monitoring purposes.
+
 ### Oversized data
 
 Targets have limits to the size of a single message. Where the destination technology has a hard limit, targets are hardcoded to that limit. Otherwise, this is a configurable option in the target configuration. When a message's data is above this limit, Snowbridge will produce a [size violation failed event](/docs/fundamentals/failed-events/index.md#size-violation), and emit it to the failure target.
@@ -34,6 +38,8 @@ Writes of oversized messages to the failure target will be recorded with 'Oversi
 
 In the unlikely event that Snowbridge encounters data which is invalid for the target destination (for example empty data is invalid for pubsub), it will create a [generic error failed event](/docs/fundamentals/failed-events/index.md#generic-error),  emit it to the failure target, and ack the original message.
 
+As of version 3.0.0, the http target may produce 'invalid' type failures. This occurs when the a POST request body cannot be formed, when the templating feature's attempts to template data result in an error, or when the response conforms to a response rules configuration which sepcifies that the failure is to be treated as invalid. You can find more details in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md).
+
 Transformation failures are also treated as invalid, as described below.
 
 Writes of invalid messages to the failure target will be recorded with 'InvalidMsg' statistics in monitoring. Any failure to write to the failure target will cause a [fatal failure](#fatal-failure).

docs/destinations/forwarding-events/snowbridge/configuration/monitoring/index.md

@@ -23,7 +23,8 @@ https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/mon
 ```hcl reference
 https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/monitoring/sentry-example.hcl
 ```
-### StatsD stats receiver 
+
+### StatsD stats receiver configuration
 
 ```hcl reference
 https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/monitoring/statsd-example.hcl
@@ -33,10 +34,10 @@ Snowbridge sends the following metrics to statsd:
 
 | Metric                   | Definitions                                                                                                                                             |
 |--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `target_success`         | Events successfully sent to the target.                                                                                                                     |
-| `target_failed`          | Events which failed to reach the target, after 5 retries. Will be retried later.                                                                        |
+| `target_success`         | Events successfully sent to the target.                                                                                                                 |
+| `target_failed`          | Events which failed to reach the target, and will be handled by the retry config. Retries which fail are also counted.                                   |
 | `message_filtered`       | Events filtered out via transformation.                                                                                                                 |
-| `failure_target_success` | Events we could not send to the target, which are not retryable, successfully sent to the failure target.                             |
+| `failure_target_success` | Events we could not send to the target, which are not retryable, successfully sent to the failure target.                                               |
 | `failure_target_failed`  | Events we could not send to the target, which are not retryable, which we failed to send to the failure target. In this scenario, Snowbridge will crash. |
 | `min_processing_latency` | Min time between entering Snowbridge and write to target/failure target.                                                                                |
 | `max_processing_latency` | Max time between entering Snowbridge and write to target/failure target.                                                                                |

docs/destinations/forwarding-events/snowbridge/configuration/retries/index.md

@@ -0,0 +1,26 @@
+---
+title: "Retry behavior (beta)"
+description: "Configure retry behaviour."
+---
+
+:::note
+This feature was added in version 3.0.0
+
+This feature is in beta status because we may make breaking changes in future versions.
+:::
+
+This feature allows you to configure the retry behaviour when the target encounters a failure in sending the data. There are two types of failure you can define:
+
+A **transient failure** is a failure which we expect to succeed again on retry. For example, some temporary network error, or when we encounter throttling. Typically, you would configure a short backoff for this type of failure. When we encounter a transient failure, we keep processing the rest of the data as normal, under the expectation that everything is operating as normal. The failed data is retried after a backoff.
+
+A **setup failure** is one we don't expect to be immediately resolved, for example an incorrect address, or an invalid API Key. Typically, you would configure a long backoff for this type of failure, under the assumption that the issue needs to be fixed with either a configuration change or a change to the target itself (e.g. permissions need to be granted). Setup errors will be retried 5 times before the app crashes.
+
+As of version 3.0.0, only the http target can be configured to return setup errors, via the response rules feature - see [the http target configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). For all other targets, all errors returned will be considered transient, and behaviour can be configured using the `transient` block of the retry configuration.
+
+Retries will be attempted with an exponential backoff. In other words, on each subsequent failure, the backoff time will double. You can configure transient failures to be retried indefinitely by setting `max_attempts` to 0.
+
+## Configuration options
+
+```hcl reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/retry-example.hcl
+```

docs/destinations/forwarding-events/snowbridge/configuration/sources/stdin.md

@@ -7,7 +7,7 @@ Stdin source is the default, and has one optional configuration to set the concu
 
 Stdin source simply treats stdin as the input.
 
-## Configuration 
+## Configuration options
 
 Here is an example of the minimum required configuration:
 

docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md

@@ -3,6 +3,10 @@ title: "HTTP Target"
 description: "Send data over HTTP."
 ---
 
+:::note 
+Version 3.0.0 makes breaking changes to the http target. Details on migrating can be found [in the migration guide](/docs/destinations/forwarding-events/snowbridge/3-X-X-upgrade-guide/index.md)
+:::
+
 ## Basic Authentication
 
 Where basicauth is used, it may be configured using the `basic_auth_username` and `basic_auth_password` options. Where an authorisation header is used, it may be set via the `headers` option.
@@ -17,6 +21,64 @@ Snowbridge supports sending authorized requests to OAuth2 - compliant HTTP targe
 
 Like in the case of basic authentication, we recommend using environment variables for sensitive values.
 
+## Dynamic Headers
+
+:::note
+This feature was added in version 2.3.0
+:::
+
+When enabled, this feature attaches a header to the data according to what your transformation provides in the `HTTPHeaders` field of `engineProtocol`. Data is batched independently per each dynamic header value before requests are sent.
+
+## Request templating
+
+:::note
+This feature was added in version 3.0.0
+:::
+
+This feature allows you to provide a [Golang text template](https://pkg.go.dev/text/template) to construct a request body from a batch of data. This feature should be useful in constructing requests to send to an API, for example.
+
+Input data must be valid JSON, any message that fails to be marshaled to JSON will be treated as invalid and sent to the failure target. Equally, if an attempt to template a batch of data results in an error, then all messages in the batch will be considered invalid and sent to the failure target.
+
+Where the dynamic headers feature is enabled, data is split into batches according to the provided header value, and the templater will operate on each batch separately.
+
+### Helper functions
+
+In addition to all base functions available in the Go text/template package, the following custom functions are available for convenience:
+
+`prettyPrint` - Because the input to the templater is a Go data structure, simply providing a reference to an object field won't output it as JSON. `prettyPrint` converts the data to prettified JSON. Use it wherever you expect a JSON object in the output. This is compatible with any data type, but it shouldn't be necessary if the data is not an object.
+
+`env` - Allows you to set and refer to an env var in your template. Use it when your request body must contain sensitive data, for example an API key.
+
+### Template example
+
+The following example provides an API key via environment variable, and iterates the batch to provide JSON-formatted data one by one into a new key, inserting a comma before all but the first event.
+
+```hcl reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/targets/http-template-full-example.file
+```
+
+## Response rules (beta)
+
+:::note
+This feature was added in version 3.0.0
+
+This feature is in beta status because we may make breaking changes in future versions.
+:::
+
+Response rules allow you to configure how the app deals with failures in sending the data. You can configure a response code and an optional string match on the response body to determine how a failure response is handled. Response codes between 200 and 299 are considered successful, and are not handled by this feature.
+
+There are three categories of failure:
+
+`invalid` means that the data is considered incompatible with the target for some reason. For example, you may have defined a mapping for a given API, but the event happens to have null data for a required field. In this instance, retrying the data won't fix the issue, so you would configure an invalid response rule, identifying which responses indicate this scenario.
+
+Data that matches an invalid response rule is sent to the failure target.
+
+`setup` means that this error is not retryable, but is something which can only be resolved by a change in configuration or a change to the target. An example of this is an authentication failure - retrying will fix the issue, the resolution is to grant the appropriate permissions, or provide the correct API key.
+
+Data that matches a setup response rule is handled by a retry as determined in the `setup` configuration block of [retry configuration](/docs/destinations/forwarding-events/snowbridge/configuration/retries/index.md).
+
+`transient` errors are everything else - we assume that the issue is temporary and retrying will resolve the problem. An example of this is being throttled by an API because too much data is being sent at once. There is no explicit configuration for transient - rather, anything that is not configured as one of the other types is considered transient.
+
 ## Configuration options
 
 Here is an example of the minimum required configuration:

docs/destinations/forwarding-events/snowbridge/configuration/targets/stdout.md

@@ -4,7 +4,8 @@ description: "Write data to an SQS queue."
 ---
 
 Stdout target doesn't have any configurable options - when configured it simply outputs the messages to stdout.
-## Configuration 
+
+## Configuration options
 
 Here is an example of the configuration:
 

docs/destinations/forwarding-events/snowbridge/configuration/telemetry/index.md

@@ -8,7 +8,7 @@ sidebar_position: 500
 
 You can read about our telemetry principles [here](/docs/get-started/snowplow-community-edition/telemetry/index.md).
 
-## Configuration via file:
+## Configuration options
 
 Enabling telemetry:
 

docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/base64Decode.md

@@ -8,7 +8,7 @@ This transformation base64 decodes the message's data from a base64 byte array,
 
 `base64Decode` has no options.
 
-Example:
+## Configuration options
 
 ```hcl reference
 https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/builtin/base64Decode-minimal-example.hcl

docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/base64Encode.md

@@ -8,7 +8,7 @@ This transformation base64 encodes the message's data to a base 64 byte array.
 
 `base64Encode` has no options.
 
-Example:
+## Configuration options
 
 ```hcl reference
 https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/builtin/base64Encode-minimal-example.hcl

docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/jq.md

@@ -0,0 +1,41 @@
+# jq
+
+:::note
+This transformation was added in version 3.0.0
+:::
+
+```mdx-code-block
+import JQDescriptionSharedBlock from "./reusable/_jqDescription.md"
+
+<JQDescriptionSharedBlock/>
+```
+
+`jq` runs a jq command on the message data, and outputs the result of the command. While jq supports multi-element results, commands must output only a single element - this single element can be an array data type.
+
+If the provided jq command results in an error, the message will be considred invalid, and will be sent to the failure target.
+
+The minimal example here returns the input data as a single element array, and the full example maps the data to a new data structure.
+
+The jq transformation will remove any keys with null values from the data.
+
+## Configuration options
+
+Minimal configuration:
+
+```hcl reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/builtin/jq-minimal-example.hcl
+```
+
+Every configuration option:
+
+```hcl reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/builtin/jq-full-example.hcl
+```
+
+## Helper functions
+
+```mdx-code-block
+import JQHelpersSharedBlock from "./reusable/_jqHelpers.md"
+
+<JQHelpersSharedBlock/>
+```

docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/jqFilter.md

@@ -0,0 +1,39 @@
+# jqFilter
+
+:::note
+This transformation was added in version 3.0.0
+:::
+
+```mdx-code-block
+import JQDescriptionSharedBlock from "./reusable/_jqDescription.md"
+
+<JQDescriptionSharedBlock/>
+```
+
+`jqFilter` filters messages based on the output of a jq command which is run against the data. The provided command must return a boolean result. `false` filters the message out, `true` keeps it.
+
+If the provided jq command returns a non-boolean value error, or results in an error, then the message will be considered invalid, and will be sent to the failure target.
+
+This example filters out all data that doesn't have an `app_id` key.
+
+## Configuration options
+
+Minimal configuration:
+
+```hcl reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/builtin/jqFilter-minimal-example.hcl
+```
+
+Every configuration option:
+
+```hcl reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/builtin/jqFilter-full-example.hcl
+```
+
+## Helper Functions
+
+```mdx-code-block
+import JQHelpersSharedBlock from "./reusable/_jqHelpers.md"
+
+<JQHelpersSharedBlock/>
+```

docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/reusable/_jqDescription.md

@@ -0,0 +1,5 @@
+[jq](https://github.com/jqlang/jq) is a lightweight and flexible command-line JSON processor. Snowbridge's jq features utilise the [gojq](https://github.com/itchyny/gojq) package, which is a pure Go implementation of jq. jq is Turing complete, so these features allow you to configure arbitrary logic dealing with JSON data structures. 
+
+jq supports formatting values, mathematical operations, boolean comparisons, regex matches, and many more useful features. To get started with jq command, see the [tutorial](https://jqlang.github.io/jq/tutorial/), and [full reference manual](https://jqlang.github.io/jq/manual/). [This open-source jq playground tool](https://jqplay.org/) may also be helpful.
+
+For most use cases, you are unlikely to encounter them, but note that there are [some small differences](https://github.com/itchyny/gojq?tab=readme-ov-file#difference-to-jq) between jq and gojq.