Snowbridge v3 (#1068)

Author: colmsnowplow

docs/destinations/forwarding-events/snowbridge/3-X-X-upgrade-guide/index.md

@@ -0,0 +1,42 @@
+---
+title: "3.X.X upgrade guide"
+sidebar_position: 400
+---
+
+## Breaking Changes
+
+The below breaking changes were made in version 3.0.0. All other functionality is backwards compatible.
+
+### Lua support removed
+
+Support for Lua transformations has been removed. If you are running a Lua transformation, you can port the logic to [Javascript](/docs/destinations/forwarding-events/snowbridge/configuration/transformations/custom-scripts/javascript-configuration/index.md) or [JQ](/docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/jq.md).
+
+## HTTP target: non-JSON data no longer supported
+
+We never intended to support non-JSON data, but prior to version 3.0.0, the request body was simply populated with whatever bytes were found in the message data, regardless of whether it is valid JSON.
+
+From version 3.0.0 onwards, only valid JSON will work, otherwise the message will be considered invalid and sent to the failure target.
+
+## HTTP target: request batching
+
+Many HTTP APIs allow sending several events in a single request by putting them into a JSON array. Since version 3.0.0, if the Snowbridge source provides data in batches, the HTTP target will batch events in this way.
+
+As a consequence, even when the source provides events in a single event batch, it will now be placed into an array of one element. For example, prior to version 3.0.0, a request body might look like this:
+
+```
+{"foo": "bar"}
+```
+
+But it will now look like this:
+
+```
+[{"foo": "bar"}]
+```
+
+As of version 3.0.0, the SQS source provides events in batches of up to ten, and the Kinesis, Kafka, and Pubsub and Stdin sources provide events in single-event batches. This behavior will likely change in a future version.
+
+You can preserve the previous behavior and ensure that requests are always single-event non-array objects, even with a batching source. To do so, set `request_max_messages` to 1, and provide this template (as long as your data is valid JSON):
+
+```go reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/targets/http-template-unwrap-example.file
+```

docs/destinations/forwarding-events/snowbridge/concepts/failure-model/index.md

@@ -18,11 +18,15 @@ There are several different failures that Snowbridge may hit.
 
 ### Target failure
 
-This is where a request to the destination technology fails or is rejected - for example a http 400 response is received. When Snowbridge hits this failure, it will retry 5 times. If all 5 attempts fail, it will be reported as a 'MsgFailed' for monitoring purposes, and will proceed without acking the failed Messages. As long as the source's acking model allows for it, these will be re-processed through Snowbridge again.
+This is where a request to the destination technology fails or is rejected - for example a HTTP 400 response is received.
 
-Note that this means failures on the receiving end (eg. if an endpoint is unavailable), mean Snowbridge will continue to attempt to process the data until the issue is fixed.
+Retry behavior for target failures is determined by the retry configuration. You can find details of this in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/retries/index.md).
 
-As of Snowbridge 2.4.2, the kinesis target does not treat kinesis write throughput exceptions as this type of failure. Rather it has an in-built backoff and retry, which will persist until each event in the batch is either successful, or fails for a different reason.
+As of Snowbridge 2.4.2, the Kinesis target does not treat kinesis write throughput exceptions as this type of failure. Rather it has an in-built backoff and retry, which will persist until each event in the batch is either successful, or fails for a different reason.
+
+Before version 3.0.0, Snowbridge treated every kind of target failure the same -  it would retry 5 times. If all 5 attempts failed, it would proceed without acking the failed Messages. As long as the source's acking model allows for it, these would be re-processed through Snowbridge again.
+
+Each target failure attempt will be reported as a 'MsgFailed' for monitoring purposes.
 
 ### Oversized data
 
@@ -34,6 +38,8 @@ Writes of oversized messages to the failure target will be recorded with 'Oversi
 
 In the unlikely event that Snowbridge encounters data which is invalid for the target destination (for example empty data is invalid for pubsub), it will create a [generic error failed event](/docs/fundamentals/failed-events/index.md#generic-error),  emit it to the failure target, and ack the original message.
 
+As of version 3.0.0, the HTTP target may produce 'invalid' type failures. This occurs when: the a POST request body cannot be formed; the templating feature's attempts to template data result in an error; or the response conforms to a response rules configuration which specifies that the failure is to be treated as invalid. You can find more details in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md).
+
 Transformation failures are also treated as invalid, as described below.
 
 Writes of invalid messages to the failure target will be recorded with 'InvalidMsg' statistics in monitoring. Any failure to write to the failure target will cause a [fatal failure](#fatal-failure).
@@ -54,6 +60,6 @@ Firstly, if it hits an error in retrieving data from the source stream, it will
 
 Secondly, as described above, where there are failures it will attempt to reprocess the data if it can, and where failures aren't recoverable it will attempt to handle that via a failure target. Normally, even reaching this point is rare.
 
-In the very unlikely event that Snowbridge reaches this point and cannot write to a failure target, the app will crash. Should this happen, and the app is re-deployed, it will begin processing data from the last acked message. Note that the likely impact of this is duplicated sends to the target, but not data loss. 
+In the very unlikely event that Snowbridge reaches this point and cannot write to a failure target, the app will crash. Should this happen, and the app is re-deployed, it will begin processing data from the last acked message. Note that the likely impact of this is duplicated sends to the target, but not data loss.
 
-Of course, if you experience crashes or other issues that are not explained by the above, please log an issue detailing the bahaviour.
+Of course, if you experience crashes or other issues that are not explained by the above, please log an issue detailing the behavior.

docs/destinations/forwarding-events/snowbridge/configuration/monitoring/index.md

@@ -23,7 +23,8 @@ https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/mon
 ```hcl reference
 https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/monitoring/sentry-example.hcl
 ```
-### StatsD stats receiver 
+
+### StatsD stats receiver configuration
 
 ```hcl reference
 https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/monitoring/statsd-example.hcl
@@ -33,10 +34,10 @@ Snowbridge sends the following metrics to statsd:
 
 | Metric                   | Definitions                                                                                                                                             |
 |--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `target_success`         | Events successfully sent to the target.                                                                                                                     |
-| `target_failed`          | Events which failed to reach the target, after 5 retries. Will be retried later.                                                                        |
+| `target_success`         | Events successfully sent to the target.                                                                                                                 |
+| `target_failed`          | Events which failed to reach the target, and will be handled by the retry config. Retries which fail are also counted.                                   |
 | `message_filtered`       | Events filtered out via transformation.                                                                                                                 |
-| `failure_target_success` | Events we could not send to the target, which are not retryable, successfully sent to the failure target.                             |
+| `failure_target_success` | Events we could not send to the target, which are not retryable, successfully sent to the failure target.                                               |
 | `failure_target_failed`  | Events we could not send to the target, which are not retryable, which we failed to send to the failure target. In this scenario, Snowbridge will crash. |
 | `min_processing_latency` | Min time between entering Snowbridge and write to target/failure target.                                                                                |
 | `max_processing_latency` | Max time between entering Snowbridge and write to target/failure target.                                                                                |

docs/destinations/forwarding-events/snowbridge/configuration/retries/index.md

@@ -0,0 +1,26 @@
+---
+title: "Retry behavior (beta)"
+description: "Configure retry behaviour."
+---
+
+:::note
+This feature was added in version 3.0.0
+
+This feature is in beta status because we may make breaking changes in future versions.
+:::
+
+This feature allows you to configure the retry behavior when the target encounters a failure in sending the data. There are two types of failure you can define:
+
+A **transient failure** is a failure which we expect to succeed again on retry. For example, some temporary network error, or when we encounter throttling. Typically, you would configure a short backoff for this type of failure. When we encounter a transient failure, we keep processing the rest of the data as normal, under the expectation that everything is operating as normal. The failed data is retried after a backoff.
+
+A **setup failure** is one we don't expect to be immediately resolved, for example an incorrect address, or an invalid API Key. Typically, you would configure a long backoff for this type of failure, under the assumption that the issue needs to be fixed with either a configuration change or a change to the target itself (e.g. permissions need to be granted). Setup errors will be retried 5 times before the app crashes.
+
+As of version 3.0.0, only the http target can be configured to return setup errors, via the response rules feature - see [the http target configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). For all other targets, all errors returned will be considered transient, and behavior can be configured using the `transient` block of the retry configuration.
+
+Retries will be attempted with an exponential backoff. In other words, on each subsequent failure, the backoff time will double. You can configure transient failures to be retried indefinitely by setting `max_attempts` to 0.
+
+## Configuration options
+
+```hcl reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/retry-example.hcl
+```

docs/destinations/forwarding-events/snowbridge/configuration/sources/kafka.md

@@ -1,5 +1,5 @@
 ---
-title: "Kafka Source"
+title: "Kafka"
 description: "Read data from a Kafka topic."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/sources/kinesis.md

@@ -1,5 +1,5 @@
 ---
-title: "Kinesis Source"
+title: "Kinesis"
 description: "Read data from a Kinesis stream."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/sources/pubsub.md

@@ -1,5 +1,5 @@
 ---
-title: "PubSub Source"
+title: "PubSub"
 description: "Read data from a PubSub topic."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/sources/sqs.md

@@ -1,5 +1,5 @@
 ---
-title: "SQS Source"
+title: "SQS"
 description: "Read data from an SQS queue."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/sources/stdin.md

@@ -1,13 +1,14 @@
 ---
-title: "Stdin Source"
+title: "Stdin"
 description: "Read data from stdin."
+sidebar_position: 1
 ---
 
 Stdin source is the default, and has one optional configuration to set the concurrency. 
 
 Stdin source simply treats stdin as the input.
 
-## Configuration 
+## Configuration options
 
 Here is an example of the minimum required configuration:
 

docs/destinations/forwarding-events/snowbridge/configuration/targets/eventhub.md

@@ -1,5 +1,5 @@
 ---
-title: "EventHub Target"
+title: "EventHub"
 description: "Write data to an EventHub."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md

@@ -1,22 +1,84 @@
 ---
-title: "HTTP Target"
+title: "HTTP"
 description: "Send data over HTTP."
 ---
 
+:::note
+Version 3.0.0 makes breaking changes to the http target. Details on migrating can be found [in the migration guide](/docs/destinations/forwarding-events/snowbridge/3-X-X-upgrade-guide/index.md)
+:::
+
 ## Basic Authentication
 
-Where basicauth is used, it may be configured using the `basic_auth_username` and `basic_auth_password` options. Where an authorisation header is used, it may be set via the `headers` option.
+Where basicauth is used, it may be configured using the `basic_auth_username` and `basic_auth_password` options. Where an authorization header is used, it may be set via the `headers` option.
 
-we recommend using environment variables for sensitive values - which can be done via HCL's native `env.MY_ENV_VAR` format (as seen below).
+We recommend using environment variables for sensitive values - which can be done via HCL's native `env.MY_ENV_VAR` format (as seen below).
 
 TLS may be configured by providing the `key_file`, `cert_file` and `ca_file` options with paths to the relevant TLS files.
 
 ## OAuth2
 
-Snowbridge supports sending authorized requests to OAuth2 - compliant HTTP targets. This can be enabled in by setting `oauth2_client_id `, `oauth2_client_secret`, `oauth2_refresh_token` (these 3 are long-lived credentials used to generate short-lived bearer access tokens) and `oauth2_token_url`(which is the URL of authorization server providing access tokens).
+Snowbridge supports sending authorized requests to OAuth2-compliant HTTP targets. This can be enabled by setting `oauth2_client_id`, `oauth2_client_secret`, `oauth2_refresh_token` (these three are long-lived credentials used to generate short-lived bearer access tokens), and `oauth2_token_url` (which is the URL of the authorization server providing access tokens).
 
 Like in the case of basic authentication, we recommend using environment variables for sensitive values.
 
+## Dynamic headers
+
+:::note
+This feature was added in version 2.3.0
+:::
+
+When enabled, this feature attaches a header to the data according to what your transformation provides in the `HTTPHeaders` field of `engineProtocol`. Data is batched independently per each dynamic header value before requests are sent.
+
+## Request templating
+
+:::note
+This feature was added in version 3.0.0
+:::
+
+This feature allows you to provide a [Golang text template](https://pkg.go.dev/text/template) to construct a request body from a batch of data. This feature should be useful in constructing requests to send to an API, for example.
+
+Input data must be valid JSON, any message that fails to be marshaled to JSON will be treated as invalid and sent to the failure target. Equally, if an attempt to template a batch of data results in an error, then all messages in the batch will be considered invalid and sent to the failure target.
+
+Where the dynamic headers feature is enabled, data is split into batches according to the provided header value, and the templater will operate on each batch separately.
+
+### Helper functions
+
+In addition to all base functions available in the Go text/template package, the following custom functions are available for convenience:
+
+`prettyPrint` - Because the input to the templater is a Go data structure, simply providing a reference to an object field won't output it as JSON. `prettyPrint` converts the data to prettified JSON. Use it wherever you expect a JSON object in the output. This is compatible with any data type, but it shouldn't be necessary if the data is not an object.
+
+`env` - Allows you to set and refer to an env var in your template. Use it when your request body must contain sensitive data, for example an API key.
+
+### Template example
+
+The following example provides an API key via environment variable, and iterates the batch to provide JSON-formatted data one by one into a new key, inserting a comma before all but the first event.
+
+```hcl reference
+https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/targets/http-template-full-example.file
+```
+
+## Response rules (beta)
+
+:::note
+This feature was added in version 3.0.0
+
+This feature is in beta status because we may make breaking changes in future versions.
+:::
+
+Response rules allow you to configure how the app deals with failures in sending the data. You can configure a response code and an optional string match on the response body to determine how a failure response is handled. Response codes between 200 and 299 are considered successful, and are not handled by this feature.
+
+There are three categories of failure:
+
+`invalid` means that the data is considered incompatible with the target for some reason. For example, you may have defined a mapping for a given API, but the event happens to have null data for a required field. In this instance, retrying the data won't fix the issue, so you would configure an invalid response rule, identifying which responses indicate this scenario.
+
+Data that matches an invalid response rule is sent to the failure target.
+
+`setup` means that this error is not retryable, but is something which can only be resolved by a change in configuration or a change to the target. An example of this is an authentication failure - retrying will fix the issue, the resolution is to grant the appropriate permissions, or provide the correct API key.
+
+Data that matches a setup response rule is handled by a retry as determined in the `setup` configuration block of [retry configuration](/docs/destinations/forwarding-events/snowbridge/configuration/retries/index.md).
+
+`transient` errors are everything else - we assume that the issue is temporary and retrying will resolve the problem. An example of this is being throttled by an API because too much data is being sent at once. There is no explicit configuration for transient - rather, anything that is not configured as one of the other types is considered transient.
+
 ## Configuration options
 
 Here is an example of the minimum required configuration:

docs/destinations/forwarding-events/snowbridge/configuration/targets/kafka.md

@@ -1,5 +1,5 @@
 ---
-title: "Kafka Target"
+title: "Kafka"
 description: "Write data to a Kafka topic."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/targets/kinesis.md

@@ -1,5 +1,5 @@
 ---
-title: "Kinesis Target"
+title: "Kinesis"
 description: "Write data to a Kinesis stream."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/targets/pubsub.md

@@ -1,5 +1,5 @@
 ---
-title: "PubSub Target"
+title: "PubSub"
 description: "Write data to a Pubsub topic."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/targets/sqs.md

@@ -1,5 +1,5 @@
 ---
-title: "SQS Target"
+title: "SQS"
 description: "Write data to an SQS queue."
 ---
 

docs/destinations/forwarding-events/snowbridge/configuration/targets/stdout.md

@@ -1,10 +1,12 @@
 ---
-title: "Stdout Target"
+title: "Stdout"
 description: "Write data to an SQS queue."
+sidebar_position: 1
 ---
 
 Stdout target doesn't have any configurable options - when configured it simply outputs the messages to stdout.
-## Configuration 
+
+## Configuration options
 
 Here is an example of the configuration:
 

docs/destinations/forwarding-events/snowbridge/configuration/telemetry/index.md

@@ -8,7 +8,7 @@ sidebar_position: 500
 
 You can read about our telemetry principles [here](/docs/get-started/snowplow-community-edition/telemetry/index.md).
 
-## Configuration via file:
+## Configuration options
 
 Enabling telemetry:
 

docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/base64Decode.md

@@ -8,7 +8,7 @@ This transformation base64 decodes the message's data from a base64 byte array,
 
 `base64Decode` has no options.
 
-Example:
+## Configuration options
 
 ```hcl reference
 https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/builtin/base64Decode-minimal-example.hcl

docs/destinations/forwarding-events/snowbridge/configuration/transformations/builtin/base64Encode.md

@@ -8,7 +8,7 @@ This transformation base64 encodes the message's data to a base 64 byte array.
 
 `base64Encode` has no options.
 
-Example:
+## Configuration options
 
 ```hcl reference
 https://github.com/snowplow/snowbridge/blob/master/assets/docs/configuration/transformations/builtin/base64Encode-minimal-example.hcl