input.streamName
Required. Name of the Kinesis stream with the enriched events
input.appName
Optional, default snowplow-bigquery-loader. Name to use for the dynamodb table, used by the underlying Kinesis Consumer Library for managing leases.
input.initialPosition
Optional, default LATEST. Allowed values are LATEST, TRIM_HORIZON, AT_TIMESTAMP. When the loader is deployed for the first time, this controls from where in the kinesis stream it should start consuming events. On all subsequent deployments of the loader, the loader will resume from the offsets stored in the DynamoDB table.
input.initialPosition.timestamp
Required if input.initialPosition is AT_TIMESTAMP. A timestamp in ISO8601 format from where the loader should start consuming events.
input.retrievalMode
Optional, default Polling. Change to FanOut to enable the enhance fan-out feature of Kinesis.
input.retrievalMode.maxRecords
Optional. Default value 1000. How many events the Kinesis client may fetch in a single poll. Only used when `input.retrievalMode` is Polling.
input.workerIdentifier
Optional. Defaults to the HOSTNAME environment variable. The name of this KCL worker used in the dynamodb lease table.
input.leaseDuration
Optional. Default value 10 seconds. The duration of shard leases. KCL workers must periodically refresh leases in the dynamodb table before this duration expires.
input.maxLeasesToStealAtOneTimeFactor
Optional. Default value 2.0. Controls how to pick the max number of shard leases to steal at one time. E.g. If there are 4 available processors, and maxLeasesToStealAtOneTimeFactor = 2.0, then allow the loader to steal up to 8 leases. Allows bigger instances to more quickly acquire the shard-leases they need to combat latency.
input.checkpointThrottledBackoffPolicy.minBackoff
Optional. Default value 100 milliseconds. Initial backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.
input.checkpointThrottledBackoffPolicy.maxBackoff
Optional. Default value 1 second. Maximum backoff used to retry checkpointing if we exceed the DynamoDB provisioned write limits.
output.bad.streamName
Required. Name of the Kinesis stream that will receive failed events.
output.bad.throttledBackoffPolicy.minBackoff
Optional. Default value 100 milliseconds. Initial backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.
output.bad.throttledBackoffPolicy.maxBackoff
Optional. Default value 1 second. Maximum backoff used to retry sending failed events if we exceed the Kinesis write throughput limits.
output.bad.recordLimit
Optional. Default value 500. The maximum number of records we are allowed to send to Kinesis in 1 PutRecords request.
output.bad.byteLimit
Optional. Default value 5242880. The maximum number of bytes we are allowed to send to Kinesis in 1 PutRecords request.
output.bad.maxRecordSize.*
Optional. Default value 1000000. Any single event failed event sent to Kinesis should not exceed this size in bytes