Skip to content

Commit c9cf383

Browse files
pierredemanpierredeman
committed
Add documentation
1 parent b0ab99a commit c9cf383

File tree

2 files changed

+216
-0
lines changed

2 files changed

+216
-0
lines changed

‎docs/index.asciidoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ The AWS Integration Plugin provides integrated plugins for working with Amazon W
2929
* {logstash-ref}/plugins-inputs-s3.html[S3 Input Plugin]
3030
* {logstash-ref}/plugins-inputs-sqs.html[Sqs Input Plugin]
3131
* {logstash-ref}/plugins-outputs-cloudwatch.html[Cloudwatch Output Plugin]
32+
* {logstash-ref}/plugins-outputs-firehose.html[Firehose Output Plugin]
3233
* {logstash-ref}/plugins-outputs-s3.html[S3 Output Plugin]
3334
* {logstash-ref}/plugins-outputs-sns.html[Sns Output Plugin]
3435
* {logstash-ref}/plugins-outputs-sqs.html[Sqs Output Plugin]

‎docs/output-firehose.asciidoc

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
:integration: aws
2+
:plugin: firehose
3+
:type: output
4+
:default_codec: json
5+
6+
///////////////////////////////////////////
7+
START - GENERATED VARIABLES, DO NOT EDIT!
8+
///////////////////////////////////////////
9+
:version: %VERSION%
10+
:release_date: %RELEASE_DATE%
11+
:changelog_url: %CHANGELOG_URL%
12+
:include_path: ../../../../logstash/docs/include
13+
///////////////////////////////////////////
14+
END - GENERATED VARIABLES, DO NOT EDIT!
15+
///////////////////////////////////////////
16+
17+
[id="plugins-{type}s-{plugin}"]
18+
=== Firehose output plugin
19+
20+
include::{include_path}/plugin_header-integration.asciidoc[]
21+
22+
==== Description
23+
24+
Push events to an Amazon Web Services (AWS) Data Firehose.
25+
26+
Amazon Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon services or HTTP endpoints owned by supported third-party service providers.
27+
See : https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html
28+
29+
This plugin use the AWS SDK to send data to the Firehose stream.
30+
See https://docs.aws.amazon.com/firehose/latest/dev/basic-write.html#writing-with-sdk
31+
32+
Your identity must have the following permissions on the stream:
33+
* `firehose:PutRecordBatch`
34+
35+
==== Batch Publishing
36+
37+
This output publishes messages to Firehose in batches in order to optimize event throughput and increase performance.
38+
This is done using the `PutRecordBatch` API.
39+
See https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecordBatch.html
40+
41+
When publishing messages to Firehose in batches, the following service limits must be respected :
42+
* Each PutRecordBatch request supports up to 500 records.
43+
* Each record in the request can be as large as 1,000 KB.
44+
* All records in the request can be as large as 4 MB.
45+
46+
This plugin will dynamically adjust the size of the batch published to Firehose in order to ensure that the total payload size does not exceed the limits.
47+
48+
[id="plugins-{type}s-{plugin}-options"]
49+
50+
==== Firehose Output Configuration Options
51+
52+
This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
53+
54+
[cols="<,<,<",options="header",]
55+
|=======================================================================
56+
|Setting |Input type|Required
57+
| <<plugins-{type}s-{plugin}-access_key_id>> |<<string,string>>|No
58+
| <<plugins-{type}s-{plugin}-aws_credentials_file>> |<<string,string>>|No
59+
| <<plugins-{type}s-{plugin}-batch_max_count>> |<<number,number>>|No
60+
| <<plugins-{type}s-{plugin}-delivery_stream_name>> |<<string,string>>|No
61+
| <<plugins-{type}s-{plugin}-endpoint>> |<<string,string>>|No
62+
| <<plugins-{type}s-{plugin}-proxy_uri>> |<<string,string>>|No
63+
| <<plugins-{type}s-{plugin}-region>> |<<string,string>>|No
64+
| <<plugins-{type}s-{plugin}-record_max_size_bytes>> |<<bytes,bytes>>|No
65+
| <<plugins-{type}s-{plugin}-record_total_max_size_bytes>> |<<bytes,bytes>>|No
66+
| <<plugins-{type}s-{plugin}-role_arn>> |<<string,string>>|No
67+
| <<plugins-{type}s-{plugin}-role_session_name>> |<<string,string>>|No
68+
| <<plugins-{type}s-{plugin}-secret_access_key>> |<<string,string>>|No
69+
| <<plugins-{type}s-{plugin}-session_token>> |<<string,string>>|No
70+
| <<plugins-{type}s-{plugin}-use_aws_bundled_ca>> |<<boolean,boolean>>|No
71+
|=======================================================================
72+
73+
Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all output plugins.
74+
75+
&nbsp;
76+
77+
[id="plugins-{type}s-{plugin}-access_key_id"]
78+
===== `access_key_id`
79+
80+
* Value type is <<string,string>>
81+
* There is no default value for this setting.
82+
83+
This plugin uses the AWS SDK and supports several ways to get credentials, which will be tried in this order:
84+
85+
1. Static configuration, using `access_key_id` and `secret_access_key` params in logstash plugin config
86+
2. External credentials file specified by `aws_credentials_file`
87+
3. Environment variables `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
88+
4. Environment variables `AMAZON_ACCESS_KEY_ID` and `AMAZON_SECRET_ACCESS_KEY`
89+
5. IAM Instance Profile (available when running inside EC2)
90+
91+
[id="plugins-{type}s-{plugin}-aws_credentials_file"]
92+
===== `aws_credentials_file`
93+
94+
* Value type is <<string,string>>
95+
* There is no default value for this setting.
96+
97+
Path to YAML file containing a hash of AWS credentials.
98+
This file will only be loaded if `access_key_id` and
99+
`secret_access_key` aren't set.
100+
The contents of the file should look like this:
101+
102+
[source,ruby]
103+
----------------------------------
104+
:access_key_id: "12345"
105+
:secret_access_key: "54321"
106+
----------------------------------
107+
108+
[id="plugins-{type}s-{plugin}-batch_max_count"]
109+
===== `batch_max_count`
110+
111+
* Value type is <<number,number>>
112+
* Default value is `500`
113+
114+
The maximum number of records to be sent in each batch.
115+
116+
[id="plugins-{type}s-{plugin}-delivery_stream_name"]
117+
===== `delivery_stream_name`
118+
119+
* Value type is <<string,string>>
120+
121+
The name of the delivery stream.
122+
Note that this is just the name of the stream, not the URL or ARN.
123+
124+
[id="plugins-{type}s-{plugin}-endpoint"]
125+
===== `endpoint`
126+
127+
* Value type is <<string,string>>
128+
* There is no default value for this setting.
129+
130+
The endpoint to connect to.
131+
By default it is constructed using the value of `region`.
132+
This is useful when connecting to S3 compatible services, but beware that these aren't guaranteed to work correctly with the AWS SDK.
133+
134+
[id="plugins-{type}s-{plugin}-record_max_size_bytes"]
135+
===== `record_max_size_bytes`
136+
137+
* Value type is <<bytes,bytes>>
138+
* There is 1_000_000.
139+
140+
The maximum number of bytes for any record sent to Firehose.
141+
Messages exceeding this size will be dropped.
142+
See https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecordBatch.html
143+
144+
[id="plugins-{type}s-{plugin}-record_total_max_size_bytes"]
145+
===== `record_total_max_size_bytes`
146+
147+
* Value type is <<bytes,bytes>>
148+
* There is 4_000_000.
149+
150+
The maximum number of bytes for all records sent to Firehose.
151+
See https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecordBatch.html
152+
153+
[id="plugins-{type}s-{plugin}-proxy_uri"]
154+
===== `proxy_uri`
155+
156+
* Value type is <<string,string>>
157+
* There is no default value for this setting.
158+
159+
URI to proxy server if required
160+
161+
[id="plugins-{type}s-{plugin}-region"]
162+
===== `region`
163+
164+
* Value type is <<string,string>>
165+
* Default value is `"us-east-1"`
166+
167+
The AWS Region
168+
169+
[id="plugins-{type}s-{plugin}-role_arn"]
170+
===== `role_arn`
171+
172+
* Value type is <<string,string>>
173+
* There is no default value for this setting.
174+
175+
The AWS IAM Role to assume, if any.
176+
This is used to generate temporary credentials, typically for cross-account access.
177+
See the https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html[AssumeRole API documentation] for more information.
178+
179+
[id="plugins-{type}s-{plugin}-role_session_name"]
180+
===== `role_session_name`
181+
182+
* Value type is <<string,string>>
183+
* Default value is `"logstash"`
184+
185+
Session name to use when assuming an IAM role.
186+
187+
[id="plugins-{type}s-{plugin}-secret_access_key"]
188+
===== `secret_access_key`
189+
190+
* Value type is <<string,string>>
191+
* There is no default value for this setting.
192+
193+
The AWS Secret Access Key
194+
195+
[id="plugins-{type}s-{plugin}-session_token"]
196+
===== `session_token`
197+
198+
* Value type is <<string,string>>
199+
* There is no default value for this setting.
200+
201+
The AWS Session token for temporary credential
202+
203+
[id="plugins-{type}s-{plugin}-use_aws_bundled_ca"]
204+
===== `use_aws_bundled_ca`
205+
206+
* Value type is <<boolean,boolean>>
207+
* Default value is `false`
208+
209+
Use bundled CA certificates that ship with AWS SDK to verify SSL peer certificates.
210+
For cases where the default certificates are unavailable, e.g. Windows, you can set this to `true`.
211+
212+
[id="plugins-{type}s-{plugin}-common-options"]
213+
include::{include_path}/{type}.asciidoc[]
214+
215+
:default_codec!:

0 commit comments

Comments
 (0)