You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The process of setting up a data stream in {{stack}} and {{serverless-full}} is similar, making use of their respective APIs. However, because {{serverless-short}} provides a built-in [data stream lifecycle](/manage-data/lifecycle/data-stream.md) mechanism and retention settings, you don't need to configure index lifecycle management ({{ilm-init}}) options as you do in an {{stack}} deployment.
14
+
13
15
To set up a data stream, follow these steps:
14
16
15
-
1.[Create an index lifecycle policy](#create-index-lifecycle-policy)
17
+
1.[Create an index lifecycle policy](#create-index-lifecycle-policy) {applies_to}`serverless: unavailable`
3.[Create an index template](#create-index-template)
18
20
4.[Create the data stream](#create-data-stream)
19
21
5.[Secure the data stream](#secure-data-stream)
20
22
21
23
You can also [convert an index alias to a data stream](#convert-index-alias-to-data-stream).
22
24
23
-
::::{important}
25
+
:::{important}
24
26
If you use {{fleet}}, {{agent}}, or {{ls}}, skip this tutorial. They all set up data streams for you.
25
27
26
28
For {{fleet}} and {{agent}}, refer to [](/reference/fleet/data-streams.md). For {{ls}}, refer to the [data streams settings](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-data_stream) for the `elasticsearch output` plugin.
27
29
28
-
::::
29
-
30
+
:::
30
31
31
32
32
33
## Create an index lifecycle policy [create-index-lifecycle-policy]
34
+
```{applies_to}
35
+
serverless: unavailable
36
+
```
33
37
34
-
While optional, we recommend using {{ilm-init}} to automate the management of your data stream’s backing indices. {{ilm-init}} requires an index lifecycle policy.
38
+
While optional, we recommend using the {{ilm}} ({{ilm-init}}) capability in {{stack}} deployments to automate the management of your data stream’s backing indices. {{ilm-init}} requires an index lifecycle policy.
39
+
40
+
:::{admonition} Simpler lifecycle management in {{serverless-short}} projects
41
+
{{ilm-init}} lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {{ilm-init}} is not available in {{serverless-short}}, where performance optimizations are automatic. Instead, [data stream lifecycle](/manage-data/lifecycle/data-stream.md) is available as a data management option.
42
+
:::
35
43
36
44
To create an index lifecycle policy in {{kib}}:
37
45
@@ -100,22 +108,32 @@ When creating your component templates, include:
100
108
* A [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) mapping for the `@timestamp` field. If you don’t specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options.
101
109
* Your lifecycle policy in the `index.lifecycle.name` index setting.
102
110
103
-
::::{tip}
111
+
:::{tip}
104
112
Use the [Elastic Common Schema (ECS)](ecs://reference/index.md) when mapping your fields. ECS fields integrate with several {{stack}} features by default.
105
113
106
114
If you’re unsure how to map your fields, use [runtime fields](../mapping/define-runtime-fields-in-search-request.md) to extract fields from [unstructured content](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#mapping-unstructured-content) at search time. For example, you can index a log message to a `wildcard` field and later extract IP addresses and other data from this field during a search.
115
+
:::
107
116
108
-
::::
109
-
117
+
::::{tab-set}
118
+
:group: set-up-ds
119
+
:::{tab-item} {{kib}}
120
+
:sync: kibana
110
121
To create a component template in {{kib}}:
111
122
112
123
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
113
124
1. In the **Index Templates** tab, click **Create component template**.
125
+
:::
114
126
115
-
You can also use the [create component template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template).
127
+
:::{tab-item} API
128
+
:sync: api
129
+
Use an API to create a component template:
130
+
131
+
* In an {{stack}} deployment, use the [create component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) API.
132
+
* In {{serverless-full}}, use the [create component template](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-cluster-put-component-template) API.
133
+
134
+
To create a component template for mappings, use this request:
116
135
117
136
```console
118
-
# Creates a component template for mappings
119
137
PUT _component_template/my-mappings
120
138
{
121
139
"template": {
@@ -136,8 +154,11 @@ PUT _component_template/my-mappings
136
154
"my-custom-meta-field": "More arbitrary metadata"
137
155
}
138
156
}
157
+
```
158
+
159
+
To create a component template for index settings, use this request:
139
160
140
-
# Creates a component template for index settings
161
+
```console
141
162
PUT _component_template/my-settings
142
163
{
143
164
"template": {
@@ -152,6 +173,8 @@ PUT _component_template/my-settings
152
173
}
153
174
```
154
175
176
+
:::
177
+
::::
155
178
156
179
## Create an index template [create-index-template]
157
180
@@ -162,12 +185,25 @@ Use your component templates to create an index template. Specify:
162
185
* Any component templates that contain your mappings and index settings.
163
186
* A priority higher than `200` to avoid collisions with built-in templates. See [Avoid index pattern collisions](../templates.md#avoid-index-pattern-collisions).
164
187
188
+
::::{tab-set}
189
+
:group: set-up-ds
190
+
:::{tab-item} {{kib}}
191
+
:sync: kibana
165
192
To create an index template in {{kib}}:
166
193
167
194
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
168
195
1. In the **Index Templates** tab, click **Create template**.
169
196
170
-
You can also use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). Include the `data_stream` object to enable data streams.
197
+
:::
198
+
199
+
:::{tab-item} API
200
+
:sync: api
201
+
Use an API to create an index template:
202
+
203
+
* In an {{stack}} deployment, use the [create an index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API.
204
+
* In {{serverless-full}}, use the [create an index template](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-put-index-template) API.
205
+
206
+
Include the `data_stream` object to enable data streams:
171
207
172
208
```console
173
209
PUT _index_template/my-index-template
@@ -183,10 +219,12 @@ PUT _index_template/my-index-template
183
219
}
184
220
```
185
221
222
+
:::
223
+
::::
186
224
187
225
## Create the data stream [create-data-stream]
188
226
189
-
[Indexing requests](../data-streams/use-data-stream.md#add-documents-to-a-data-stream)[Indexing requests](./use-data-stream.md#add-documents-to-a-data-stream)add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field.
227
+
[Indexing requests](../data-streams/use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field.
190
228
191
229
To automatically create your data stream, submit an indexing request that targets the stream’s name. This name must match one of your index template’s index patterns.
You can also use an API to manually create the data stream:
206
245
207
-
You can also manually create the stream using the [create data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream). The stream’s name must still match one of your template’s index patterns.
246
+
* In an {{stack}} deployment, use the [create a data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream) API.
247
+
* In {{serverless-full}}, use the [create a data stream](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-create-data-stream) API.
208
248
209
249
```console
210
250
PUT _data_stream/my-data-stream
@@ -216,14 +256,19 @@ After it's been created, you can view and manage this and other data streams fro
216
256
217
257
Use [index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices) to control access to a data stream. Granting privileges on a data stream grants the same privileges on its backing indices.
218
258
219
-
For an example, see[Data stream privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
259
+
For an example, refer to[Data stream privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
220
260
221
261
222
262
## Convert an index alias to a data stream [convert-index-alias-to-data-stream]
223
263
224
-
Prior to {{es}} 7.9, you’d typically use an [index alias with a write index](../../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](../../lifecycle/data-tiers.md).
264
+
Prior to {{es}} 7.9, you’d typically use an index alias with a write index to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](../../lifecycle/data-tiers.md).
265
+
266
+
You can convert an index alias with a write index to a data stream with the same name, using an API:
225
267
226
-
To convert an index alias with a write index to a data stream with the same name, use the [migrate to data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-migrate-to-data-stream). During conversion, the alias’s indices become hidden backing indices for the stream. The alias’s write index becomes the stream’s write index. The stream still requires a matching index template with data stream enabled.
268
+
* In an {{stack}} deployment, use the [convert an index alias to a data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-migrate-to-data-stream) API.
269
+
* In {{serverless-full}}, use the [convert an index alias to a data stream](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-migrate-to-data-stream) API.
270
+
271
+
During conversion, the alias's indices become hidden backing indices for the stream. The alias's write index becomes the stream's write index. The stream still requires a matching index template with data stream enabled.
227
272
228
273
```console
229
274
POST _data_stream/_migrate/my-time-series-data
@@ -232,28 +277,64 @@ POST _data_stream/_migrate/my-time-series-data
232
277
233
278
## Get information about a data stream [get-info-about-data-stream]
234
279
280
+
You can review metadata about each data stream using the {{kib}} UI (visual overview) or the API (raw JSON).
281
+
282
+
::::{tab-set}
283
+
:group: set-up-ds
284
+
:::{tab-item} {{kib}}
285
+
:sync: kibana
235
286
To get information about a data stream in {{kib}}:
236
287
237
288
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
238
289
1. In the **Data Streams** tab, click the data stream’s name.
239
290
240
-
You can also use the [get data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream).
Starting with {{es}} version 9.2, you can use the [**Streams**](/solutions/observability/streams/streams.md) page to view the details of a data stream. The **Streams** page provides a centralized interface for managing your data in {{kib}}.Select a stream to view its details.
294
+
:::
295
+
296
+
:::
297
+
298
+
:::{tab-item} API
299
+
:sync: api
300
+
You can also use an API to get this information:
301
+
302
+
* In an {{stack}} deployment, use the [get data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream) API.
303
+
* In {{serverless-full}}, use the [get data streams](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-get-data-stream) API.
241
304
242
305
```console
243
306
GET _data_stream/my-data-stream
244
307
```
245
308
309
+
:::
310
+
::::
246
311
247
312
## Delete a data stream [delete-data-stream]
248
313
314
+
You can delete a data stream and its backing indices via the {{kib}} UI or an API. To complete this action, you need the `delete_index`[security privilege](elasticsearch://reference/elasticsearch/security-privileges.md) for the data stream.
315
+
316
+
::::{tab-set}
317
+
:group: set-up-ds
318
+
:::{tab-item} {{kib}}
319
+
:sync: kibana
320
+
249
321
To delete a data stream and its backing indices in {{kib}}:
250
322
251
323
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
252
-
1. In the **Data Streams** view, click the trash icon. The icon only displays if you have the `delete_index`[security privilege](elasticsearch://reference/elasticsearch/security-privileges.md) for the data stream.
324
+
1. In the **Data Streams** view, click the trash can icon. The icon only displays if you have the `delete_index` security privilege for the data stream.
325
+
326
+
:::
327
+
:::{tab-item} API
328
+
:sync: api
329
+
330
+
You can also use an API to delete a data stream:
253
331
254
-
You can also use the [delete data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-stream).
332
+
* In an {{stack}} deployment, use the [delete data streams](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-stream) API.
333
+
* In {{serverless-full}}, use the [delete data streams](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-delete-data-stream) API.
0 commit comments