Skip to content

Commit d1fcdb5

Browse files
yetanothertwmarciw
andauthored
Fix serverless applicability in setting up data streams docs (#4083)
Fixes #3384 and updates the page so that it can be used for both Stack and Serverless scopes. ## Summary I've added a bit more context around the step to create an ILM policy included in this tutorial, and why it is not required in a Serverless context. I've also added tabs to the sections that list both Kibana steps and steps using the API. Each section now also links to the corresponding Serverless API doc page. ## Generative AI disclosure <!-- To help us ensure compliance with the Elastic open source and documentation guidelines, please answer the following: --> 1. Did you use a generative AI (GenAI) tool to assist in creating this contribution? - [ ] Yes - [x] No <!-- 2. If you answered "Yes" to the previous question, please specify the tool(s) and model(s) used (e.g., Google Gemini, OpenAI ChatGPT-4, etc.). Tool(s) and model(s) used: --> --------- Co-authored-by: Marci W <[email protected]>
1 parent e0037ca commit d1fcdb5

File tree

1 file changed

+101
-20
lines changed

1 file changed

+101
-20
lines changed

manage-data/data-store/data-streams/set-up-data-stream.md

Lines changed: 101 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -10,28 +10,36 @@ products:
1010

1111
# Set up a data stream [set-up-a-data-stream]
1212

13+
The process of setting up a data stream in {{stack}} and {{serverless-full}} is similar, making use of their respective APIs. However, because {{serverless-short}} provides a built-in [data stream lifecycle](/manage-data/lifecycle/data-stream.md) mechanism and retention settings, you don't need to configure index lifecycle management ({{ilm-init}}) options as you do in an {{stack}} deployment.
14+
1315
To set up a data stream, follow these steps:
1416

15-
1. [Create an index lifecycle policy](#create-index-lifecycle-policy)
17+
1. [Create an index lifecycle policy](#create-index-lifecycle-policy) {applies_to}`serverless: unavailable`
1618
2. [Create component templates](#create-component-templates)
1719
3. [Create an index template](#create-index-template)
1820
4. [Create the data stream](#create-data-stream)
1921
5. [Secure the data stream](#secure-data-stream)
2022

2123
You can also [convert an index alias to a data stream](#convert-index-alias-to-data-stream).
2224

23-
::::{important}
25+
:::{important}
2426
If you use {{fleet}}, {{agent}}, or {{ls}}, skip this tutorial. They all set up data streams for you.
2527

2628
For {{fleet}} and {{agent}}, refer to [](/reference/fleet/data-streams.md). For {{ls}}, refer to the [data streams settings](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-data_stream) for the `elasticsearch output` plugin.
2729

28-
::::
29-
30+
:::
3031

3132

3233
## Create an index lifecycle policy [create-index-lifecycle-policy]
34+
```{applies_to}
35+
serverless: unavailable
36+
```
3337

34-
While optional, we recommend using {{ilm-init}} to automate the management of your data stream’s backing indices. {{ilm-init}} requires an index lifecycle policy.
38+
While optional, we recommend using the {{ilm}} ({{ilm-init}}) capability in {{stack}} deployments to automate the management of your data stream’s backing indices. {{ilm-init}} requires an index lifecycle policy.
39+
40+
:::{admonition} Simpler lifecycle management in {{serverless-short}} projects
41+
{{ilm-init}} lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {{ilm-init}} is not available in {{serverless-short}}, where performance optimizations are automatic. Instead, [data stream lifecycle](/manage-data/lifecycle/data-stream.md) is available as a data management option.
42+
:::
3543

3644
To create an index lifecycle policy in {{kib}}:
3745

@@ -100,22 +108,32 @@ When creating your component templates, include:
100108
* A [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) mapping for the `@timestamp` field. If you don’t specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options.
101109
* Your lifecycle policy in the `index.lifecycle.name` index setting.
102110

103-
::::{tip}
111+
:::{tip}
104112
Use the [Elastic Common Schema (ECS)](ecs://reference/index.md) when mapping your fields. ECS fields integrate with several {{stack}} features by default.
105113

106114
If you’re unsure how to map your fields, use [runtime fields](../mapping/define-runtime-fields-in-search-request.md) to extract fields from [unstructured content](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#mapping-unstructured-content) at search time. For example, you can index a log message to a `wildcard` field and later extract IP addresses and other data from this field during a search.
115+
:::
107116

108-
::::
109-
117+
::::{tab-set}
118+
:group: set-up-ds
119+
:::{tab-item} {{kib}}
120+
:sync: kibana
110121
To create a component template in {{kib}}:
111122

112123
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
113124
1. In the **Index Templates** tab, click **Create component template**.
125+
:::
114126

115-
You can also use the [create component template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template).
127+
:::{tab-item} API
128+
:sync: api
129+
Use an API to create a component template:
130+
131+
* In an {{stack}} deployment, use the [create component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) API.
132+
* In {{serverless-full}}, use the [create component template](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-cluster-put-component-template) API.
133+
134+
To create a component template for mappings, use this request:
116135

117136
```console
118-
# Creates a component template for mappings
119137
PUT _component_template/my-mappings
120138
{
121139
"template": {
@@ -136,8 +154,11 @@ PUT _component_template/my-mappings
136154
"my-custom-meta-field": "More arbitrary metadata"
137155
}
138156
}
157+
```
158+
159+
To create a component template for index settings, use this request:
139160

140-
# Creates a component template for index settings
161+
```console
141162
PUT _component_template/my-settings
142163
{
143164
"template": {
@@ -152,6 +173,8 @@ PUT _component_template/my-settings
152173
}
153174
```
154175

176+
:::
177+
::::
155178

156179
## Create an index template [create-index-template]
157180

@@ -162,12 +185,25 @@ Use your component templates to create an index template. Specify:
162185
* Any component templates that contain your mappings and index settings.
163186
* A priority higher than `200` to avoid collisions with built-in templates. See [Avoid index pattern collisions](../templates.md#avoid-index-pattern-collisions).
164187

188+
::::{tab-set}
189+
:group: set-up-ds
190+
:::{tab-item} {{kib}}
191+
:sync: kibana
165192
To create an index template in {{kib}}:
166193

167194
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
168195
1. In the **Index Templates** tab, click **Create template**.
169196

170-
You can also use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). Include the `data_stream` object to enable data streams.
197+
:::
198+
199+
:::{tab-item} API
200+
:sync: api
201+
Use an API to create an index template:
202+
203+
* In an {{stack}} deployment, use the [create an index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API.
204+
* In {{serverless-full}}, use the [create an index template](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-put-index-template) API.
205+
206+
Include the `data_stream` object to enable data streams:
171207

172208
```console
173209
PUT _index_template/my-index-template
@@ -183,10 +219,12 @@ PUT _index_template/my-index-template
183219
}
184220
```
185221

222+
:::
223+
::::
186224

187225
## Create the data stream [create-data-stream]
188226

189-
[Indexing requests](../data-streams/use-data-stream.md#add-documents-to-a-data-stream) [Indexing requests](./use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field.
227+
[Indexing requests](../data-streams/use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field.
190228

191229
To automatically create your data stream, submit an indexing request that targets the stream’s name. This name must match one of your index template’s index patterns.
192230

@@ -203,8 +241,10 @@ POST my-data-stream/_doc
203241
"message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
204242
}
205243
```
244+
You can also use an API to manually create the data stream:
206245

207-
You can also manually create the stream using the [create data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream). The stream’s name must still match one of your template’s index patterns.
246+
* In an {{stack}} deployment, use the [create a data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream) API.
247+
* In {{serverless-full}}, use the [create a data stream](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-create-data-stream) API.
208248

209249
```console
210250
PUT _data_stream/my-data-stream
@@ -216,14 +256,19 @@ After it's been created, you can view and manage this and other data streams fro
216256

217257
Use [index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices) to control access to a data stream. Granting privileges on a data stream grants the same privileges on its backing indices.
218258

219-
For an example, see [Data stream privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
259+
For an example, refer to [Data stream privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
220260

221261

222262
## Convert an index alias to a data stream [convert-index-alias-to-data-stream]
223263

224-
Prior to {{es}} 7.9, you’d typically use an [index alias with a write index](../../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](../../lifecycle/data-tiers.md).
264+
Prior to {{es}} 7.9, you’d typically use an index alias with a write index to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](../../lifecycle/data-tiers.md).
265+
266+
You can convert an index alias with a write index to a data stream with the same name, using an API:
225267

226-
To convert an index alias with a write index to a data stream with the same name, use the [migrate to data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-migrate-to-data-stream). During conversion, the alias’s indices become hidden backing indices for the stream. The alias’s write index becomes the stream’s write index. The stream still requires a matching index template with data stream enabled.
268+
* In an {{stack}} deployment, use the [convert an index alias to a data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-migrate-to-data-stream) API.
269+
* In {{serverless-full}}, use the [convert an index alias to a data stream](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-migrate-to-data-stream) API.
270+
271+
During conversion, the alias's indices become hidden backing indices for the stream. The alias's write index becomes the stream's write index. The stream still requires a matching index template with data stream enabled.
227272

228273
```console
229274
POST _data_stream/_migrate/my-time-series-data
@@ -232,28 +277,64 @@ POST _data_stream/_migrate/my-time-series-data
232277

233278
## Get information about a data stream [get-info-about-data-stream]
234279

280+
You can review metadata about each data stream using the {{kib}} UI (visual overview) or the API (raw JSON).
281+
282+
::::{tab-set}
283+
:group: set-up-ds
284+
:::{tab-item} {{kib}}
285+
:sync: kibana
235286
To get information about a data stream in {{kib}}:
236287

237288
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
238289
1. In the **Data Streams** tab, click the data stream’s name.
239290

240-
You can also use the [get data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream).
291+
:::{tip}
292+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
293+
Starting with {{es}} version 9.2, you can use the [**Streams**](/solutions/observability/streams/streams.md) page to view the details of a data stream. The **Streams** page provides a centralized interface for managing your data in {{kib}}.Select a stream to view its details.
294+
:::
295+
296+
:::
297+
298+
:::{tab-item} API
299+
:sync: api
300+
You can also use an API to get this information:
301+
302+
* In an {{stack}} deployment, use the [get data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream) API.
303+
* In {{serverless-full}}, use the [get data streams](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-get-data-stream) API.
241304

242305
```console
243306
GET _data_stream/my-data-stream
244307
```
245308

309+
:::
310+
::::
246311

247312
## Delete a data stream [delete-data-stream]
248313

314+
You can delete a data stream and its backing indices via the {{kib}} UI or an API. To complete this action, you need the `delete_index` [security privilege](elasticsearch://reference/elasticsearch/security-privileges.md) for the data stream.
315+
316+
::::{tab-set}
317+
:group: set-up-ds
318+
:::{tab-item} {{kib}}
319+
:sync: kibana
320+
249321
To delete a data stream and its backing indices in {{kib}}:
250322

251323
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
252-
1. In the **Data Streams** view, click the trash icon. The icon only displays if you have the `delete_index` [security privilege](elasticsearch://reference/elasticsearch/security-privileges.md) for the data stream.
324+
1. In the **Data Streams** view, click the trash can icon. The icon only displays if you have the `delete_index` security privilege for the data stream.
325+
326+
:::
327+
:::{tab-item} API
328+
:sync: api
329+
330+
You can also use an API to delete a data stream:
253331

254-
You can also use the [delete data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-stream).
332+
* In an {{stack}} deployment, use the [delete data streams](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-stream) API.
333+
* In {{serverless-full}}, use the [delete data streams](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-delete-data-stream) API.
255334

256335
```console
257336
DELETE _data_stream/my-data-stream
258337
```
259338

339+
:::
340+
::::

0 commit comments

Comments
 (0)