Skip to content

Commit 7e82396

Browse files
Merge branch 'main' into sec-rn-9.2.2
2 parents d8e6178 + 065aae2 commit 7e82396

27 files changed

+291
-52
lines changed

manage-data/data-store/data-streams/set-up-data-stream.md

Lines changed: 101 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -10,28 +10,36 @@ products:
1010

1111
# Set up a data stream [set-up-a-data-stream]
1212

13+
The process of setting up a data stream in {{stack}} and {{serverless-full}} is similar, making use of their respective APIs. However, because {{serverless-short}} provides a built-in [data stream lifecycle](/manage-data/lifecycle/data-stream.md) mechanism and retention settings, you don't need to configure index lifecycle management ({{ilm-init}}) options as you do in an {{stack}} deployment.
14+
1315
To set up a data stream, follow these steps:
1416

15-
1. [Create an index lifecycle policy](#create-index-lifecycle-policy)
17+
1. [Create an index lifecycle policy](#create-index-lifecycle-policy) {applies_to}`serverless: unavailable`
1618
2. [Create component templates](#create-component-templates)
1719
3. [Create an index template](#create-index-template)
1820
4. [Create the data stream](#create-data-stream)
1921
5. [Secure the data stream](#secure-data-stream)
2022

2123
You can also [convert an index alias to a data stream](#convert-index-alias-to-data-stream).
2224

23-
::::{important}
25+
:::{important}
2426
If you use {{fleet}}, {{agent}}, or {{ls}}, skip this tutorial. They all set up data streams for you.
2527

2628
For {{fleet}} and {{agent}}, refer to [](/reference/fleet/data-streams.md). For {{ls}}, refer to the [data streams settings](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-data_stream) for the `elasticsearch output` plugin.
2729

28-
::::
29-
30+
:::
3031

3132

3233
## Create an index lifecycle policy [create-index-lifecycle-policy]
34+
```{applies_to}
35+
serverless: unavailable
36+
```
3337

34-
While optional, we recommend using {{ilm-init}} to automate the management of your data stream’s backing indices. {{ilm-init}} requires an index lifecycle policy.
38+
While optional, we recommend using the {{ilm}} ({{ilm-init}}) capability in {{stack}} deployments to automate the management of your data stream’s backing indices. {{ilm-init}} requires an index lifecycle policy.
39+
40+
:::{admonition} Simpler lifecycle management in {{serverless-short}} projects
41+
{{ilm-init}} lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {{ilm-init}} is not available in {{serverless-short}}, where performance optimizations are automatic. Instead, [data stream lifecycle](/manage-data/lifecycle/data-stream.md) is available as a data management option.
42+
:::
3543

3644
To create an index lifecycle policy in {{kib}}:
3745

@@ -100,22 +108,32 @@ When creating your component templates, include:
100108
* A [`date`](elasticsearch://reference/elasticsearch/mapping-reference/date.md) or [`date_nanos`](elasticsearch://reference/elasticsearch/mapping-reference/date_nanos.md) mapping for the `@timestamp` field. If you don’t specify a mapping, {{es}} maps `@timestamp` as a `date` field with default options.
101109
* Your lifecycle policy in the `index.lifecycle.name` index setting.
102110

103-
::::{tip}
111+
:::{tip}
104112
Use the [Elastic Common Schema (ECS)](ecs://reference/index.md) when mapping your fields. ECS fields integrate with several {{stack}} features by default.
105113

106114
If you’re unsure how to map your fields, use [runtime fields](../mapping/define-runtime-fields-in-search-request.md) to extract fields from [unstructured content](elasticsearch://reference/elasticsearch/mapping-reference/keyword.md#mapping-unstructured-content) at search time. For example, you can index a log message to a `wildcard` field and later extract IP addresses and other data from this field during a search.
115+
:::
107116

108-
::::
109-
117+
::::{tab-set}
118+
:group: set-up-ds
119+
:::{tab-item} {{kib}}
120+
:sync: kibana
110121
To create a component template in {{kib}}:
111122

112123
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
113124
1. In the **Index Templates** tab, click **Create component template**.
125+
:::
114126

115-
You can also use the [create component template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template).
127+
:::{tab-item} API
128+
:sync: api
129+
Use an API to create a component template:
130+
131+
* In an {{stack}} deployment, use the [create component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) API.
132+
* In {{serverless-full}}, use the [create component template](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-cluster-put-component-template) API.
133+
134+
To create a component template for mappings, use this request:
116135

117136
```console
118-
# Creates a component template for mappings
119137
PUT _component_template/my-mappings
120138
{
121139
"template": {
@@ -136,8 +154,11 @@ PUT _component_template/my-mappings
136154
"my-custom-meta-field": "More arbitrary metadata"
137155
}
138156
}
157+
```
158+
159+
To create a component template for index settings, use this request:
139160

140-
# Creates a component template for index settings
161+
```console
141162
PUT _component_template/my-settings
142163
{
143164
"template": {
@@ -152,6 +173,8 @@ PUT _component_template/my-settings
152173
}
153174
```
154175

176+
:::
177+
::::
155178

156179
## Create an index template [create-index-template]
157180

@@ -162,12 +185,25 @@ Use your component templates to create an index template. Specify:
162185
* Any component templates that contain your mappings and index settings.
163186
* A priority higher than `200` to avoid collisions with built-in templates. See [Avoid index pattern collisions](../templates.md#avoid-index-pattern-collisions).
164187

188+
::::{tab-set}
189+
:group: set-up-ds
190+
:::{tab-item} {{kib}}
191+
:sync: kibana
165192
To create an index template in {{kib}}:
166193

167194
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
168195
1. In the **Index Templates** tab, click **Create template**.
169196

170-
You can also use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template). Include the `data_stream` object to enable data streams.
197+
:::
198+
199+
:::{tab-item} API
200+
:sync: api
201+
Use an API to create an index template:
202+
203+
* In an {{stack}} deployment, use the [create an index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) API.
204+
* In {{serverless-full}}, use the [create an index template](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-put-index-template) API.
205+
206+
Include the `data_stream` object to enable data streams:
171207

172208
```console
173209
PUT _index_template/my-index-template
@@ -183,10 +219,12 @@ PUT _index_template/my-index-template
183219
}
184220
```
185221

222+
:::
223+
::::
186224

187225
## Create the data stream [create-data-stream]
188226

189-
[Indexing requests](../data-streams/use-data-stream.md#add-documents-to-a-data-stream) [Indexing requests](./use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field.
227+
[Indexing requests](../data-streams/use-data-stream.md#add-documents-to-a-data-stream) add documents to a data stream. These requests must use an `op_type` of `create`. Documents must include a `@timestamp` field.
190228

191229
To automatically create your data stream, submit an indexing request that targets the stream’s name. This name must match one of your index template’s index patterns.
192230

@@ -203,8 +241,10 @@ POST my-data-stream/_doc
203241
"message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
204242
}
205243
```
244+
You can also use an API to manually create the data stream:
206245

207-
You can also manually create the stream using the [create data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream). The stream’s name must still match one of your template’s index patterns.
246+
* In an {{stack}} deployment, use the [create a data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream) API.
247+
* In {{serverless-full}}, use the [create a data stream](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-create-data-stream) API.
208248

209249
```console
210250
PUT _data_stream/my-data-stream
@@ -216,14 +256,19 @@ After it's been created, you can view and manage this and other data streams fro
216256

217257
Use [index privileges](elasticsearch://reference/elasticsearch/security-privileges.md#privileges-list-indices) to control access to a data stream. Granting privileges on a data stream grants the same privileges on its backing indices.
218258

219-
For an example, see [Data stream privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
259+
For an example, refer to [Data stream privileges](../../../deploy-manage/users-roles/cluster-or-deployment-auth/granting-privileges-for-data-streams-aliases.md#data-stream-privileges).
220260

221261

222262
## Convert an index alias to a data stream [convert-index-alias-to-data-stream]
223263

224-
Prior to {{es}} 7.9, you’d typically use an [index alias with a write index](../../lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](../../lifecycle/data-tiers.md).
264+
Prior to {{es}} 7.9, you’d typically use an index alias with a write index to manage time series data. Data streams replace this functionality, require less maintenance, and automatically integrate with [data tiers](../../lifecycle/data-tiers.md).
265+
266+
You can convert an index alias with a write index to a data stream with the same name, using an API:
225267

226-
To convert an index alias with a write index to a data stream with the same name, use the [migrate to data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-migrate-to-data-stream). During conversion, the alias’s indices become hidden backing indices for the stream. The alias’s write index becomes the stream’s write index. The stream still requires a matching index template with data stream enabled.
268+
* In an {{stack}} deployment, use the [convert an index alias to a data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-migrate-to-data-stream) API.
269+
* In {{serverless-full}}, use the [convert an index alias to a data stream](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-migrate-to-data-stream) API.
270+
271+
During conversion, the alias's indices become hidden backing indices for the stream. The alias's write index becomes the stream's write index. The stream still requires a matching index template with data stream enabled.
227272

228273
```console
229274
POST _data_stream/_migrate/my-time-series-data
@@ -232,28 +277,64 @@ POST _data_stream/_migrate/my-time-series-data
232277

233278
## Get information about a data stream [get-info-about-data-stream]
234279

280+
You can review metadata about each data stream using the {{kib}} UI (visual overview) or the API (raw JSON).
281+
282+
::::{tab-set}
283+
:group: set-up-ds
284+
:::{tab-item} {{kib}}
285+
:sync: kibana
235286
To get information about a data stream in {{kib}}:
236287

237288
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
238289
1. In the **Data Streams** tab, click the data stream’s name.
239290

240-
You can also use the [get data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream).
291+
:::{tip}
292+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
293+
Starting with {{es}} version 9.2, you can use the [**Streams**](/solutions/observability/streams/streams.md) page to view the details of a data stream. The **Streams** page provides a centralized interface for managing your data in {{kib}}.Select a stream to view its details.
294+
:::
295+
296+
:::
297+
298+
:::{tab-item} API
299+
:sync: api
300+
You can also use an API to get this information:
301+
302+
* In an {{stack}} deployment, use the [get data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream) API.
303+
* In {{serverless-full}}, use the [get data streams](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-get-data-stream) API.
241304

242305
```console
243306
GET _data_stream/my-data-stream
244307
```
245308

309+
:::
310+
::::
246311

247312
## Delete a data stream [delete-data-stream]
248313

314+
You can delete a data stream and its backing indices via the {{kib}} UI or an API. To complete this action, you need the `delete_index` [security privilege](elasticsearch://reference/elasticsearch/security-privileges.md) for the data stream.
315+
316+
::::{tab-set}
317+
:group: set-up-ds
318+
:::{tab-item} {{kib}}
319+
:sync: kibana
320+
249321
To delete a data stream and its backing indices in {{kib}}:
250322

251323
1. Go to the **Index Management** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
252-
1. In the **Data Streams** view, click the trash icon. The icon only displays if you have the `delete_index` [security privilege](elasticsearch://reference/elasticsearch/security-privileges.md) for the data stream.
324+
1. In the **Data Streams** view, click the trash can icon. The icon only displays if you have the `delete_index` security privilege for the data stream.
325+
326+
:::
327+
:::{tab-item} API
328+
:sync: api
329+
330+
You can also use an API to delete a data stream:
253331

254-
You can also use the [delete data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-stream).
332+
* In an {{stack}} deployment, use the [delete data streams](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-stream) API.
333+
* In {{serverless-full}}, use the [delete data streams](https://www.elastic.co/docs/api/doc/elasticsearch-serverless/operation/operation-indices-delete-data-stream) API.
255334

256335
```console
257336
DELETE _data_stream/my-data-stream
258337
```
259338

339+
:::
340+
::::

manage-data/ingest/ingest-reference-architectures/agent-es-airgapped.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,10 @@ Use when
2424

2525
## Resources [airgapped-es-resources]
2626

27-
Info for air-gapped environments:
27+
For more information on using Elastic in air-gapped environments, check:
2828

29-
* [Installing the {{stack}} in an air-gapped environment](../../../deploy-manage/deploy/cloud-enterprise/air-gapped-install.md)
29+
* [Install the {{stack}} in an air-gapped environment](/deploy-manage/deploy/self-managed/air-gapped-install.md)
30+
* [Install {{ece}} in an air-gapped environment](/deploy-manage/deploy/cloud-enterprise/air-gapped-install.md)
31+
* [Run {{agents}} in an air-gapped environment](/reference/fleet/air-gapped.md)
3032
* [Using a proxy server with Elastic Agent and Fleet](/reference/fleet/fleet-agent-proxy-support.md)
3133

reference/fleet/air-gapped.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,13 @@
11
---
2+
navigation_title: Air-gapped environments
23
mapped_pages:
34
- https://www.elastic.co/guide/en/fleet/current/air-gapped.html
45
products:
56
- id: fleet
67
- id: elastic-agent
78
---
89

9-
# Air-gapped environments [air-gapped]
10+
# Run {{agents}} in an air-gapped environment [air-gapped]
1011

1112
When running {{agent}}s in a restricted or closed network, you need to take extra steps to make sure:
1213

0 commit comments

Comments
 (0)