Skip to content

Commit 77adcd2

Browse files
HofiOneHofiOne
authored
OpenTelemetry (#42)
Documented OpenTelemetry source, destination and parser. Signed-off-by: Zsolt Gyulai (zgyulai) <[email protected]>
2 parents 95ed4c3 + 1db9c0a commit 77adcd2

File tree

6 files changed

+313
-0
lines changed

6 files changed

+313
-0
lines changed

‎_data/external_links.yml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -906,6 +906,21 @@ opensearch:
906906
url: https://opensearch.org
907907
title: [ "OpenSearch" ]
908908

909+
op-tel:
910+
id: op-tel
911+
url: https://opentelemetry.io/
912+
title: [ "OpenTelemetry" ]
913+
914+
op-tel-prtcl:
915+
id: op-tel-prtcl
916+
url: https://opentelemetry.io/docs/specs/otlp/
917+
title: [ "OpenTelemetry Protocol (OTLP/gRPC)" ]
918+
919+
gh-op-tel-prot:
920+
id: gh-op-tel-prot
921+
url: https://github.com/open-telemetry/opentelemetry-proto/tree/v0.20.0/opentelemetry/proto
922+
title: [ "OpenTelemetry proto files" ]
923+
909924
osquery:
910925
id: osquery
911926
url: https://osquery.io
@@ -1165,3 +1180,4 @@ w3c-xml:
11651180
id: w3c-xml
11661181
url: https://www.w3.org/TR/REC-xml/
11671182
title: [ "W3C Recommendation: Extensible Markup Language (XML)" ]
1183+

‎doc/_admin-guide/060_Sources/102_OpenTelemetry/000_opentelemetry_source_options.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
---
2+
title: opentelemetry() source options
3+
id: adm-src-optel-opt
4+
---
5+
6+
The following options are available for the `opentelemetry()` source.
7+
8+
## auth()
9+
10+
The `auth()` option can be used to set the authentication of the driver. The default state of this option is `insecure`, as it is not defined.
11+
12+
The following sub-options are available for `auth()`.
13+
14+
### adc()
15+
16+
This option is an authentication method that is only available for destinations. For more information, see Application Default Credentials.
17+
18+
### alts()
19+
20+
This option is an accessible authentication available for Google infrastructures. Service accounts can be listed with the nested `target-service-account()` option, to match these against the server.
21+
22+
#### Example: configure an opentelemetry() source using auth(alts())
23+
24+
```config
25+
source {
26+
opentelemetry(
27+
port(4317)
28+
auth(alts())
29+
);
30+
};
31+
32+
```
33+
34+
### insecure()
35+
36+
This option can be used to disable authentication: `auth(insecure())`.
37+
38+
### tls()
39+
40+
The `tls()` option accepts the following nested sub-options.
41+
* ca-file()
42+
* key-file()
43+
* cert-file()
44+
45+
#### Example: configure an OpenTelemetry source using auth(tls())
46+
47+
```config
48+
destination {
49+
opentelemetry(
50+
url("your-otel-server:1234")
51+
auth(
52+
tls(
53+
ca-file("/path/to/ca.pem")
54+
key-file("/path/to/key.pem")
55+
cert-file("/path/to/cert.pem")
56+
)
57+
)
58+
);
59+
};
60+
61+
```
62+
63+
{% include doc/admin-guide/options/channel-args.md %}
64+
65+
## concurrent-requests()
66+
67+
| Type:| integer|
68+
|Default:| 2|
69+
70+
*Description:* This option configures the upper limit of in-flight gRPC requests per worker. It is advisd to set this value in the range of 10s or 100s when there are a high number of clients sending simultaneously. In an optimzed solution, the number of `workers()` and `concurrent-requests()` is greater than or equal to the number of clients. However, this can cause an increase in memory usage.
71+
72+
## log-fetch-limit()
73+
74+
| Type:| number|
75+
|Default:| 100|
76+
77+
*Description:* This option specifies the upper limit of messages received from a source during a single poll loop. The destination queues can fill up before flow-control could stop reading if the defined `log-fetch-limit()` is too high.
78+
79+
{% include doc/admin-guide/options/port.md %}
80+
81+
{% include doc/admin-guide/options/workers.md %}

‎doc/_admin-guide/060_Sources/102_OpenTelemetry/README.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
---
2+
title: 'OpenTelemetry source'
3+
short_title: OpenTelemetry
4+
id: adm-src-optel
5+
description: >-
6+
In syslog-ng OSE 4.3 and later versions it is possible to collect logs, metrics and traces from OpenTelemetry clients. For more information see the OpenTelemetry Protocol (OTLP/gRPC).
7+
---
8+
9+
In the example below, an OpenTelemetry source and destination are defined. The incoming data is forwarded to a receiver.
10+
11+
### Example: Forwarding OpenTelemetry data
12+
13+
```config
14+
log otel_forward_mode_alts {
15+
source {
16+
opentelemetry(
17+
port(4317)
18+
auth(alts())
19+
);
20+
};
21+
22+
destination {
23+
opentelemetry(
24+
url("otel-server:1234")
25+
auth(alts())
26+
);
27+
};
28+
};
29+
```
30+
31+
**NOTE:** The syslog-ng OSE application does not parse the fields of the incoming messages into name-value pairs. It is only capable of forwarding messages using the `opentelemetry()` destination. For information on parsing OpenTelemetry messages, see the OpenTelemetry parser section.
32+
{: .notice--info}

‎doc/_admin-guide/070_Destinations/157_OpenTelemetry/000_opentelemetry-destination-options.md

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
title: opentelemetry() destination options
3+
id: adm-dest-optel-opt
4+
---
5+
6+
The following options are available for the opentelemetry() destination.
7+
8+
## auth()
9+
10+
The `auth()` option can be used to set the authentication of the driver. The default state of this option is `insecure`, as it is not defined.
11+
12+
The following sub-options are available for `auth()`.
13+
14+
### adc()
15+
16+
This option is an authentication method that is only available for destinations. For more information, see Application Default Credentials.
17+
18+
### alts()
19+
20+
This option is an accessible authentication available for Google infrastructures. Service accounts can be listed with the nested `target-service-account()` option, to match these against the server.
21+
22+
#### Example: configure an opentelemetry() destination using auth(alts())
23+
24+
```config
25+
destination {
26+
opentelemetry(
27+
url("otel-server:1234")
28+
auth(alts())
29+
);
30+
};
31+
32+
```
33+
34+
### insecure()
35+
36+
This option can be used to disable authentication: `auth(insecure())`.
37+
38+
### tls()
39+
40+
The `tls()` option accepts the following nested sub-options.
41+
* ca-file()
42+
* key-file()
43+
* cert-file()
44+
45+
```config
46+
destination {
47+
opentelemetry(
48+
url("your-otel-server:12346")
49+
auth(
50+
tls(
51+
ca-file("/path/to/ca.pem")
52+
key-file("/path/to/key.pem")
53+
cert-file("/path/to/cert.pem")
54+
)
55+
)
56+
);
57+
};
58+
```
59+
{% include doc/admin-guide/options/batch-bytes.md %}
60+
61+
{% include doc/admin-guide/options/batch-lines.md %}
62+
63+
{% include doc/admin-guide/options/batch-timeout.md %}
64+
65+
{% include doc/admin-guide/options/channel-args.md %}
66+
67+
## compression()
68+
69+
| Type:| boolean|
70+
|Default:| `no`|
71+
72+
Available in syslog-ng OSE 4.5 and later versions.
73+
74+
*Description:* This option enables compression in gRPC requests. Currently, only the deflate compression method is supported.
75+
76+
{% include doc/admin-guide/options/workers.md %}

‎doc/_admin-guide/070_Destinations/157_OpenTelemetry/README.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
title: 'OpenTelemetry destination'
3+
short_title: OpenTelemetry
4+
id: adm-dest-optel
5+
description: >-
6+
In syslog-ng OSE 4.3 and later versions it is possible to send logs, metrics and traces from OpenTelemetry clients. For more information see the OpenTelemetry Protocol (OTLP/gRPC).
7+
---
8+
9+
In the example below, an OpenTelemetry source and destination are defined. It receives data and forwards it to a receiver.
10+
11+
### Example: Forwarding OpenTelemetry data
12+
13+
```config
14+
log otel_forward_mode_alts {
15+
source {
16+
opentelemetry(
17+
port(12345)
18+
auth(alts())
19+
);
20+
};
21+
22+
destination {
23+
opentelemetry(
24+
url("my-otel-server:12345")
25+
auth(alts())
26+
);
27+
};
28+
};
29+
```
30+
31+
**NOTE:** Only the `url()` option is mandatory to definie in the destination, which includes the port number.
32+
{: .notice--info}
33+
34+
### Example: Sending log messages to an OpenTelemetry destination
35+
36+
```config
37+
log non_otel_to_otel_tls {
38+
source {
39+
network(
40+
port(12346)
41+
);
42+
};
43+
44+
destination {
45+
opentelemetry(
46+
url("my-otel-server:12346")
47+
auth(
48+
tls(
49+
ca-file("/path/to/ca.pem")
50+
key-file("/path/to/key.pem")
51+
cert-file("/path/to/cert.pem")
52+
)
53+
)
54+
);
55+
};
56+
};
57+
```

‎doc/_admin-guide/120_Parser/024_opentelemetry_parser.md

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
title: OpenTelemetry parser
3+
parser: OpenTelemetry
4+
id: adm-parser-optel
5+
---
6+
7+
The syslog-ng OSE application does not parse the fields of the incoming messages into name-value pairs. It is only capable of forwarding messages using the `opentelemetry()` destination. To parse the fields into name-value pairs, use the `opentelemetry()` parser.
8+
9+
This parser parses the fields into name-value pairs starting with the `.otel.` prefix.
10+
11+
* The type of the message is stored in `.otel.type`. Accepted values:
12+
* `log`
13+
* `metric`
14+
* `span`
15+
* Resource information is mapped into `.otel.resource.<...>`. Examples:
16+
* `.otel.resource.dropped_attributes_count`
17+
* `.otel.resource.schema_url`
18+
* Scope information is mapped into `.otel.scope.<...>`. Examples:
19+
* `.otel.scope.name`
20+
* `.otel.scope.schema_url`
21+
* The fields of log records are mapped into `.otel.log.<...>`. Examples:
22+
* `.otel.log.body`
23+
* `.otel.log.severity_text`
24+
* The fields of metrics are mapped into `.otel.metric.<...>`. Example:
25+
* `.otel.metric.name`
26+
* `.otel.metric.unit`
27+
* The type of the metric is mapped into `.otel.metric.data.type`. Possible values:
28+
* `gauge`
29+
* `sum`
30+
* `histogram`
31+
* `exponential_histogram`
32+
* `summary`
33+
* The actual data is mapped into `.otel.metric.data.<type>.<...>`. Example:
34+
* `.otel.metric.data.gauge.data_points.0.time_unix_nano`
35+
* The fields of traces are mapped into `.otel.span.<...>`. Example:
36+
* `.otel.span.name`
37+
* `.otel.span.trace_state`
38+
Repeated fields have an index, for example, `.otel.span.events.5.time_unix_nano`.
39+
40+
For more information on the parsed fields, the OpenTelemetry proto files can be accessed on GitHub.
41+
42+
## Mapping data types
43+
44+
String, bool, int64, double, and bytes values are mapped to their respective syslog-ng OSE name-value type, for example, `.otel.resource.attributes.string_key` becomes a string value.
45+
46+
The mapping of AnyValue type fields is limited.
47+
48+
`ArrayValue` and `KeyValueList` types are stored serialized with the `protobuf` type.
49+
50+
**NOTE:** `protobuf` and bytes types are only available, unless explicitly type cast. For example, `bytes(${.otel.log.span_id})`. When using template functions, use --include-bytes, for example, `$({format-json} .otel.* --include-bytes)`. In the case of `$({format-json})`, the content is base64-encoded into the bytes content.
51+
{: .notice--info}

0 commit comments

Comments
 (0)