From 5094fb10c91150f7ec56bb3578f27fb59926551d Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 24 Oct 2024 12:19:50 +0200 Subject: [PATCH 01/17] docs(auditlogs): add audit logging sig proposal --- projects/audit-logging.md | 109 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 projects/audit-logging.md diff --git a/projects/audit-logging.md b/projects/audit-logging.md new file mode 100644 index 000000000..e6f2cab9c --- /dev/null +++ b/projects/audit-logging.md @@ -0,0 +1,109 @@ + + +# Audit Logging + +## Background and description + + + +Audit logging describes the capability of capturing audit-trail relevant events of a system to meet compliance requirements. Such events may originate from the infrastructure (e.g. a Kubernetes cluster) up to the application-level. It is a capability that is particularly relevant for providers of enterprise software. + +Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery). + +Examples of audit logs include: +- permission changes (e.g. of a service account or application user) +- modification of data +- accessing sensitive information +- failed login attempts + +### Current challenges + + + +Audit Logging is currently not within the scope of OpenTelemetry + +- no semantic conventions for audit logs in OTEL +- OTEL APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. +- OTEL collectors may lose audit logs in transit (i.e. no guarantee of delivery) + +### Goals, objectives, and requirements + + + +The goal of this project is to make OTEL fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: + +- REQ-CON-01: Semantic conventions for application-level audit logs are defined +- REQ-CON-02: Semantic conventions for infrastructure-level audit logs are defined +- REQ-SRC-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. +- REQ-PIP-01: OTEL collector must provide guaranteed delivery of audit logs, including when its process is interrupted + +## Deliverables + + + + + +- semantic convention for audit logs +- extend OTEL APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) +- extend OTEL collector for audit logging purposes (in collaboration with the respective SIG) + +## Staffing / Help Wanted + + + +The following vendors are interested in improving this area: +- SAP + +Other vendors are invited to join the discussion. + +### Required staffing + + + + + + + + +* Project lead: SAP (name tbd) +* Sponsors: tbd +* GC liaison: tbd +* Engineers: + * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) +* Maintainers/approvers: tbd + +## Timeline + + + +TBD based on community involvement. + +## Labels + + + +- audit-logging (tbc) + +## Project Board + + + + + + + + + + + +TODO: add link + +## SIG Meetings and Other Info + + + + + + + +TODO: add information \ No newline at end of file From 9337b7f9df3bd9bf3835dbf3ef53f81a0392f1a7 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 24 Oct 2024 12:21:24 +0200 Subject: [PATCH 02/17] docs(auditlogs): re-number requirements --- projects/audit-logging.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index e6f2cab9c..7e6e0afd4 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -32,10 +32,10 @@ Audit Logging is currently not within the scope of OpenTelemetry The goal of this project is to make OTEL fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: -- REQ-CON-01: Semantic conventions for application-level audit logs are defined -- REQ-CON-02: Semantic conventions for infrastructure-level audit logs are defined -- REQ-SRC-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. -- REQ-PIP-01: OTEL collector must provide guaranteed delivery of audit logs, including when its process is interrupted +- REQ-CONV-01: Semantic conventions for application-level audit logs are defined +- REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined +- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. +- REQ-PIPE-01: OTEL collector must provide guaranteed delivery of audit logs, including when its process is interrupted ## Deliverables From 75f2c579dfc4b897a9eb91fd2e1575c987e62e2c Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 24 Oct 2024 12:22:48 +0200 Subject: [PATCH 03/17] docs(auditlogs): remove template instructions --- projects/audit-logging.md | 41 --------------------------------------- 1 file changed, 41 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 7e6e0afd4..a749f5165 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -1,11 +1,7 @@ - - # Audit Logging ## Background and description - - Audit logging describes the capability of capturing audit-trail relevant events of a system to meet compliance requirements. Such events may originate from the infrastructure (e.g. a Kubernetes cluster) up to the application-level. It is a capability that is particularly relevant for providers of enterprise software. Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery). @@ -18,8 +14,6 @@ Examples of audit logs include: ### Current challenges - - Audit Logging is currently not within the scope of OpenTelemetry - no semantic conventions for audit logs in OTEL @@ -28,8 +22,6 @@ Audit Logging is currently not within the scope of OpenTelemetry ### Goals, objectives, and requirements - - The goal of this project is to make OTEL fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: - REQ-CONV-01: Semantic conventions for application-level audit logs are defined @@ -39,18 +31,12 @@ The goal of this project is to make OTEL fit for audit logging purposes that mee ## Deliverables - - - - - semantic convention for audit logs - extend OTEL APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) - extend OTEL collector for audit logging purposes (in collaboration with the respective SIG) ## Staffing / Help Wanted - - The following vendors are interested in improving this area: - SAP @@ -58,13 +44,6 @@ Other vendors are invited to join the discussion. ### Required staffing - - - - - - - * Project lead: SAP (name tbd) * Sponsors: tbd * GC liaison: tbd @@ -74,36 +53,16 @@ Other vendors are invited to join the discussion. ## Timeline - - TBD based on community involvement. ## Labels - - - audit-logging (tbc) ## Project Board - - - - - - - - - - TODO: add link ## SIG Meetings and Other Info - - - - - - TODO: add information \ No newline at end of file From f81c2f44c5e5142169f44d542298189c1f7ccf54 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit <2060790+mlenkeit@users.noreply.github.com> Date: Tue, 19 Nov 2024 14:24:44 +0100 Subject: [PATCH 04/17] docs(auditlogs): use OTel over OTEL Co-authored-by: Reiley Yang --- projects/audit-logging.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index a749f5165..8084321e6 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -16,24 +16,24 @@ Examples of audit logs include: Audit Logging is currently not within the scope of OpenTelemetry -- no semantic conventions for audit logs in OTEL -- OTEL APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. -- OTEL collectors may lose audit logs in transit (i.e. no guarantee of delivery) +- no semantic conventions for audit logs in OTel +- OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. +- OTel collectors may lose audit logs in transit (i.e. no guarantee of delivery) ### Goals, objectives, and requirements -The goal of this project is to make OTEL fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: +The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: - REQ-CONV-01: Semantic conventions for application-level audit logs are defined - REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined - REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. -- REQ-PIPE-01: OTEL collector must provide guaranteed delivery of audit logs, including when its process is interrupted +- REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted ## Deliverables - semantic convention for audit logs -- extend OTEL APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) -- extend OTEL collector for audit logging purposes (in collaboration with the respective SIG) +- extend OTel APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) +- extend OTel collector for audit logging purposes (in collaboration with the respective SIG) ## Staffing / Help Wanted From 65ae32e7b31d2628f345ad2f56396df0c6f7821a Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Tue, 19 Nov 2024 14:40:29 +0100 Subject: [PATCH 05/17] docs(auditlogs): list @reyang as first sponsor --- projects/audit-logging.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 8084321e6..d6e3519a2 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -45,7 +45,9 @@ Other vendors are invited to join the discussion. ### Required staffing * Project lead: SAP (name tbd) -* Sponsors: tbd +* Sponsors: + - Reiley Yang (@reyang) + - tbd * GC liaison: tbd * Engineers: * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) From 776b821f8aebcffc8ce41e882375dce0a1e6d430 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Tue, 19 Nov 2024 14:45:22 +0100 Subject: [PATCH 06/17] docs(auditlogs): add Microsoft to interested vendors --- projects/audit-logging.md | 1 + 1 file changed, 1 insertion(+) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index d6e3519a2..aaa0c92fd 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -39,6 +39,7 @@ The goal of this project is to make OTel fit for audit logging purposes that mee The following vendors are interested in improving this area: - SAP +- Microsoft Other vendors are invited to join the discussion. From 6dd519d9861a088c816bb8a1d10642555c5a6090 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Tue, 19 Nov 2024 14:45:45 +0100 Subject: [PATCH 07/17] docs(auditlogs): add contacts to vendor list --- projects/audit-logging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index aaa0c92fd..bf720eb67 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -38,8 +38,8 @@ The goal of this project is to make OTel fit for audit logging purposes that mee ## Staffing / Help Wanted The following vendors are interested in improving this area: -- SAP -- Microsoft +- SAP (@mlenkeit, @FWinkler79) +- Microsoft (@reyang) Other vendors are invited to join the discussion. From 2ec002d8de4ce7a74759ba80310837523edeb85b Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Tue, 19 Nov 2024 14:48:52 +0100 Subject: [PATCH 08/17] docs(auditlogs): use consistent punctuation for requirement list --- projects/audit-logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index bf720eb67..77be9c121 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -26,7 +26,7 @@ The goal of this project is to make OTel fit for audit logging purposes that mee - REQ-CONV-01: Semantic conventions for application-level audit logs are defined - REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined -- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK. +- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK - REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted ## Deliverables From d7e265f5eec18374dc3015ba0dd830d0b72e9601 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit <2060790+mlenkeit@users.noreply.github.com> Date: Wed, 20 Nov 2024 16:00:51 +0100 Subject: [PATCH 09/17] docs(auditlogs): minor word change in Challenges chapter --- projects/audit-logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 77be9c121..0b0a6e4e6 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -14,7 +14,7 @@ Examples of audit logs include: ### Current challenges -Audit Logging is currently not within the scope of OpenTelemetry +OpenTelemetry does not have a good solution for audit logging - no semantic conventions for audit logs in OTel - OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. From 405ddb540f0628577d6be167eeec28827b8d6681 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 13:37:34 +0100 Subject: [PATCH 10/17] docs(auditlogs): describe guarantee of delivery in appendix --- projects/audit-logging.md | 40 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 77be9c121..0709e6f32 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -68,4 +68,42 @@ TODO: add link ## SIG Meetings and Other Info -TODO: add information \ No newline at end of file +TODO: add information + +## Appendix + +### Appendix A: Guarantee of Delivery + +In the context of this document, guarantee of delivery describes the ability of delivering audit logs from source to destination through OTel means while ensuring that all such signals arrive at the destination and/or providing the source with a means to handle failed delivery. + +Messaging protocols that support different levels of delivery guarantees may refer to this behavior as _at least once_ or _exactly once_, as opposed to _at most once_. + +We assume that every component that is involved in the delivery of audit logs from source to destination must support guarantee of delivery individually, rather than assuming that this ability can be provided by e.g. only the collector or SDK. + +The implications of guarantee of delivery can be illustrated with an example consisting of a workload, an OTel collector, and a durable storage. The workload acts as the source and produces audit logs via the OTel SDK/API. It writes the data via OTLP to the OTel collector. The OTel collector is configured to export audit logs to a durable storage that acts as the destination such as an S3 bucket. + +The following implications would apply: + +- workload produces an audit-relevant event: + + The workload emits the event via the OTel SDK/API. It may wait for acknowledgement of receipt from the collector before proceeding. If the event is rejected or receipt is not acknowledged in time, the workload or SDK may act accordingly, e.g. retry, rollback a database transaction, inform the user, etc. + +- OTel collector receives the event: + + To ensure that the event is not lost even if the collector process is terminated or crashes, the collector may need to persist the event before acknowledging receipt to the workload or SDK. If the event cannot be persisted, receipt must be rejected. + +- OTel collector exports the event: + + Once the event is exported and the target (i.e. S3) acknowledges receipt, the event can dropped from the collector's persistence. + +- S3 receives the event: + + Acknowledges receipt after persisting the event. + + Note that this is outside the scope of the OTel. More general, when using OTel for audit logging purposes, it's the users (e.g. Ops) responsibility to configure a suitable export target. + +Note that this example may contain implementation details for illustration purposes. The actual implementation may differ as long as the requirements are met. + +The example is kept simple for illustration purposes. Many edge cases need to be discussed by the SIG, such as batch-sending of signals or handling of multiple export targets. + +It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes. \ No newline at end of file From 0adb8e5933af2929e0390668f637089111a318e3 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:23:09 +0100 Subject: [PATCH 11/17] docs(auditlogs): add sample audit logs to appendix --- projects/audit-logging.md | 81 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 0709e6f32..3109c2e25 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -106,4 +106,83 @@ Note that this example may contain implementation details for illustration purpo The example is kept simple for illustration purposes. Many edge cases need to be discussed by the SIG, such as batch-sending of signals or handling of multiple export targets. -It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes. \ No newline at end of file +It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes. + +### Appendix B: Sample Audit Log Events + +The following list contains sample audit log events in a YAML format for better readability and intentionally do not follow any OTel-related schema. + +An event consists of the event name, event-specific data, and general metadata. The individual properties of these events would ideally be reflected in common or audit log-specific semantic conventions. + +- failed login attempts + + ```yaml + event: UserLoginFailure + data: + loginMethod: oidc + failureReason: userLocked + metadata: + id: 50b925b5-0ba9-42f3-b476-8a6795000046 + timestamp: 1732193414483 + ip: 10.11.12.13 + initiator: john-doe + application: payroll + tenant: fab54af9-f978-463e-9c02-f92db1afc2b4 + ``` + +- permission changes (e.g. of a service account or application user) + + ```yaml + event: AuthnRoleToUserAdd + data: + user: jane-doe + role: editor + metadata: + id: 50b925b5-0ba9-42f3-b476-8a6795000046 + timestamp: 1732193414483 + ip: 10.11.12.13 + initiator: john-doe + application: payroll + tenant: fab54af9-f978-463e-9c02-f92db1afc2b4 + ``` + +- accessing sensitive information + + ```yaml + event: DppDataAccess + data: + channelType: web + channelId: https://payroll.example.com/user/jane-doe/compensation + dataSubjectType: employeeID + dataSubjectId: jane-doe + objectType: compensation + objectId: 1196f42b-8f12-4df0-9b1f-01c98d2c7291 + attribute: salary + value: 50000 + metadata: + id: 50b925b5-0ba9-42f3-b476-8a6795000046 + timestamp: 1732193414483 + ip: 10.11.12.13 + initiator: john-doe + application: payroll + tenant: fab54af9-f978-463e-9c02-f92db1afc2b4 + ``` + + +- modification of data + + ```yaml + event: DataModification + data: + objectType: CronJob + objectId: my-sample-cronjob + attribute: schedule + oldValue: 0 0 1 * * # monthly + newValue: 0 0 1 1 * # annually + metadata: + id: 50b925b5-0ba9-42f3-b476-8a6795000046 + timestamp: 1732193414483 + ip: 10.11.12.13 + initiator: john-doe + k8sCluster: my-sample-cluster + ``` \ No newline at end of file From a5ef343bafcd9141c3f41f10625fb8e336da1cbb Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:27:23 +0100 Subject: [PATCH 12/17] docs(auditlogs): add links to sample audit logs --- projects/audit-logging.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 3109c2e25..dd4df711a 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -6,11 +6,11 @@ Audit logging describes the capability of capturing audit-trail relevant events Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery). -Examples of audit logs include: +Examples of audit logs include: (see [Appendix B: Sample Audit Log Events]) +- failed login attempts - permission changes (e.g. of a service account or application user) -- modification of data - accessing sensitive information -- failed login attempts +- modification of data ### Current challenges @@ -185,4 +185,7 @@ An event consists of the event name, event-specific data, and general metadata. ip: 10.11.12.13 initiator: john-doe k8sCluster: my-sample-cluster - ``` \ No newline at end of file + ``` + + +[Appendix B: Sample Audit Log Events]: #appendix-b-sample-audit-log-events \ No newline at end of file From 711dc46f2cd64bafdfa3982c5ba832c616486ba3 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:27:42 +0100 Subject: [PATCH 13/17] docs(auditlogs): add links to appendix A --- projects/audit-logging.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index dd4df711a..b1ebcfd6d 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -20,6 +20,8 @@ Audit Logging is currently not within the scope of OpenTelemetry - OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. - OTel collectors may lose audit logs in transit (i.e. no guarantee of delivery) +See [Appendix A: Guarantee of Delivery] for more details + ### Goals, objectives, and requirements The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: @@ -29,6 +31,8 @@ The goal of this project is to make OTel fit for audit logging purposes that mee - REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK - REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted +See [Appendix A: Guarantee of Delivery] for more details + ## Deliverables - semantic convention for audit logs @@ -188,4 +192,5 @@ An event consists of the event name, event-specific data, and general metadata. ``` +[Appendix A: Guarantee of Delivery]: #appendix-a-guarantee-of-delivery [Appendix B: Sample Audit Log Events]: #appendix-b-sample-audit-log-events \ No newline at end of file From 087865c139a17be07204a5784279cfe2c7d4e01c Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:28:11 +0100 Subject: [PATCH 14/17] docs(auditlogs): use GitHub handle only in staffing list --- projects/audit-logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index b1ebcfd6d..072321d45 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -51,7 +51,7 @@ Other vendors are invited to join the discussion. * Project lead: SAP (name tbd) * Sponsors: - - Reiley Yang (@reyang) + - @reyang - tbd * GC liaison: tbd * Engineers: From 3876a315eaad74539bc52110f8a430a985acdb55 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Thu, 21 Nov 2024 14:28:51 +0100 Subject: [PATCH 15/17] docs(auditlogs): add svrnm as GC liaison --- projects/audit-logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 072321d45..da4112ea8 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -53,7 +53,7 @@ Other vendors are invited to join the discussion. * Sponsors: - @reyang - tbd -* GC liaison: tbd +* GC liaison: @svrnm * Engineers: * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) * Maintainers/approvers: tbd From 066501b92566ce770eb79f5633d2c1e5166aa9a5 Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Fri, 22 Nov 2024 12:13:50 +0100 Subject: [PATCH 16/17] docs(auditlogs): minor changes in wording --- projects/audit-logging.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 227c866a0..214c5026c 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -6,7 +6,7 @@ Audit logging describes the capability of capturing audit-trail relevant events Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery). -Examples of audit logs include: (see [Appendix B: Sample Audit Log Events]) +Examples of audit logs include: (see [Appendix B: Examples of Audit Log Events]) - failed login attempts - permission changes (e.g. of a service account or application user) - accessing sensitive information @@ -18,7 +18,7 @@ OpenTelemetry does not have a good solution for audit logging - no semantic conventions for audit logs in OTel - OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself. -- OTel collectors may lose audit logs in transit (i.e. no guarantee of delivery) +- OTel Collector instances may lose audit logs in transit (i.e. no guarantee of delivery) See [Appendix A: Guarantee of Delivery] for more details @@ -29,7 +29,7 @@ The goal of this project is to make OTel fit for audit logging purposes that mee - REQ-CONV-01: Semantic conventions for application-level audit logs are defined - REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined - REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK -- REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted +- REQ-PIPE-01: OTel Collector instances must provide guaranteed delivery of audit logs, including when its process is interrupted See [Appendix A: Guarantee of Delivery] for more details @@ -37,7 +37,7 @@ See [Appendix A: Guarantee of Delivery] for more details - semantic convention for audit logs - extend OTel APIs/SDKs for audit logging purposes (in collaboration with the respective SIG) -- extend OTel collector for audit logging purposes (in collaboration with the respective SIG) +- extend OTel Collector for audit logging purposes (in collaboration with the respective SIG) ## Staffing / Help Wanted @@ -54,8 +54,9 @@ Other vendors are invited to join the discussion. - @reyang - tbd * GC liaison: @svrnm -* Engineers: +* Engineers for API/SDK: * SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go) +* Engineers for OTel Collector: tbd * Maintainers/approvers: tbd ## Timeline @@ -84,23 +85,23 @@ Messaging protocols that support different levels of delivery guarantees may ref We assume that every component that is involved in the delivery of audit logs from source to destination must support guarantee of delivery individually, rather than assuming that this ability can be provided by e.g. only the collector or SDK. -The implications of guarantee of delivery can be illustrated with an example consisting of a workload, an OTel collector, and a durable storage. The workload acts as the source and produces audit logs via the OTel SDK/API. It writes the data via OTLP to the OTel collector. The OTel collector is configured to export audit logs to a durable storage that acts as the destination such as an S3 bucket. +The implications of guarantee of delivery can be illustrated with an example consisting of a workload, an OTel Collector instance, and a durable storage. The workload acts as the source and produces audit logs via the OTel API/SDK. It writes the data via OTLP to the collector. The collector is configured to export audit logs to a durable storage that acts as the destination such as an S3 bucket. The following implications would apply: - workload produces an audit-relevant event: - The workload emits the event via the OTel SDK/API. It may wait for acknowledgement of receipt from the collector before proceeding. If the event is rejected or receipt is not acknowledged in time, the workload or SDK may act accordingly, e.g. retry, rollback a database transaction, inform the user, etc. + The workload emits the event via the OTel API/SDK. It may wait for acknowledgement of receipt from the collector before proceeding. If the event is rejected or receipt is not acknowledged in time, the workload or SDK may act accordingly, e.g. retry, rollback a database transaction, inform the user, etc. -- OTel collector receives the event: +- OTel Collector receives the event: To ensure that the event is not lost even if the collector process is terminated or crashes, the collector may need to persist the event before acknowledging receipt to the workload or SDK. If the event cannot be persisted, receipt must be rejected. -- OTel collector exports the event: +- OTel Collector exports the event: Once the event is exported and the target (i.e. S3) acknowledges receipt, the event can dropped from the collector's persistence. -- S3 receives the event: +- the target (i.e. S3) receives the event: Acknowledges receipt after persisting the event. @@ -112,7 +113,7 @@ The example is kept simple for illustration purposes. Many edge cases need to be It may turn out that all OTel receivers, processors, or exporters can be made compatible with guarantee of delivery for audit logging purposes. -### Appendix B: Sample Audit Log Events +### Appendix B: Examples of Audit Log Events The following list contains sample audit log events in a YAML format for better readability and intentionally do not follow any OTel-related schema. @@ -193,4 +194,4 @@ An event consists of the event name, event-specific data, and general metadata. [Appendix A: Guarantee of Delivery]: #appendix-a-guarantee-of-delivery -[Appendix B: Sample Audit Log Events]: #appendix-b-sample-audit-log-events \ No newline at end of file +[Appendix B: Examples of Audit Log Events]: #appendix-b-examples-of-audit-log-events \ No newline at end of file From 70cbac40fad98ce67683de7fbf960c12c3beaecb Mon Sep 17 00:00:00 2001 From: Maximilian Lenkeit Date: Fri, 22 Nov 2024 13:00:19 +0100 Subject: [PATCH 17/17] docs(auditlogs): shorten requirement ids to pass spell check --- projects/audit-logging.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/projects/audit-logging.md b/projects/audit-logging.md index 214c5026c..14ab563b7 100644 --- a/projects/audit-logging.md +++ b/projects/audit-logging.md @@ -26,10 +26,10 @@ See [Appendix A: Guarantee of Delivery] for more details The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular: -- REQ-CONV-01: Semantic conventions for application-level audit logs are defined -- REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined -- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK -- REQ-PIPE-01: OTel Collector instances must provide guaranteed delivery of audit logs, including when its process is interrupted +- REQ-01: Semantic conventions for application-level audit logs are defined +- REQ-02: Semantic conventions for infrastructure-level audit logs are defined +- REQ-03: Guaranteed delivery of audit logs exported via OpenTelemetry SDK +- REQ-04: OTel Collector instances must provide guaranteed delivery of audit logs, including when its process is interrupted See [Appendix A: Guarantee of Delivery] for more details