MCAP spec proposal added

Signed-off-by: Timm Ruppert <[email protected]>

Author: TimmRuppert

doc/architecture/formatting_script.adoc

@@ -2,11 +2,16 @@ ifndef::include-only-once[]
 :root-path: ../
 include::{root-path}_config.adoc[]
 endif::[]
-= OSI trace file naming conventions
+= Native binary and Human-readable Formats
 
-**Name format**
+== Binary .osi Format
+Messages are separated by a four-byte, little-endian, unsigned integer specifying the length of each message.
 
-The names of OSI trace files should have the following format:
+== Human-readable .txth Format
+Messages are stored as plain text, separated by newlines.
+
+== Naming Convention
+Binary .osi and human-readable .txth files should follow this naming convention:
 
 ----
 <timestamp>_<type>_<osi-version>_<protobuf-version>_<number-of-frames>_<custom-trace-name>.osi
@@ -76,3 +81,5 @@ The recommended file name is:
 ----
 20210818T150542Z_sv_312_300_1523_highway.osi
 ----
+
+NOTE: This naming convention does not apply to .mcap files.

doc/architecture/trace_file_naming.adoc doc/architecture/trace_file_binary_and_human_readable_formats.adoc

@@ -0,0 +1,39 @@
+= MCAP Format
+
+== General Requirements
+- Must comply with the https://mcap.dev/spec[MCAP format specification] version `0x30`
+- Must allow other non-OSI data to be present in the MCAP file
+- Message records must be written into `chunk records` for indexed files
+
+== Schema
+- `name` field: Full message type name, including package (e.g., `osi3.SensorData`)
+- `encoding` field: Must be `protobuf`
+- `data` field: String-encoded `google::protobuf::FileDescriptorSet` for the OSI top-level message
+
+== Channel
+- `message_encoding` field: Must be "protobuf"
+- `metadata` field: Should include a brief description of data origin and purpose
+
+== Message
+- `publish_time` field: 
+  - Must reflect the timestamp of the stored OSI top-level message if it contains a timestamp field
+  - Should represent a plausible value to express message order in relation to timestamps of other top-level messages if the message doesn't contain a timestamp (e.g. SensorViewConfiguration)
+  - Must be in nanoseconds
+- `log_time` field: Must reflect the time when the message was enqueued for MCAP file addition
+  - Must reflect the time when the top-level OSI message was enqueued for addition to the MCAP file
+  - Zero time doesn't need to coincide with the Unix epoch, but it is recommended
+  - Must be in nanoseconds
+
+== Metadata
+- Must include metadata with the name `versions` containing at least the following key-value pairs:
+  * `osi`: SemVer version of OSI specification
+  * `protobuf`: SemVer version of protobuf library
+- Must include metadata with the name `creation_date` containing at least the following key-value pairs:
+  * `timestamp`: ISO 8601 formatted creation time
+- Must include metadata with the name `description` containing at least the following key-value pairs:
+  * `text`: Scenario description
+- Additional custom metadata may be added
+
+== Compression
+- OSI-compliant tooling must support compression types: `none`, `lz4`, and `zstd`
+- Recommended compression in use should be either `lz4` or `zstd`

doc/architecture/trace_file_example.adoc

@@ -3,22 +3,25 @@ ifndef::include-only-once[]
 include::{root-path}_config.adoc[]
 endif::[]
 [#top-osi_trace_file_formats]
-= OSI trace file formats
+= Overview file formats
 
-There are two formats for storing multiple serialized OSI messages in one trace file.
+There are three formats for storing OSI messages in trace files:
 
 *.osi::
-Binary trace file.
-Messages are separated by a length specification before each message.
-The length is represented by a four-byte, little-endian, unsigned integer.
-The length does not include the integer itself.
+Native binary trace file.
 
 *.txth::
 Human-readable plain-text trace file.
-Messages are separated by newlines.
+
+*.mcap::
+Binary trace file supporting more advanced features like indexed data, additional metadata and more.
 
 NOTE: Previous releases of OSI also supported a so-called plain-text trace file format, with file extension `.txt`.
 This legacy format did not contain plain-text, but rather binary protobuf messages separated by a special separator.
 For obvious reasons the format was deprecated and fully replaced with the `.osi` binary file format.
 This release no longer contains any support for the legacy `.txt` file format.
 These files may be used for manual checks.
+
+
+TIP: For efficient handling of trace files, you can utilize the specification-compliant tools and libraries provided in the companion https://github.com/OpenSimulationInterface/osi-utilities[osi-utilities] repository. For example, convenient writer and reader classes are provided handling OSI messages for the different file formats.
+

doc/architecture/trace_file_mcap_format.adoc

@@ -57,19 +57,11 @@ include::./architecture/packaging_layer.adoc[leveloffset=+3]
 
 === OSI trace files
 
-include::./architecture/trace_file_formats.adoc[leveloffset=+3]
+include::./architecture/trace_file_overview_file_formats.adoc[leveloffset=+3]
 
-include::./architecture/trace_file_naming.adoc[leveloffset=+3]
+include::./architecture/trace_file_binary_and_human_readable_formats.adoc[leveloffset=+3]
 
-// === Files and scripts
-
-// include::./architecture/proto-files.adoc[leveloffset=+3]
-
-// include::./architecture/test_scripts.adoc[leveloffset=+3]
-
-include::./architecture/trace_file_example.adoc[leveloffset=+3]
-
-include::./architecture/formatting_script.adoc[leveloffset=+3]
+include::./architecture/trace_file_mcap_format.adoc[leveloffset=+3]
 
 
 // Setting up OSI