Initial multi channel trace file format specification

[pmai]

Based on the initial specification work in #833 with refactoring for better integration into OSI and with clearer separation of concerns.

[pmai]

Note that this leaves out for now specifics of how to structure additional non-OSI meta-data, which should likely be handled in layered specifications based on the rdns prefixes, and any larger changes to the naming convention. The later is only a recommendation in any case, and if further changes are wanted they should apply to the whole convention and be handled in a separate PR (e.g. one could think about making the timestamp optional, or dropping the protobuf version, but that would make sense - or not - for all formats and hence is not related to the multi trace file format).

[pmai]

Review comments from discussion with @jdsika and @thomassedlmayer

[ClemensLinnhoff]

I don't think any further review from @TimmRuppert or me is necessary here. Feel free to merge your solution whenever you are ready.

[jdsika]

@pmai is it possible to commit the discussed changes until 1pm and I have a final review and put the PR into ready state with the tag "readyForCCB"?

[jdsika]

@TimmRuppert I actually cannot find a version number for the MCAP specification but only different ones for the associated libraries and CLI which is then different for Python, C++, etc .... I am a bit confused. Is this the specification version 0?

[pmai]

The spec goes about it in a very round-about way:

Magic
An MCAP file must begin and end with the following magic bytes:

0x89, M, C, A, P, 0x30, \r, \n

The byte following "MCAP" is the major version byte. 0x30 is the ASCII character 0. Any changes to this specification document (i.e. adding fields to records, introducing new records) will be binary backward-compatible within the major version.

(Emphasis by me)

So it is stated quite unclearly, but I would treat this as the major version of the specification, given the statements.

[jdsika]

That is what confuse me. "include the major version like this" but not "this spec IS major version 0" ...

[ijelec]

There are just two points on naming convention and time definitions that were discussed during our meeting. No other objections to merging.

[jdsika]

from internal:

I have some questions about the PR:

1. The OSI multi channel trace file format is a binary file format that allows for storing multiple serialized OSI message streams of the same or different types in one trace file, along with additional meta-data, and other related data streams. - what is "other related data streams" how do we know whether it is related or not.
2. For each OSI top-level message type that is used in one or more OSI channels, the OSI multi channel trace file must contain a corresponding schema record in the summary section. - What happens to non-top-level message type?
3. An OSI channel is a data stream within the OSI multi channel trace file that contains serialized OSI top-level messages of the same type. Note that non-top-level messages must not be stored directly in OSI channels. - How are non top level messages stored?

[TimmRuppert]

1. The OSI multi channel trace file format is a binary file format that allows for storing multiple serialized OSI message streams of the same or different types in one trace file, along with additional meta-data, and other related data streams. - what is "other related data streams" how do we know whether it is related or not.

Ultimately, I think it's up to the user to decide. For example, a sensor model that also outputs some non-OSI debug data could still be considered a related data stream. On the other hand, a list of my latest cookie recipes probably wouldn't be ;)

2. For each OSI top-level message type that is used in one or more OSI channels, the OSI multi channel trace file must contain a corresponding schema record in the summary section. - What happens to non-top-level message type?

The answer is basically question 3.

3. An OSI channel is a data stream within the OSI multi channel trace file that contains serialized OSI top-level messages of the same type. Note that non-top-level messages must not be stored directly in OSI channels. - How are non top level messages stored?

Well, you don’t. It seems the documentation might leave room for misinterpretation. Perhaps it should be made more explicit - for example, by stating clearly at the beginning of the section 'The following rules apply to OSI multi-channel trace files' that non-top-level messages may only be stored as sub-elements within the channel of a top-level message.

[thomassedlmayer]

The OSI multi channel trace file format is a binary file format that allows for storing multiple serialized OSI message streams of the same or different types in one trace file, along with additional meta-data, and other related data streams. - what is "other related data streams" how do we know whether it is related or not.

I think it is the responsibility of the creator of a file to ensure that related data streams are written/described in a way so that a reader of the file knows that it is indeed related and also how it is related. For the top-level OSI messages we thought about that and therefore defined rules for timestamps etc. But for data streams/channels that are not specified in this PR this is not the responsibility of the OSI specification to define. What is the goal of putting unrelated data in there?

An OSI channel is a data stream within the OSI multi channel trace file that contains serialized OSI top-level messages of the same type. Note that non-top-level messages must not be stored directly in OSI channels. - How are non top level messages stored?

You can still put non-top-level messages as "non-official top-level messages" into an mcap container if you desire to do so. This does not meet the OSI-mcap specification in the sense that there is a high probability that standard-conform tools are able to work with it. Though, the specification does not keep you from putting arbitrary data into your mcap containers. That's why it specifically allows to have "other related data streams" in the same mcap container. And in my opinion, if it is technically possible to pack non-top-level OSI protobuf-encoded messages into a separate mcap channel, you're free to do so, even if the relation to the rest of the OSI data might not be clear anymore (because of missing timestamp etc.).

As discussed with @TimmRuppert all issues have been resolved.