Navigation to and from the endpoint serving version metadata

[arnoweiss]

Community members have provided feedback (see #143) to the current versioning mechanism in the Dataspace Protocol. Those are valuable contributions and I've tried to distill the initial discussion into a more structured format. Please provide your criticism below: Are the diagrams correct? Are there additional features to be considered for analysis?

Approach A - Status Quo

sequenceDiagram
    participant PC as Provider DSP-Server
    participant PW as Provider .well-known
    participant PD as Provider Did Doc
    participant CC as Consumer DSP-Server

    autonumber
    CC ->> PD: get
    PD -->> CC: did doc with service pointing to<br/> a CatalogService endpoint
    CC ->> CC: strip all except <root>
    CC ->> PW: get <root>/.well-known/dspace-version
    PW -->> CC: HTTP 200 with dsp-versions
    CC ->> CC: choose endpoint<br/>strip .well-known/dspace-version<br/>append path-segs
    CC ->> PC: POST <root>/path-segs/catalog/request

Loading

Approach B - dedicated metadata endpoint

sequenceDiagram
    participant PC as Provider DSP-server
    participant PD as Provider Did Doc
    participant CC as Consumer DSP-server

    autonumber
    CC ->> PD: get
    PD -->> CC: did doc with service pointing to<br/> <root>/path
    CC->> CC: append metadata
    CC ->> PC: get <root>/path/metadata
    PC -->> CC: HTTP 200 with path-segs and versions
    CC ->> CC: choose endpoint<br/>strip metadata<br/>append path-segs
    CC ->> PC: POST <root>/path-segs/catalog/request

Loading

Approach C - Status Quo with different did-service

sequenceDiagram
    participant PC as Provider DSP-server
    participant PW as Provider .well-known
    participant PD as Provider Did Doc
    participant CC as Consumer DSP-server

    autonumber
    CC ->> PD: get
    PD -->> CC: did doc with service pointing to<br/> a DspMetadata endpoint at<br/> <root>/.well-known/dspace-version
    CC ->> PW: get <root>/.well-known/dspace-version
    PW -->> CC: HTTP 200 with path-segs and versions
    CC ->> CC: choose endpoint<br/>strip .well-known/dspace-version<br/>append path-segs
    CC ->> PC: POST <root>/path-segs/catalog/request

Loading

Analysis

	Approach A	Approach B	Approach C	Comment
client can try version-unaware	+	-	-	Only pointing to the catalog service directly allows a client to simply try without being sure. All other approaches take another indirection via the versions document
approach also works for DataService	+	+	+	The Catalog.service array contains link where specific datasets can be negotiated for. To save an API-call, the version data would have to be added to the DataService object.
discovery in a non-did based dataspace	+	-	+	If a client knows only the , he knows where to obtain the versions data.
ease of path navigation	-	+	+	The case can be made that it's easier to append than to substract path segments. I'm pretty sure all approaches are deterministic.
fault-tolerance for did-based but non RFC8615 compliant connectors	-	+	+	If the navigation substracts specific path elements instead of substracting all until, it is possible to ""heal"" a /path/.well-known endpoint by putting it into the did doc like that. It's clear that this should not happen anyway. Compare step A3 and C2-3.

[jimmarino]

Thanks @arnoweiss for putting this together.

I think we need to have a clear problem statement. Specifically, is it claimed that the current approach cannot support key URL and deployment patterns? For example, is it claimed that given the current approach, the following URL patterns:

https://provider.com/connectors/foo
https://provider.com/connectors/bar

cannot be backed by two catalog services supporting different DSP protocol versions?

Before diving into details, I also want to point out that URL (and by extension, URI) parsing is extremely easy in modern programming languages. For example, in Java, to handle the current scheme, one would do the following:

URI.create("foo.bar.com/some/random/path").getHost()  // returns 'foo.bar.com'

Other languages such as C#, Go, and Rust are just as easy. Given that, I don't think an argument can be made that the current approach is "too difficult to parse".

Here are notes for discussion:

client can try version-unaware

It's important to note that Approach A is independent of DIDs. Discovery is achieved by ensuring the metadata URL is derivable from the catalog URL via the ./well-known RFC. Using this approach, the simple cases (i.e., a single DSP version) remain simple: the client invokes the provider's catalog service URL. Versioning cases are also straightforward: derive the metadata URL from the catalog service URL (one line of Java code, see above), parse the result, and follow the link to the appropriate version. We could make this even simpler by requiring the path returned in the /.well-known document to be an absolute URL (see below).

Approach B ties DSP to Web DIDs because it requires the client to always resolve the provider's Web DID to obtain the metadata URL and then parse the result to resolve an appropriate catalog service URL. In my opinion, this requirement is a non-starter. It also makes the simple case complex. Every request must resolve a DID, parse it, resolve the metadata URL, parse it, and resolve the catalog service.

Approach C appears to replace the catalog service endpoint with a metadata endpoint. This is more complex because clients must always resolve and parse the metadata.

approach also works for DataService

Adding version information to the DataService object will require more specification work and is a different way to convey the same information, thereby creating duplicate approaches. In addition, it can be extremely difficult to implement. Consider the case where the infrastructure backing a data service is a separate deployment. How will the catalog service know which versions the remote deployment supports? The current approach can handle this by using subdomains.

discovery in a non-did based dataspace

As mentioned above, IMO it's a non-starter to require Web DIDs for DSP. The current EDWG specification architecture mandates a separation between the base protocol (DSP) and identity layers. We would need to change that mandate.

ease of path navigation

Based on the parsing example above, Approach A is as easy as Approach B, since no subtraction has to be done in modern languages. There is one case that is slightly more difficult, but not unreasonably so, given the following:

https://provider.com/connectors/foo
https://provider.com/connectors/bar

the ./well-known/dspace-version may return:

{
  "protocolVersions": [
    {
      "version": "2024-1",
      "path": "/connectors/foo/2024-1"
    },
    {
      "version": "2025-1",
      "path": "connectors/bar/2025-1"
    },
    {
      "version": "2025-1",
      "path": "connectors/bar/2025-1"
    }
  ]
}

We could specify that the path property must be an absolute URL.

fault-tolerance for did-based but non-RFC8615 compliant connectors

There are a couple of notes worth mentioning here. "Fault tolerance" concerns whether an approach supports a non-compliant RFC8615 implementation. Specifically, RFC8615 mandates that, "Well-known URIs are rooted in the top of the path's hierarchy; they are not well-known by definition in other parts of the path." Approach A follows the RFC specification. However, it does not restrict implementations from supporting other approaches. Whether this is desirable from an interoperability standpoint is another story, since provider connectors that use improperly formed Well-Known URIs will not be interoperable.

By extension, approaches B and C specifically violate RFC8615.

[lgblaumeiser]

I do not agree to the initial analysis, here are my thoughts on the different criteria:

client can try version-unaware:

There are two cases to distinguish:

1. The data space has a homogenous version of the DSP. In this case, the DSP spec specifies, how the path is built within a DSP service, so in this case, approach A has only a slight advantage, because a client only has to apply standard DSP rules to get to the necessary endpoint.
2. The data space allows to use multiple versions. In this case, the provided url in approach A has a semantic issue, because the client is unaware, which catalog endpoint version he will call, hence, he has no information on what to expect as result. He has to compare paths provided by version management and some parts of the given url to determine the version. This is heuristic, as he has to deduct the version from path comparision and has no clear information on which version has been provided to him.

approach also works for DataService:

That would be an expectation, that a catalog received with a call in a certain DSP version contains only links that point to endpoints of the same version. If version magic would apply here every time, this does not make sense. Or do I miss here something.

discovery in a non-did based dataspace:

The problem in a non-did based dataspace is, that fundamentally at least one url has to be transfered from the provider to the client. I see no difference in sending a pure host based url that allows to use the .well-known interface or a complete base path to a service instance that would support approach B or C. There is no difference as we have no idea how the information is passed from provider to client. In fact, B or C allows to give again a precise link to the right service, whereas in approach A the client has to guess which of the offerings from the .well-known mechanism is the right one.

fault-tolerance for did-based but non RFC8615 compliant connectors:

I do not see this as a criteria, all approaches require a transfer of an url from provider to client either did or non-did based. It is a question of path magic needed how to come up with the right endpoint.

One criteria is missing that is actually the issue with approach A:

detection of another version of the catalog endpoint of the same DSP service:

This feature cannot be provided by approach A, because it is in general not possible to determine for example the 2024-1 catalog endpoint of the same DSP service that has been provided by the did with the 2025-1 endpoint. This is only possible in specific cases, like only one DSP service available under this host. If there are multiple DSP services available, the current mechanism does not allow to isolate the same DSP service offering of another version, as the path magic needed leads to ambiguous results. If the returned DSP service does not provide a 2024-1 version of the catalog endpoint, the client has no chance to find out if he has to stop the interaction. This is solved by both approach B and C.

So the main disadvantages of approach A are:

1. It is unclear which version of the catalog endpoint is provided with the did document
2. It is in general not possible to find the catalog endpoint of the same DSP service, in only works in specific cases -> This is blocking the adoption of multi-version support
3. The approach requires a lot of path comparisons to deduct version information heuristically instead of having precise version information about the available path.

All of these disadvantages are solved by approach B and C. Both, B and C work with precise paths and version information and only require path additions as defined by the spec to get every relevant endpoint. This is the case for non-did based single version data spaces without any big disadvantage that I see.

[lgblaumeiser]

To express also the issue in a sequence diagram

Approach A - Status Quo

sequenceDiagram
    participant PC as Provider DSP-Server
    participant PW as Provider .well-known
    participant PD as Provider Did Doc
    participant CC as Consumer DSP-Server

    autonumber
    CC ->> PD: get
    PD -->> CC: did doc with service pointing to<br/> a CatalogService endpoint https://<something>/catalog
    CC ->> CC: strip all except <root>
    CC ->> PW: get <root>/.well-known/dspace-version
    PW -->> CC: HTTP 200 with dsp-versions
    CC ->> CC: Do partial path comparisons to find out which version the catalog endpoint is pointing to
    CC ->> PC: In case the right version is sent: POST https://<something>/catalog/request
    CC ->> CC: In case another version is needed: Try to find  a version endpoint in the right version with the biggest overlap,<br/>hope that it is from the same service,<br/>if not, no chance is there to decide that version is not supported.
    CC ->> PC: POST https://<something different>/catalog/request and<br/>deal with all unexpected behavior because the wrong service might be addressed.

Loading

[lgblaumeiser]

And in addition, let's look at an example:

DID Document:

  "id":"some:id",
  "type": "CatalogService",
  "serviceEndpoint": "https://example.com/some/path/v10/catalog"

and the dspace-version endpoint result:

{
  "protocolVersions": [
    {
      "version": "2025-1",
      "path": "/some/path/v10"
    },
    {
      "version": "2024-1",
      "path": "/some/other_path/v5"
    },
    {
      "version": "2025-1",
      "path": "/some/third_path/v10"
    },
    {
      "version": "2024-1",
      "path": "/some/third_path/v5"
    }
  ]
}

Four questions:

• Is this a valid according to the spec?
• Which DSP version is supported by the catalog endpoint returned by the did document?
• Does the same DSP service offer a 2024-1 version of the catalog provided by the did document?
• Which endpoint to call, if version 2024-1 is required?

I have no clue, how to make the answer, to question three and four, and the efforts to answer question two are simply not acceptable imho.

[lgblaumeiser]

Thanks, @arnoweiss , that is a first step to break the blockade, so that it is usable at all. But still the mechanism would be too incovenient to be used in a setup like Catena-X

Therefore I propose two changes in the specification, that would allow a logical freedom for adoptors to optimize discovery and version detection. The freedom is logical because the specification offers freedom concerning the connector discovery anyway but constrain everyone afterwards. This adds inconvenience for some discovery mechanisms. There is nothing gained by constraining this, in my opinion, as the only potential reason, interoperability, is not really leveraged a lot by the common mechanism, if at all.

The proposed changes are:

4.3.2 HTTPS Binding

The Well-Known Version Metadata Endpoint

Each implementation MUST provide the version metadata endpoint at a path that can be statically determined from the path information given by the data space specific discovery mechanism. The version endpoint MUST be unversioned and unauthenticated. The contents of the response is a JSON object defined in section [[[#exposure-of-dataspace-protocol-versions]]].

A recommended mechanism is the use of the dspace-version Well-Known Uniform Resource Identifier [[rfc8615]] at the top of the path hierarchy:

/.well-known/dspace-version

and as a second change proposal

5.4.5 Discovery of Catalog Services

A Participant may publicize their Catalog Service via a DID document, see [[?did-core]]. In that case, the
Participant MUST add an entry to the DID document's service array comparable to the corresponding JSON schema.

Allowed deviations are the use of a different service type or a different semantic concerning the target of path reference, e.g., the implementations base path instead of the catalog service.

• end of proposal

Note: The deviation in the service type would allow to identify the dataspace by the type key. This would support interoperability, because a client can determine by the key whether the catalog service reference is of interest and which further dataspace properties to expect.

[arnoweiss]

That would make discovery of connectors dependent on specifics of a particular Dataspace, requiring an additional type of profile.

[lgblaumeiser]

Yes, that is right, but that is already the case as discovery is in general let to be defined by a dataspace. In 5.4.5, the spec says: A Participant **may** publicize their Catalog Service via a DID document, that is an option. And in the discussion above, there are arguments around using different discovery methods. So the spec gives freedom for discovery but later on it forces everyone into a tight corset. That is contradicting. I doubt, that the community has a common and clear understanding of favorable mechanisms, so that is why I propose to provide freedom until a better alignment can be observed.

The second important argument is, that interoperability is only possible, if a client can find out the semantics of a dataspace. Only then, there is a chance to decide whether the semantics of that dataspace are supported or not. By having a general information like a CatalogService or by general references to a catalog service in the dspace-version endpoint, nothing is ensured. A client has no clue, what to expect from that connector and whether there is a chance for a successful connection. That is not supporting interoperability.

[arnoweiss]

The .well-known is the reliable fallback, the did-doc service is convenience. Removing the fallback as per the proposal would remove a feature from the spec that's critical for cross-version compatibility.

The CatalogService term is defined in the jsonld context. The client's behavior can be derived from the spec and has to be implemented manually anyway. It is defined to end after the Catalog Protocol's /catalog path segment. Regarding auth, there's a proposal that's merged to main already.

[lgblaumeiser]

I do not see, how the current proposals address my concerns in a way that would be acceptable for the Catena-X dataspace. I created three discussions that propose slight changes, that allow us to have the freedom to continue without the need to not comply with the standard. I am still convinced that the whole discovery process in a multi-version, multi-dataspace concept is way to less understood to have a maturity for an ISO standard.

My proposals are:

• Requirements on implementation concerning version support #149
• Adapt requirements in section 5.4.5 Discovery of Catalog Services #150
• Ambiguity in section 4.3 concerning the .well-known/dspace-version endpoint #151