Multi-DSP version support has issues due to use of .well-known and the way, the DID document is specified

[lgblaumeiser]

The issue I want to discuss here is, that in section 4.3.2 the HTTPS Binding for the exposure of supported version that are provided by a connector is specified, which makes use of the Well-Known Uniform Resource Identifier defined in RFC8615 (https://www.rfc-editor.org/rfc/rfc8615). The surprising fact of this feature is, that the RFC specifies that the /.well-known/ endpoint must at the top of the path hierarchy. What does this mean, the endpoint is only in special cases part of the connector service, in general it is an own resource that needs to be managed separately.

The DSP specifies in section 4.3.2 the endpoint “/.well-known/dspace-version”, which returns according to section 4.3.1 an array of mapping information between a “version” and a subpath beginning from the top of the endpoint hierarchy to the base path of the connector in that version, i.e., if a connector supports two versions 2014-1 and 2015-1 the array would contain two completely independent entries with key 2014-1 mapping to the path where all DSP endpoints according to that version are provided and with key 2015-1 the same for that version.

Let’s assume, a provider operates multiple connectors, but under the same domain, e.g.,

• Connector 1: https://provider.com/connectors/root
• Connector 2: https://provider.com/connectors/pcf
• Connector 3: https://provider.com/connectors/dcm

This would require an endpoint https://provider.com/.well-known/dspace-version which returns an array, that contains redundant keys for each version supported by any of the connectors, e.g., “2014-1” and a path like https://provider.com/connectors/pcf/api/2024-1/ and the same in future for version “2015-1” with a path like https://provider.com/connectors/pcf/api/2025-1/.

A big disadvantage is, that the supported version information cannot be managed by the connector service runtime itself, although it would be simple to do so. Instead, the operator of the connectors has to manage all version paths separately and provide a mechanism to offer the https://provider.com/.well-known/dspace-version endpoint as additional service.

The even bigger issue comes with section 5.4.5 of the DSP spec handling the Discovery of Catalog Services. In this section, it is specified, that a catalog service endpoint of a provider can be discovered by providing the information in the DID document of the provider in the service section as service of type “CatalogService”. There it is specified, that the reference would point to something like in the example above: https://provider.com/connectors/root/api/2015-1/catalog.

The problem now is that it is not clear, which version this catalog request is implementing. What a consumer has to do is, that he has to navigate to https://provider.com/.well-known/dspace-version, retrieve the version information and compare the truncated path “/connectors/root/api/2015-1/catalog” to the provided path “/connectors/root/api/2015-1” to identify the version by the key of the matching subpath.
If only the 2015-1 version is added in the DID document, the detection of “CatalogService” endpoint of, e.g., version 2014-1 is in general impossible. That is because starting from the information from the DID document namely the URI https://provider.com/connectors/root/api/2015-1/catalog. Based on that, the https://provider.com/.well-known/dspace-version has to be navigated to and based on the version information given from this endpoint, the search now is to find the entry in the array, with the right version key, in this case “2014-1” which has the largest intersection with the catalog endpoint found in the DID document. In the example we would have the following entries:

a. 2014-1 : https://provider.com/connectors/root/api/2014-1
b. 2014-1 : https://provider.com/connectors/pcf/api/2014-1
c. 2014-1 : https://provider.com/connectors/dcm/api/2014-1

The algorithm would find based on https://provider.com/connectors/root/api/2015-1/catalog that the largest intersection is with option a, so “hopefully” the root catalog information for version 2014-1 can be retrieved from https://provider.com/connectors/root/api/2014-1/catalog. If, in the above example a and b are not there, only connector c supports version 2014-1, the algorithm would detect option c as the 2014-1 version of the root catalog, which would be wrong. The issue is, that semantically, there is nothing that allows a consumer to distinguish between a part segment that is used for routing towards the connector from a path segment, that is part of the service endpoint hierarchy.

Actually, in my opinion, the two mechanisms do not interact well and therefore I propose the following changes for further discussion.

• Using the .well-known RFC is actually not adding value to the DSP spec, as the versions supported by a connector is service-local, not a general property of the domain in which the connector is operated. Therefore, in section 4.3.2, there should be a mechanism that allows to get the version information relative to the base path of the connector. E.g., from the examples above, this could be something like "https://provider.com/connectors/root/api/metadata". This would be much easier to handle for an operator without any disadvantage.

• The DID Document spec in section 5.4.5 should not give the path to the catalog endpoint, but should provide the base path of the overall connector. This change has two advantages:

  • The handling of the path is purely additive, there is no need to navigate up in a given path. All paths can be build by following the rules of the DSP spec and add the corresponding segments to the path.
    Navigating upwards always has the issue, that it is not clear, when the navigation leaves the context of the current service and gets into path segments that are used to manage the service in a domain.
  • The catalog endpoint is already version specific, which makes the path not directly usable, as it is unclear, which version of the supported DSP protocol is to be used, so a navigation to the version information is necessary anyway. It would also motivate to move the handling of versions to the DID document by providing somehow version tagged CatalogService objects there.

  It is then an additional discussion whether in this case the DID document should contain a CatalogService object or something different, e.g., a "Connector".

[lgblaumeiser]

Hi @arnoweiss I have added my thoughts on the versioning issue here, feel free to add the content we created during our discussion earlier.

[arnoweiss]

Shouldn't the a,b,c list above also contain
d. 2015-1 : https://provider.com/connectors/root/api/2015-1

I think we always need to make the assumption that the version data endpoint (wherever it may be) is aware of all endpoints and their versions.

[arnoweiss]

Please let me know if this fits your understanding of the situation.

Current state of the spec

sequenceDiagram
    participant PC as Provider DSP-Server
        participant PW as Provider .well-known

    participant PD as Provider Did Doc
    participant CC as Consumer DSP-Server

    autonumber
    CC ->> PD: get
    PD -->> CC: DSP-endpoint holding <root>/path/catalog
    CC ->> CC: isolate <root>
    CC ->> PW: get <root>/.well-known/dspace-version
    PW -->> CC: HTTP 200 with path-segs & version
    CC ->> CC: strip URL-segments,<br/>choose endpoint
    CC ->> PC: POST <root>/path-segs/catalog/request

Loading

Your suggestion

sequenceDiagram
    participant PC as Provider DSP-server
    participant PD as Provider Did Doc

    participant CC as Consumer DSP-server

    autonumber
    CC ->> PD: get
    PD -->> CC: did doc with service<br/>"DspMetadata" holding<br/> <root>/path/metadata
    CC ->> PC: get <root>/path/metadata
    PC -->> CC: HTTP 200 with path-segs and versions
    CC ->> CC: choose endpoint
    CC ->> PC: POST <root>/path-segs/catalog/request

Loading

[lgblaumeiser]

Close, the proposal is, to return in step 2 /path, so that the consumer has a base path that allows him to reach all relevant endpoints by adding specified (by DSP spec) path segments

[arnoweiss]

sequenceDiagram
    participant PC as Provider DSP-server
    participant PD as Provider Did Doc

    participant CC as Consumer DSP-server

    autonumber
    CC ->> PD: get
    PD -->> CC: did doc with service pointing to<br/> <root>/path
    CC ->> PC: get <root>/path/metadata
    PC -->> CC: HTTP 200 with path-segs and versions
    CC ->> CC: choose endpoint
    CC ->> PC: POST <root>/path-segs/catalog/request

Loading

[jimmarino]

The issue I want to discuss here is, that in section 4.3.2 the HTTPS Binding for the exposure of supported version that are provided by a connector is specified, which makes use of the Well-Known Uniform Resource Identifier defined in RFC8615 (https://www.rfc-editor.org/rfc/rfc8615). The surprising fact of this feature is, that the RFC specifies that the /.well-known/ endpoint must at the top of the path hierarchy. What does this mean, the endpoint is only in special cases part of the connector service, in general it is an own resource that needs to be managed separately.

The DSP specifies in section 4.3.2 the endpoint “/.well-known/dspace-version”, which returns according to section 4.3.1 an array of mapping information between a “version” and a subpath beginning from the top of the endpoint hierarchy to the base path of the connector in that version, i.e., if a connector supports two versions 2014-1 and 2015-1 the array would contain two completely independent entries with key 2014-1 mapping to the path where all DSP endpoints according to that version are provided and with key 2015-1 the same for that version.

Let’s assume, a provider operates multiple connectors, but under the same domain, e.g.,

• Connector 1: https://provider.com/connectors/root
• Connector 2: https://provider.com/connectors/pcf
• Connector 3: https://provider.com/connectors/dcm

This can be handled very simply by:

https://root.provider.com/connector
https://pcf.provider.com/connector
https://dcm.provider.com/connector

Also, assume a single case:

https://provider.com/connectors

It is trivial to deploy a forwarding entry in a load balancer (e.g. NGINX) for:

https://provider.com/.well-known/...

to the backing connector or serve a static file.

There is a reason why well-known is based on the root path; otherwise, discoverability is not possible given indeterminate path segments. On a related note, I don't think the "connector" address should be in the DID service entry as it is the catalog server, not the connector, that is the entry-point for discovering contract negotiation endpoints (which may be different).

[lgblaumeiser]

You basically confirm all the issues that there are with the current specification.

Sorry, I don't follow.

1. Yes, you can define a subdomain for every connector, but this is simply an unnecessary constraint, you limit the freedom of an operator for no reason at all, that is bad style.

That is only one option, and I disagree that it is bad design. There is no reason why other approaches will not work. That does not mean all approaches must work. A specification is about choices and trade-offs. I don't understand what is limiting about this. You can still use subpaths for different uses cases. What you can't do is use subpaths and have completely different versions per sub-path. Since this case can be handled by other approaches, I don't see this as an argument to go against well-established internet standards and tie DSP to DIDs, which it currently is not.

The design decision you took, simply limits options, without adding any value. You did a trade-off that has no advantage but disadvantages, that is the point here. Handling multiple DSP services which support multiple versions within the same domain name becomes error prone and tedious. If there is no willingness to even see the issue, we have to discuss within Catena-X what we make out of this. Whether we can live with the restriction introduced for no reason or have to deviate from the protocol, to regain the freedom in operations.

2. I do not agree on your assumption that there is a good reason for the well-known to be based on the root level. If your domain provides multiple services the .well-known resources should only provide properties of the whole domain, or of a single service within that domain, but not an aggregated state of several services. That is conceptually difficult. For me, it is clear, that the idea behind the .well-known resources is not a match to what you want to express conceptually for supporting multiple versions within one service. That is a service property, not a domain property. Btw., interestingly, you already came over that idea in the DCP, as Arno showed me that there is in RC2 a metadata endpoing to provide metadata on the service.

I think you are confusing domains and subdomains and also not fully appreciating how .well-known works. ./well-known can provide information for different services by registering and using Well-Known URIs.

The metadata API in DCP is something entirely different. It is not conveying information about the protocol but about the type of credentials offered by an issuer. As such, it must be versioned, supports Json-Ld, and may change based on the protocol version. These two things should not be conflated.

I do not get, how you come to the conclusion that I am condusing domain and subdomain. I have read the RFC and it is pretty clear from there, that the way you are using it to define properties of a single service within the given domain is not a standard use case for this mechanism. You move a local property to something global, and again I do not see any advantage in doing so.

3. Yes, it is a catalog service you want to provide in the DID document. But who says, that a base path of a DSP service provides more then the catalog service. Providing the catalog endpoint is conceptually not working because it has version specific parts in the path. This requires either that you handle versions in the DID document instead of a metadata/.well-known mechanism, or that you have to do backward navigation on an url, which is a wrong thing to do, as this brings you to areas unknown. A path segment in an url is semantically not uniquely defined. Therefore, I would strongly recommend to only use additive paths starting from a base path, because that is uncritical from a semantic perspective. That requires to only use the base path of the DSP service in the DID document, every other path does not fulfill the requirement of semantic uniqueness. An alternative would be to use something like https:///catalog/2025-1, but this would require much more changes in the specification.

Sorry, I don't follow the argument here. Using ./well-known is quite simple to do in the context of a service entry.

Also, what you are suggesting is to require DSP to depend on DIDs, which we should not do. As it currently stands, The Catalog Service entry is optional and protocol discovery can be done using web technologies (./well-known). Placing discovery information in a DID upends that approach.

That is absolutely not my point. The issue is the difference between:

"service":[
  {
    "id":"some:id",`
    "type": "CatalogService",
    "serviceEndpoint": "https://bar.example.com/api/2025-1/catalog"
  }
]

and

"service":[
  {
    "id":"some:id",`
    "type": "CatalogService",
    "serviceEndpoint": "https://bar.example.com/"
  }
]

The difference is, that with the second alternative I have only an additive usage of the url, because all I need to do is to follow the DSP spec to get the version information from the .well-known/dspace-version add it to the provided url, add in addition catalog and have the right endpoint in the right version

With the first alternative, I have to use the .well-known information to deduct which version I am in, by comparing path segments and identify which well known path is completely in the intersection of the two paths. Then I can deduct the version of the catalog service. If it is not the right version, I have to do some more path magic to find out an catalog endpoint in the right version without any assurance that it is the same DSP service as the one given from the DID document.

I propose that instead of defending the current spec by all means you try to get into the points I raised, there is simply a bug in the spec because the two mechanism do not play together properly, and either we fix it downstream or we have a proper discussion to improve the spec.

[jimmarino]

Sorry, but this comment is not appropriate:

I propose that instead of defending the current spec by all means you try to get into the points I raised, there is simply a bug in the spec because the two mechanism do not play together properly, and either we fix it downstream or we have a proper discussion to improve the spec.

I am trying to explain how ./well-known can operate with various URL forms and the intent behind why the specification is designed the way it is. I am also having difficulty following what you are proposing. Since the above comment is completely out-of-place in a specification discussion, I would suggest starting over in a separate discussion with a succinct problem statement, solution proposal, and a focus on technical argumentation.

[lgblaumeiser]

Sorry, but this comment is not appropriate:

I propose that instead of defending the current spec by all means you try to get into the points I raised, there is simply a bug in the spec because the two mechanism do not play together properly, and either we fix it downstream or we have a proper discussion to improve the spec.

I am trying to explain how ./well-known can operate with various URL forms and the intent behind why the specification is designed the way it is. I am also having difficulty following what you are proposing. Since the above comment is completely out-of-place in a specification discussion, I would suggest starting over in a separate discussion with a succinct problem statement, solution proposal, and a focus on technical argumentation.

That comment was a result of me getting the impression that the way you react on my problem description was inappropriate in the first place. And no, I do not see a reason to start again, because I spent quite some effort in expressing the problem statement and I see no benefit in stating it again.

If I could not get my point clear, I am happy to answer to questions you might have due to unclarity, but the issue is as described, the two sections 4.3.2 and 5.4.5 do not play together well together which results in a real bug in a general setup. This can only be solved by restricting the operation requirements to a limitation that is unnecessary and actually not acceptable.

There is now another proposal by SAP from Axel in this discussion as well, so there are two clear proposals to improve the spec, but if that is not welcome at all, we have to live with a buggy spec that we have to fix for a real world adoption.

[jimmarino]

My initial response was to point out that the ./well-known specification can handle the use case you highlighted and to push back on the notion that the specification needs to be changed by presenting technical arguments for why that is the case. I also stated that I did not understand why you are claiming that it is not the case, and what your specific solution is. That is an entirely appropriate response and part of my role as a committer.

Based on conversations with others, I am not alone in having difficulty understanding why the current specification is deficient. What is inappropriate, however, is the claim that I am asking questions because I am only interested in "defending the current spec by all means."

I am aware of Axel's proposal and have discussed it with his colleague. It, however, presents interoperability issues, which we also went over. If you want to make progress, I would suggest sticking with technical facts and not impugning commiters' motives. I am open to continuing the conversation in another discussion centered on a succinct description of the problem, but will not do so here.

[arnoweiss]

Feedback is welcome. The committers will consider all proposals.

[axelalbrechtatsap]

IMHO we must:

• allow a consumer to find an counterpart endpoint for a selected version
• support state-of-the-art deployment models, including hosting multiple connectors under the same host
• avoid any "search and cut URL paths" activities

my team and I propose:

1. the DID document should contain the version endpoint, not (only) to the catalog endpoint.

• For deployment models where only a single connector is deployed on one host, this can be the .well-known endpoint.
• For deployment models using suffix, a non-RFC compliant endpoint can be provided, as long as it contains the version information for this specific connector.
  From there each consumer can pick the right version base path and access the catalog.

2. Add to DSP spec point 4.3.2.1 a sentence like “Behind the version specific endpoint, which is mentioned in the .well-known endpoint, you can expect a catalog endpoint /catalog/request, but you cannot expect a negotiation endpoint”

[arnoweiss]

So that would be

sequenceDiagram
    participant PC as Provider DSP-server
    participant PD as Provider Did Doc
    participant CC as Consumer DSP-server

    autonumber
    CC ->> PD: get
    PD -->> CC: did doc with service pointing to<br/> a DspMetadata endpoint at<br/> <root>/.well-known/dspace-version
    CC ->> PC: get <root>/.well-known/dspace-version
    PC -->> CC: HTTP 200 with path-segs and versions
    CC ->> CC: choose endpoint
    CC ->> PC: POST <root>/path-segs/catalog/request

Loading

[arnoweiss]

As this discussion is a little hard to stay on-top-of, I've tried to compile a tl;dr [1]. Let's focus on what's possible or not possible technologically there.

[1] #144