Fully repeatable, consistent (deterministic?) value IDs with SKOS import

[ekansa]

The SKOS import does a good job of allowing repeated import of UUIDs (based on the URIs of controlled list items) created in Arches.

It would be very useful to have the same repeatability for the UUIDs assigned to prefLabels and other values associated with controlled list items. A deterministically generated UUID for prefLabels and other values may be very useful to promote interoperability between multiple Arches instances that import the same SKOS file. It would also help with ETL workflows, since these value_ids often need to be referenced.

The UUID5 generator may be a good way to go about it. To deterministically generate UUIDs for list item values, the namespace argument can be given from the list item URI, and the name argument can be something like f'{valuetype_id}:{value}'

[johnatawnclementawn]

Thanks for this write-up, @ekansa - there are a few challenges here that I want your take on.

One risk with generating a UUIDv5 from the URI and value is creating duplicate ids for items that have multiple positions in a hierarchy. Arches Controlled Lists diverges from the RDM in that for each occurrence of a concept in the RDF graph, a distinct list item record is created along with distinct list item value records for each of the concept's labels & notes. Generating a UUIDv5 from that concept's URI and the value would result in duplicate list_item_value_ids. We could try to generate a UUIDv5 for a list item and then regenerate part of the UUID (like we do for certain nodes in the graph designer), and then use that list_item_id instead of the valuetype_id as you suggest above (f'{list_item_id}:{value}'), but this would still create differing ids that you'd need to contend with during a resource ETL.

If the paint point is needing to look up list_item_ids and list_item_value_ids, then, as @robgaston mentioned during standup, we ought to think about what pieces of information are stored in tiles for the reference datatype - certainly removing id from the labels array is doable, but there are challenges for removing list_item_id - e.g. how do we remove list_item_id from the labels array without losing the context of where in the list hierarchy that item came from (if the URI occurs n-times in a list). We could also think about removing the list_id and deriving that from the node config.

Moving from

[
    {
        "uri": "http://localhost:8000/plugins/controlled-list-manager/item/6b87a748-0d38-4100-933b-c3a6feb7aa0e",
        "labels": [
            {
                "id": "aa141395-3e95-4574-ab21-9a91127e7dab",
                "value": "NFPA Flammability - 0",
                "language_id": "en",
                "list_item_id": "6b87a748-0d38-4100-933b-c3a6feb7aa0e",
                "valuetype_id": "prefLabel"
            }
        ],
        "list_id": "66fe06b4-9a61-4ee0-82cd-d33866e2be0c"
    }
]

to this:

[
    {
        "uri": "http://localhost:8000/plugins/controlled-list-manager/item/6b87a748-0d38-4100-933b-c3a6feb7aa0e",
        "labels": [
            {
                "value": "NFPA Flammability - 0",
                "language_id": "en",
                "valuetype_id": "prefLabel"
            }
        ],
    }
]

[ekansa]

@johnatawnclementawn I agree that that one would greatly improve / simplify ETL workflows with the objects that you proposed:

[
    {
        "uri": "http://localhost:8000/plugins/controlled-list-manager/item/6b87a748-0d38-4100-933b-c3a6feb7aa0e",
        "labels": [
            {
                "value": "NFPA Flammability - 0",
                "language_id": "en",
                "valuetype_id": "prefLabel"
            }
        ],
    }
]

That would reduce the need for one to care about the internal UUIDs managed by Arches for prefLabels, etc. So I would love to see this implemented for the controlled lists and the relational views (cc @robgaston ).

At the same time, I still think it's worth exploring deterministic UUID generation for the values associated with controlled list items. I think your suggestion (quoted below) sounds reasonable:

We could try to generate a UUIDv5 for a list item and then regenerate part of the UUID (like we do for certain nodes in the graph designer), and then use that list_item_id instead of the valuetype_id as you suggest above (f'{list_item_id}:{value}'), but this would still create differing ids that you'd need to contend with during a resource ETL.

While yes, this would still make different IDs, at least these IDs would be deterministic and stable. Once I know these IDs, I can reuse them. It would also make it easier and more predictable to exchange controlled list data between different Arches instances.

Also, we can do both. Have some more deterministic UUIDs for controlled list value data AND simplify the representation of controlled list items (with value ids omitted) for ETL processes as you proposed.

[johnatawnclementawn]

Totally agree with you that this doesn't have to be an Either-Or situation