Schema refinements

[caufieldjh]

Needed to make some schema changes to get this to work: more classes needed id fields and needed to define their category with the designates_type flag.
Will resolve RDF serialization in future.

[caufieldjh]

Doesn't work yet - getting this error when testing on a single yaml entry:

~/kg-registry/resource$ linkml-convert -s ../src/kg_registry/kg_registry_schema/schema/kg_registry_schema.yaml -C Resource -f yml -t ttl -o spoke.ttl spoke.yml
Traceback (most recent call last):
  File "/home/harry/kg-registry/.venv/bin/linkml-convert", line 8, in <module>
    sys.exit(cli())
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml/utils/converter.py", line 164, in cli
    obj = loader.load(source=input, target_class=py_target_class, **inargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml_runtime/loaders/loader_root.py", line 76, in load
    results = self.load_any(*args, **kwargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml_runtime/loaders/yaml_loader.py", line 42, in load_any
    return self._construct_target_class(data_as_dict, target_class)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml_runtime/loaders/loader_root.py", line 137, in _construct_target_class
    return target_class(**data_as_dict)
  File "<string>", line 29, in __init__
  File "test", line 338, in __post_init__
  File "test", line 237, in __post_init__
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml_runtime/utils/yamlutils.py", line 105, in _normalize_inlined_as_list
    self._normalize_inlined(slot_name, slot_type, key_name, keyed, True)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml_runtime/utils/yamlutils.py", line 190, in _normalize_inlined
    cooked_obj = slot_type(**as_dict(list_entry))
  File "<string>", line 9, in __init__
  File "test", line 830, in __post_init__
  File "test", line 771, in __post_init__
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/jsonasobj2/_jsonobj.py", line 160, in __getattr__
    return super().__getattribute__(item)
AttributeError: 'Individual' object has no attribute 'unknown_category'

[caufieldjh]

Using the pydantic classes yields a more informative error:

~/kg-registry/resource$ linkml-convert -m ../src/kg_registry/kg_registry_schema/datamodel/kg_registry_schema.py -C Resource -f yml -t ttl -o spoke.ttl spoke.yml
Traceback (most recent call last):
...
pydantic_core._pydantic_core.ValidationError: 1 validation error for Resource
category
  Input should be 'Resource' [type=literal_error, input_value='KnowledgeGraph', input_type=extended_str]
    For further information visit https://errors.pydantic.dev/2.10/v/literal_error

KnowledgeGraph is_a Resource - is the converter just not aware of that?

[sierra-moxon]

as an extra test, can you run the gen-linkml generator on your original schema, then with the output of that, generate the pydantic classes? that would help me debug. gen-linkml does a lot of inference that may or may not be taking advantage of in linkml-convert.

[caufieldjh]

OK - if I use gen-linkml like so:

gen-linkml -o kg_registry_schema_gen.yaml -f yaml ../src/kg_registry/kg_registry_schema/schema/kg_registry_schema.yaml 

and then make Pydantic like so:

gen-pydantic kg_registry_schema_gen.yaml > kg_registry_schema_gen.py

Then if I run the linkml-convert alone with just the schema, I get this:

$ linkml-convert -s kg_registry_schema_gen.yaml -C Resource -f yml -t ttl -o spoke.ttl spoke.yml
Traceback (most recent call last):
  File "/home/harry/kg-registry/.venv/bin/linkml-convert", line 8, in <module>
    sys.exit(cli())
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml/utils/converter.py", line 164, in cli
    obj = loader.load(source=input, target_class=py_target_class, **inargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml_runtime/loaders/loader_root.py", line 76, in load
    results = self.load_any(*args, **kwargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml_runtime/loaders/yaml_loader.py", line 42, in load_any
    return self._construct_target_class(data_as_dict, target_class)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml_runtime/loaders/loader_root.py", line 137, in _construct_target_class
    return target_class(**data_as_dict)
  File "<string>", line 23, in __init__
  File "test", line 417, in __post_init__
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/jsonasobj2/_jsonobj.py", line 160, in __getattr__
    return super().__getattribute__(item)
AttributeError: 'Resource' object has no attribute 'unknown_category'

and if I use the pydantic, I get this:

$ linkml-convert -m kg_registry_schema_gen.py -C Resource -f yml -t ttl -o spoke.ttl -s kg_registry_schema_gen.yaml spoke.yml
Traceback (most recent call last):
  File "/home/harry/kg-registry/.venv/bin/linkml-convert", line 8, in <module>
    sys.exit(cli())
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml/utils/converter.py", line 172, in cli
    validation.validate_object(obj, schema)
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/linkml/utils/validation.py", line 43, in validate_object
    top_class=target_class.class_name,
  File "/home/harry/kg-registry/.venv/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 264, in __getattr__
    raise AttributeError(item)
AttributeError: class_name

[cmungall]

pydantic isn't expected to work this way as it relies on conventions for the data classes

(we should have an option for convert to bypass the schema/python - not necessary for most transforms!)

I would first try simpler conversions, e.g json to json

[caufieldjh]

I think this is the core of the issue here, better summarized by running linkml-validate on CLI:

$ linkml-validate -s src/kg_registry/kg_registry_schema/schema/kg_registry_schema.yaml -C Resource resource/spoke.yml
[ERROR] [resource/spoke.yml/0] 'KnowledgeGraph' is not one of ['Resource'] in /category

My naive assumption is that the validator should know that every KnowledgeGraph is a Resource.
But all it knows is that this is supposed to be an object with the category name "Resource", so any string other than that is invalid.

The linkml docs (https://linkml.io/linkml/schemas/type-designators.html) say that transitive subclasses should be OK, and the schema says KnowledgeGraph is_a Resource, so when the error says 'KnowledgeGraph' is not one of ['Resource'] in /category where is it getting that list?

[caufieldjh]

Similarly, if I set the category in the test data to Resource and try to validate it as a NamedThing, I get the error:

$ linkml-validate -s src/kg_registry/kg_registry_schema/schema/kg_registry_schema.yaml -C NamedThing resource/spoke.yml
[ERROR] [resource/spoke.yml/0] Additional properties are not allowed ('activity_status', 'description', 'domain', 'layout', 'name', 'repository', 'url' were unexpected) in /
[ERROR] [resource/spoke.yml/0] 'Resource' is not one of ['NamedThing'] in /category

[caufieldjh]

This is getting messy, so I'm going to hold off on trying to make linkml-convert work in the exact way I think it should, and instead keep the schema updates in this PR because they'll be helpful in the future.