Add the ability to import skeletons

[lochhh]

For SLEAP we could have something like load_skeleton.from_sleap_json()

Originally posted by @niksirbi in #88 (comment)

[niksirbi]

Skan is a pure Python library for skeleton analysis, may be relevant/useful for us some day. Shout out to @vigji for bringing it to my attention.

They even have a guide on displaying 3D skeletons in napari.

[vigji]

Out of curiosity, how come you don't consider anywhere in the current implementation of the movement data the skeleton?

For me it is important enough to deserve a dedicated entry in the pose xarray, what do you think? I could not track down a discussion on this, sorry if I missed it.

[niksirbi]

I agree that skeletons are important; we had just neglected them when we first designed the data structure.

I guess there are two questions to resolve:

Skeleton Representation

• A minimal approach would be to have a dictionary with the reserved name "skeleton" added to the dataset's attributes. The dictionary can map nodes to other nodes, i.e., represent the edges.
• Another option is to dedicate a special DataArray to the skeleton. This would be a boolean array of shape (n_keypoints, n_keypoints), with True where there is an edge (though this adds some redundancy, as the matrix would be symmetric).
• A completely separate object corresponding to a graph, for example adopting the Skeleton representations of sleap-io or Skan. A "hybrid" approach could be to serialise this object to the attrs dictionary attached to the xarray Dataset.

Were you thinking of one of the above, or something else?

In any case, the skeleton should be optional, in my opinion, because:

• They are not necessary for many analyses.
• They cannot be defined for some tracking modalities (centroid, bbox, or segmentation mask tracking).

Loading Skeletons

I haven't given much thought to this, but I'll share some ideas off the top of my head.

• Add an argument to existing loader functions, for example:

  load_poses.from_dlc_file(file_path: str | Path, fps: float | None, skeleton_file: str | Path | None)

  Here, the skeleton would be parsed from a file and attached to the poses dataset in one step.
• Create completely separate functions for loading or creating skeletons.

  skeleton = load_skeleton(file_path: str | Path, format: str = "sleap_json")
  skeleton = create_skeleton(skeleton_dict: dict, name: str)

  Here, "attaching" the skeleton to a poses dataset would be handled separately from its loading or creation.

Ultimately, "how to load skeletons" will depend on how we choose to represent them, so the two questions are interconnected.

[vigji]

I think 0 sofistication is required here, and the minimal representation of list of edges is by far the most convenient one. Would not add the burden of dealing with a separate library dedicated to them, as I can't see the need for any fancy topological optimisation on skeleton operations, and everyone understands easily a list of nodes.

Fully agree on not having it mandatory; I would indeed just add the option to have it passed as an argument, either row (list of lists) or via a config file (eg for DLC) from which it could be read. slp files should already contain it

What do you think?

[niksirbi]

I'm with you on keeping it simple.
I like the list-of-edges representation, as this is exactly how most users define skeletons in DLC or SLEAP.

So an example skeleton could be:

skeleton = [
	("snout", "left_ear"),
	("snout", "right_ear"),
    ("snout", "neck"),
    ("neck", "centroid"),
    ("centroid", "tail_base"),
    ("tail_base", "tail_end"),
]

The above shows list-of-tuples, but list-of-lists should also be acceptable.

We could have a function named add_skeleton(ds: xarray.Dataset, edges: list[tuple] | list[list]). That function would validate the edges (as in, ensure the nodes really exist as keypoints in ds), and then call ds.attrs["skeleton"] = validated_edges.

The user could directly overwrite that attribute at any point, at their own risk.

In load_poses.from_file() functions, I'm thinking of adding a "skeleton_filepath" argument pointing to where the skeleton is stored, which is usually not the same as the file where predicted poses are stored (for DLC it's the config file, for sleap the skeleton can be separately saved as a .json, or read directly from the .slp).

[sfmig]

We think the main use cases for the skeleton data would be:

• visualisation
• compute length of segments
• compute angles between segments