Add support for IncludeWorkflow operators

[banchan86]

Motivation

The IncludeWorkflow operator is an easy way to create new Bonsai operators and reuse existing code. They are used extensively in some packages like Bonvision and Bonsai.ML. However these operators are essentially blind to docfx and are not automatically documented in the Reference section.

Background

The standard dotnet docfx command actually runs two commands in its pipeline:

dotnet docfx metadata  

This command generates ManagedReference files in .yml format from .NET source code. In addition, it creates a .manifest file which is used in docfx build, as well as namespace.yml files for the namespace API list and toc.yml for the Table of Contents. All of these are placed in the dest folder defined in docfx.json (api/)

dotnet docfx build  

This command generates .html files from the .yml files in api/ and places them in the output folder defined in docfx.json (_site).

Proposed Solution

One possible solution would be a patch to perform the role of the dotnet docfx metadata in creating .yml files but for .bonsai files. In addition, it would modify .manifest, toc.yml and namespace.yml to include these operators.

Such a patch would come in the docfx pipeline like this:

dotnet docfx metadata      
python bonsai/template/api/plugins/Patch-IncludeWorkflow.py 
dotnet docfx build         

An advantage of this solution is that because it is injecting into the existing pipeline for C# operators, it can make use of any future improvements we make (such as the proposed ManagedReference based API template #9).

Proposed Implementation

1. Parse .bonsai files (essentially XML files) and extract the operator description, ExternalizedMapping properties and their description (if available), and exclude PropertyMapped properties which are hidden.

2. If there are embedded C# operators that are part of the package repository, parse the .cs files in the src folder for more information.

3. If there are other embedded IncludeWorkflow operators, parse them.

4. If the properties come from operators that are not part of the package repository, but are instead part of the package dependencies, parse the XML doc file that is included with the dependency in the .bonsai local environment.

Potential problems

For embedded operators that come from a dependency, there is limited information in the XML doc file (and sometimes, no accompanying XML doc file). That might complicate the search for property types and inputs/outputs.

Alternative Solutions Considered

Dummy CS placeholders
One early hack that I explored, was to create dummy .cs files with just the bare minimum information (namespace, class declaration) to be recognized by the docfx metadata command, and then use the docfx overwrite function during docfx build to include supplementary markdown articles that would appear on the API page. That works, but the overwrite articles still need to be individually created, and would be in a different format than the other operators.

Docfx build plugin
Another option I explored was Docfx's support for document processors and custom plugins, which run during the docfx build process. I made some progress on a plugin to generate .yml files from .bonsai files but I abandoned this effort for several reasons:

• The plugin runs during the docfx build process, which I am not sure is the right way place to generate .yml ManagedReference files. I found myself having to run docfx build twice (once to generate .yml files from .bonsai files, and then .html files from those new .yml files).
• I did not find any classes that would have helped to generate ManagedReference files and ended up having to implement the logic from scratch.
• With no C# background it became exceedingly hard and complicated for me (even with ChatGPT :P)

Using existing IncludeWorkflow associated classes and methods in the Bonsai core library
I'm sure there are easier ways of doing the patch using existing classes and methods included in the Bonsai Core library, but I am not sure where to start and how to go about doing it :P

[glopesdev]

@banchan86 Thanks for the write-up, very useful demo to setup a custom docfx processor plugin. I did get to a similar place where you were, but realizing now that to resolve all the edge cases (including property type determination), we do need to load the entire environment, just as we currently do when exporting workflow images.

This makes me wonder that an easier place to implement this feature might be to have another CLI command in the editor which exports all package docs metadata, i.e. operator icon and docs, etc. for all operators in a specified package, including embedded workflow resources. Maybe a cleaner approach could be taken, but at this moment the editor is really the place where everything comes together. Even YamlDotNet is already an installed package so we could easily export a structure into YAML.