Make Python magic optional

[SylvainCorlay]

It would be great if the Python magic was a plugin, so that we don't need CPython to build xeus-clang.

A byproduct of requiring CPython is that conda-forge needs to make variants of the build for all supported versions of Python.

[vgvassilev]

@SylvainCorlay thanks for the suggestion! Can you elaborate a little more?

cc: @ioanaif

[SylvainCorlay]

I think that making CPython a dependency, and baking in the Python magic complicates packaging signicantly. I would rather like a very minimal (yet extensible) C++ kernel.

• packages that have a python dependency on conda have to be built for each flavor of Python supported. If we make the Python magic an extension, only the python magic needs to be compiled for all versions.
• it will make packaging for the frontend (e.g. emscripten-forge) much easier if we drop the CPython dependency.

[vgvassilev]

Do you have an example where we can learn and package it in a similar way?

[SylvainCorlay]

Do you have an example where we can learn and package it in a similar way?

The way I would go about it is the following:

• Each clang-repl plugin provides a shared object (dynamic library) installed in the standard location (PREFIX/lib most typically) providing a standardized API, for example a free function load() to be called upon starting the plugin.
• Each plugin gets installed with a "manifest" in PREFIX/share/clang-repl/plugins.d in the form of a JSON file providing information about the plugin (the name of the dynamic library to be loaded, and metadata such as a description).

void* handle = dlopen("plugin.so");
if (handle != NULL)
{
    func* load = dlsym(handle, "load");
    if (load != NULL)
    {
        int ret = load();
        return 0;
    }
}
return 1;

This can obviously be a lot more advanced. Plugins can provide more functions, be listed by listing the items in the plugins.d directory. Installing a plugin is done by placing files in the installation prefix, so that it can easily be done via a package manager.

[JohanMabille]

So basically you need a tiny library for abstracting the way you dynamically load libraries. Things to decide before starting to design one (or using an laready existing one such as https://github.com/gazebosim/gz-plugin):

• do we want different categories of plugins and be able to distinguish them? If so, that means we want to be able to provide different interfaces (more or less rich). Otherwise we can provide a super simple one with a single method, but the xeus kernel library needs to provide a very broad API.
• do we want to handle plugin dependencies? I think it would be a nice feature, but that would make the implementation a bit more complex
• do we want to have the notion of plugin priority: many plugins could implement the same feature, and if so we may want to specify which one we want to load. A solution can be to prepend the manifest with a number defining the priority (the lower number, the higher priority, similar to udev ruls for instance).