jupyter: kernelspecs library outputs incomplete list

[williamstein]

On my Dev machine (Ubuntu 22.04), the output of the kernelspecs package from @rgbkrk is incomplete:

Compare my machine:

~/cocalc/src$ jupyter kernelspec list
Available kernels:
  python3           /usr/local/lib/python3.10/dist-packages/ipykernel/resources
  bash              /usr/local/share/jupyter/kernels/bash
  ir                /usr/local/share/jupyter/kernels/ir
  ir-sage           /usr/local/share/jupyter/kernels/ir-sage
  javascript        /usr/local/share/jupyter/kernels/javascript
  julia-1.9         /usr/local/share/jupyter/kernels/julia-1.9
  octave            /usr/local/share/jupyter/kernels/octave
  python3-sage      /usr/local/share/jupyter/kernels/python3-sage
  python3-ubuntu    /usr/local/share/jupyter/kernels/python3-ubuntu
  sagemath          /usr/local/share/jupyter/kernels/sagemath

with the kernelspecs library

> Object.keys(await require("kernelspecs").findAll())
[
  'bash',
  'ir',
  'ir-sage',
  'javascript',
  'julia-1.9',
  'octave',
  'python3-sage',
  'python3-ubuntu',
  'sagemath'
]

This kernel:

python3           /usr/local/lib/python3.10/dist-packages/ipykernel/resources

which is actually the system default, is missing.

I realize that kernelspecs hasn't been touched in over 4 years. Maybe the jupyter kernel search algorithm or spec (if there is one) has changed?

[williamstein]

This isn't particular impactful in the context of cocalc.com, because we have a curated list of kernels. Also, things seem fine on compute servers. But it would still be nice to sort this out. I greatly prefer using kernelspecs over say a shell call out to jupyter kernelspec list, since using kernelspecs is 10x - 100x faster...

[haraldschilly]

On your machine, what does $ jupyter --paths say? I wonder if this actually a problem of https://github.com/nteract/jupyter-paths

[williamstein]

Here are the two:

jupyter --paths is essentially the same, so that doesn't seem to be the problem.

Basically jupyter kernelspec list is picking up "the system default kernel" (which jupyterlab etc uses by default, so pretty important!) at /usr/local/lib/python3.10/dist-packages/ipykernel/resources which is especially weird to me since that has only logos in it:

~/cocalc/src$ ls /usr/local/lib/python3.10/dist-packages/ipykernel/resources
logo-32x32.png  logo-64x64.png  logo-svg.svg

The path to this system kernel isn't even in the output of jupyter --paths. I suspect jupyter kernelspec just uses a different algorithm or definition than the kernelspecs library, or maybe Ubuntu patches it or something else...

[haraldschilly]

Hmm, weird. Well, kernelspecs' code discards all spec dirs where there is no kernel.json file. That makes sense to me, though.

[williamstein]

Hmm, weird. Well, kernelspecs' code discards all spec dirs where there is no kernel.json file.

Good job figuring that out.

That makes sense to me, though.

It makes sense, but it is also inconsistent with what upstream Jupyter (or Ubuntu?) actually does, so there is a bug somewhere. This probably arose out of some conflict in packaging policies, e.g., maybe in Ubuntu one package has files in several different directories, whereas in Jupyter kernels we think of everything in one directory.

Reading https://jupyter-client.readthedocs.io/en/5.2.1/kernels.html just makes me think Ubuntu is doing something weird/wrong...