Use pybind11-stubgen for generating stubs

[ProfFan]

Use pybind11-stubgen to generate the stubs, which is a stub generator specifically built for pybind11 types.

Also changed the way stub is generated. Should hopefully fix the issues with mypy's generator.

Tested locally on both VSCode and JupyterLab (macOS M1):

[tianyilim]

Responding here to track #1814

I manually had to create the folders python-stubs and python-unstable-stubs in build/python/CMakeFiles, I guess it's not too difficult to fix.

However, by default in VSCode the stub info for me disappears by default (no error, but it's reported as Unknown)

Perhaps this is more of a Pylance thing now, where it doesn't know where to find the generated stubs?

[ProfFan]

@tianyilim It works for me in Pylance under macOS M1. Are you sure that you are using the correct Python environment?

Please do the following:

Make sure you uninstall ALL of your previous GTSAM python bindings by doing pip uninstall gtsam

Reinstall the python bindings. In VSCode, check bottom right if you are in the correct Python environment:

Choose the correct one, then Cmd+Shift+P and say Python: Restart Language Server.

If this does not work, check:

1. Remove your entire build folder (your 1st problem is because of this)
2. Build GTSAM with Python bindings
3. Install with make python-install or ninja python-install

[tianyilim]

Okay, I tried with a fresh conda environment and it seems to have worked:

conda create --name gtsamStubs python=3.10 -y
conda activate gtsamStubs
cd gtsam/
git fetch -ap
git checkout fan/gen_stubs
mkdir build
python -m pip install -r python/dev_requirements.txt
python -m pip install pyparsing
cd build/
cmake .. -DGTSAM_BUILD_PYTHON=1
make python-install -j$(nproc)

I now get similar output to when gtsam-stubs was installed externally.

Previously, I was using the flags -DGTSAM_PYTHON_VERSION=3.10 and -DCMAKE_INSTALL_PREFIX="./install" but I removed them this time.

Thanks for the help! 😄

[dellaert]

Cool, I’ll try on my linux env and Mac to be sure - in conda env, then merge if works

[varunagrawal]

I'm a bit lost. If the issue is gtsam.noiseModel not showing up properly, why are we looking at the way pybind11 type stubs are generated? What is this PR trying to fix?

[ProfFan]

I'm a bit lost. If the issue is gtsam.noiseModel not showing up properly, why are we looking at the way pybind11 type stubs are generated? What is this PR trying to fix?

Just in my experience pybind11-stubgen generates better type annotations (a lot of C++ types are set to just Any with stubgen). This PR fixes the noiseModel issue, refactors the stub generation into their own targets, and replaced mypy's stubgen to pybind11-stubgen.

[varunagrawal]

Except for one thing, looks good to me.

[varunagrawal]

Do we need this?

[ProfFan]

Yeah to fix latest clang 15 on Linux, I tested on my arch. (we are using std::find)