Currently this project uses pybind11, which is coupled with both torch and Python ABIs. This leads to a huge amount of release artifacts (M * N, now 86) for each version, and sometimes causes some trouble to downstream projects that want to vendor it (vllm-project/vllm#41516, vllm-project/vllm#41487).
To make it portable, my suggestions are:
- Switch to https://github.com/apache/tvm-ffi as a whole. This might not be easy since this project depends on torch a lot. Or,
- Provides ABI stability on both torch and Python by migrating to LibTorch Stable ABI and compile the Cpp interface with
Py_LIMITED_API. So we can have 1 binary wheel on each architecture that is compatible with Torch 2.9+ and Python 3.8+. Or,
- Moving to
TORCH_LIBRARY (not using stable ABIs) and opt-inPy_LIMITED_API. Then the wheel still depends on a specific torch version, but a wide range of Python versions from 3.8+. Or,
- Replacing
pybind11 with nanobind. This requires minimal effort with Python stable ABI support, but only ranging from 3.12+.
I can help with any option if needed.
Currently this project uses pybind11, which is coupled with both torch and Python ABIs. This leads to a huge amount of release artifacts (M * N, now 86) for each version, and sometimes causes some trouble to downstream projects that want to vendor it (vllm-project/vllm#41516, vllm-project/vllm#41487).
To make it portable, my suggestions are:
Py_LIMITED_API. So we can have 1 binary wheel on each architecture that is compatible with Torch 2.9+ and Python 3.8+. Or,TORCH_LIBRARY(not using stable ABIs) and opt-inPy_LIMITED_API. Then the wheel still depends on a specific torch version, but a wide range of Python versions from 3.8+. Or,pybind11withnanobind. This requires minimal effort with Python stable ABI support, but only ranging from 3.12+.I can help with any option if needed.