Unnecessary restrictions on project layout due to bad heuristic

[h3har]

Maturin seems to make incorrect assumptions about the project layout that restricts the kinds of wheels you can build with it. For example, when building mixed Rust/Python projects with python-source and module-name set, maturin assumes that the path pointed to by module-name is an existing Python module if: it either contains an __init__.py file or does not contain a Cargo.toml file.

This comes from the following excerpt in Maturin's source code:

let python_module = if !python_module.join("__init__.py").is_file()
    && python_module.join("Cargo.toml").is_file()
{
    debug!("No __init__.py file found in {}", python_module.display());
    None
} else {
    Some(python_module)
};

This prevents you having multiple top-level modules, e.g.:

module/
    python_code.py
_module.cp312-win_amd64.pyd

compiled from:

rust/
    lib.rs
src/
    module/
        python_code.py
Cargo.toml
pyproject.toml

This layout cannot be achieved by setting:

[tool.maturin]
python-source = "src"
module-name = "_module"

since maturin looks for src/_module, sees that src/_module/Cargo.toml doesn't exist, and determines that it must therefore be a Python module, yielding the following error when running maturin develop:

💥 maturin failed
  Caused by: python-source is set to `...\python\src`, but the python module at
`...\python\src\_module` does not exist. Either create the Python module or remove the `python-source` setting from
pyproject.toml.

Fundamentally: Maturin's code seems to assume that a mixed Rust/Python project consists of one Python module that contains the Rust extension module. However, this assumption is not correct: Python distributions can contain arbitrarily many top-level modules and there's no reason to require that the Rust extension module is inside the main Python module.

Feature Request

Maturin should not make unnecessary assumptions about project layout. Python distributions can contain arbitrarily many top-level modules - some may be plain Python code, some may be compiled extension modules, and others may be directories that contain both. Maturin's goal should basically be to copy the Python source tree into the wheel (subject to filtering), and insert the compiled extension module at the desired place implied by module-name, which may be at the root of the distribution.

[messense]

See #1372 (comment)

[h3har]

Apologies @messense , the issue/PR you linked appears to provide a way to include multiple pure Python packages in the top-level, but the latest version of maturin still appears to break if you set the module-name for the Rust extension module to a top-level module in a mixed Rust/Python project with a single pure Python package (as illustrated in the OP).

Is there a reason that Maturin needs to treat "the python_module" specially at all? If I understand correctly, all that building the project should require is:

• For binary wheels: copy each of the python-packages into the root of the wheel and copy the extension module into the path within the wheel implied by module-name. If module-name is not within any of the python-packages, copy the extension module into the root of the wheel ¹.
• For editable wheels: add the unique parent directories of each of the python-packages to a .pth file in the wheel and copy the extension module into appropriate path within the source tree implied by module-name. If the module-name is not within any of the python-packages, copy the extension module into the root of the wheel ².

Is this an over-simplification? It seems like maturin's internal representation could be simplified while also making it more flexible (why is there a field for the python_module?).

Footnotes

1. If module-name is the same as one of python-packages, then that should be an error. Also, it's not entirely clear from my reading of the source whether this is an issue, but there's also no reason that maturin should have to assume that the files listed in python-packages are directories. So if top_level.py is listed in python-packages, then it would be an error if module-name is also top_level (in general, you'd strip Python source file or extension module suffixes before checking if there was a clash). ↩

2. More generally, you should be able to copy the extension module into the wheel whenever there is no sibling __init__.py file at the corresponding path within the source tree due to how namespace packages work. However, perhaps that's an unnecessary complication. Maybe the first rule I proposed is simpler and less surprising: only copy into the wheel if it's a top-level module that can't be placed within one of the python-packages. ↩

[messense]

The simplifcation could be the right direction, but I would encourge you to actually try to implement it and send a PR.

[h3har]

The simplifcation could be the right direction, but I would encourge you to actually try to implement it and send a PR.

I may have time in the next few weeks.

For now I have solved my issues by extending my own Python build backend to support PyO3 projects.