namespace packages are broken in rules_python py_binary/py_test rules ([email protected])

[kwlzn]

🐞 bug report

Affected Rule

The issue is caused by the rule: py_binary, py_test in rules_python==1.3.0

Is this a regression?

I don't believe so.

Description

when attempting to use namespace packages with rules_py, the transitive 3rdparty environment is not setup as expected which leads to immediate import failures:

with bazel run against a py_binary:

$ bazel run //:isaaclab_bin
INFO: Analyzed target //:isaaclab_bin (194 packages loaded, 75009 targets configured).
INFO: Found 1 target...
Target //:isaaclab_bin up-to-date:
  bazel-bin/isaaclab_bin
INFO: Elapsed time: 114.086s, Critical Path: 9.77s
INFO: 5 processes: 5 internal.
INFO: Build completed successfully, 5 total actions
INFO: Running command line: bazel-bin/isaaclab_bin
Traceback (most recent call last):
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/_main/isaaclab.py", line 3, in <module>
    from isaaclab.app import AppLauncher
  File "/home/kwilson/dev/bazel_plus_isaaclab/isaaclab.py", line 3, in <module>
    from isaaclab.app import AppLauncher
ModuleNotFoundError: No module named 'isaaclab.app'; 'isaaclab' is not a package

with bazel test against a py_test:

$ bazel test //:isaaclab_test --test_output=all
INFO: Analyzed target //:isaaclab_test (1 packages loaded, 2 targets configured).
FAIL: //:isaaclab_test (see /home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/testlogs/isaaclab_test/test.log)
INFO: From Testing //:isaaclab_test:
==================== Test output for //:isaaclab_test:
Traceback (most recent call last):
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/sandbox/linux-sandbox/1/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_test.runfiles/_main/isaaclab.py", line 3, in <module>
    from isaaclab.app import AppLauncher
  File "/home/kwilson/dev/bazel_plus_isaaclab/isaaclab.py", line 3, in <module>
    from isaaclab.app import AppLauncher
ModuleNotFoundError: No module named 'isaaclab.app'; 'isaaclab' is not a package
================================================================================
INFO: Found 1 test target...
Target //:isaaclab_test up-to-date:
  bazel-bin/isaaclab_test
INFO: Elapsed time: 7.476s, Critical Path: 6.44s
INFO: 6 processes: 4 internal, 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 6 total actions
//:isaaclab_test                                                         FAILED in 0.1s
  /home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/testlogs/isaaclab_test/test.log

Executed 1 out of 1 test: 1 fails locally.

namespace packages are a popular way to modularize libraries that have many underlying sub-deps, which enables each modularized dep to be versioned independently (and enables e.g. multi-team development workflows). in our case, we hit this with NVIDIA's IsaacSim's IsaacLab libraries which heavily depend on this basic python packaging feature.

Expected Result

the from isaaclab.app import AppLauncher import should be able to import all of it's transitive deps, as demonstrated in this uv-based reproduction:

$ OMNI_KIT_ACCEPT_EULA=YES \
  uv run \
    --with=torch==2.5.1+cu121 \
    --extra-index-url=https://download.pytorch.org/whl/cu121 \
    --with="isaacsim[rl,extscache]==4.5.0" \
    --with "isaaclab[isaacsim,all]==2.0.2" \
    --extra-index-url https://pypi.nvidia.com \
    --python=python3.10 \
    python3
Python 3.10.16 (main, Feb 12 2025, 14:50:02) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from isaaclab.app import AppLauncher
>>>

🔬 Minimal Reproduction

A minimal repro has been provided in the rules_python_1.3.0_no_rules_py branch of this repo.

Reproduction one-liner:

$ git clone https://github.com/kwlzn/bazel_plus_isaaclab && \
     cd bazel_plus_isaaclab && \
     git checkout rules_python_1.3.0_no_rules_py && \
     bazel run //:isaaclab_bin

🔥 Exception or Error

INFO: Analyzed target //:isaaclab_bin (194 packages loaded, 75009 targets configured).
INFO: Found 1 target...
Target //:isaaclab_bin up-to-date:
  bazel-bin/isaaclab_bin
INFO: Elapsed time: 114.086s, Critical Path: 9.77s
INFO: 5 processes: 5 internal.
INFO: Build completed successfully, 5 total actions
INFO: Running command line: bazel-bin/isaaclab_bin
Traceback (most recent call last):
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/_main/isaaclab.py", line 3, in 
    from isaaclab.app import AppLauncher
  File "/home/kwilson/dev/bazel_plus_isaaclab/isaaclab.py", line 3, in 
    from isaaclab.app import AppLauncher
ModuleNotFoundError: No module named 'isaaclab.app'; 'isaaclab' is not a package

🌍 Your Environment

Operating System:

  
Ubuntu 24.x
  

Output of bazel version:

  
Bazelisk version: development
Build label: 7.6.0
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Mar 24 15:56:19 2025 (1742831779)
Build timestamp: 1742831779
Build timestamp as int: 1742831779
  

Rules_python version:

  
1.3.0
  

Anything else relevant?

• relates to Implement venv/site-packages based binaries #2156

[groodt]

I agree, with @aignas, most of your issues with very complex packages will be duplicates of the well-known issue described above related to lack of site-packages layouts causing friction. So please follow along there for updates.

However, a small fyi, your issue message is not related to namespace packages. Your error is related to a clash between your first-party code (isaaclab.py) and the module isaaclab on PYTHONPATH.

When I update isaaclab.py -> main.py on your branch, the error is:

/tmp/bazel_plus_isaaclab (rules_python_1.3.0_no_rules_py)$ bazel run //:isaaclab_bin      
INFO: Invocation ID: b24f7b09-37cd-4b46-8134-bd99932e329f
INFO: Analyzed target //:isaaclab_bin (1 packages loaded, 2 targets configured).
INFO: Found 1 target...
Target //:isaaclab_bin up-to-date:
  bazel-bin/isaaclab_bin
INFO: Elapsed time: 2.871s, Critical Path: 1.80s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions
INFO: Running command line: bazel-bin/isaaclab_bin
Unable to find 'omniverse-kit' package

And typically, the workarounds for now are to patch the packages (either inside bazel, or outside bazel and published to a private mirror).

[kwlzn]

aha, thanks for pointing out that namespace clash.

from a downstream consumer POV, I see #2156 as a feature implementation (e.g. a roadmap item) - where as #2757 reports a concrete failure mode (e.g. a bug in the current state impl that others might find in the event they hit the same or similar issue). iow, maybe "#2156 closes #2757" where post #2156 the repro in #2757 would succeed and thus be closed - but #2156 isn't necessarily guaranteed to solve this without explicitly testing it prior to close. additionally, the two other linked issues in torch in #2156 remain open and presumably would be validated in the same manner. but perhaps you all prefer to manage your issue tracking oriented at the developer team modality vs the user experience. I'm not opinionated either way, just pointing out the thought process behind taking the time to file this separate bug report.

[aignas]

My goals here with linking issues and having only a single issue is:

• Have adequate linking between various user reports.
• Limit the number of issues that we have to look at at any given time. This is a volunteer project and knowing if we have a new bug is more important than following any prescribed workflow.
• The User Acceptance Testing (UAT) workflow is good when there is a larger dev team, as I said, this is a volunteer project.

[groodt]

Yup, I also agree it's a duplicate fwiw. Even more so now the issue is entirely unrelated to namespaces. :)

It's just another duplicate of the same, well-known problem, with the same workarounds that work fine today. User-space patching to dependencies that assume a site-packages layout.

And by marking as duplicate, we do still keep a historical linkage.

[kwlzn]

This is a volunteer project

aha, I was not aware of that - for some reason I was thinking this project was at least partially staffed/supported by Google since it seems pretty intrinsic to Bazel and was previously under the bazelbuild github org (and still reflects as so on the surface as the top goog search hit for "rules_python" etc). But I also (now) see the explicit "not supported by Google" callout in the repo README. thanks for laying out the rationale - all makes more sense now.