Subprocess error, looking for attrdict module but not finding it

[Porkepix]

What you were trying to do (and why)

Have pipgrip executed to find a package dependencies (it was actually run by homebrew, but I could reproduce it directly by myself after with a setup right from PyPI inside a venv)

What happened (including command output)

It crashed and raised an exception on this specific module (other modules didn't encounter issues).

Command output

$ pipgrip -vvv wxPython==4.2.0
DEBUG: environment: {'implementation_name': 'cpython', 'implementation_version': '3.10.5', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '5.18.12-arch1-1', 'platform_system': 'Linux', 'platform_version': '#1 SMP PREEMPT_DYNAMIC Fri, 15 Jul 2022 15:33:02 +0000', 'python_full_version': '3.10.5', 'platform_python_implementation': 'CPython', 'python_version': '3.10', 'sys_platform': 'linux'}
DEBUG: pip version: [22, 0, 4]
DEBUG: pipgrip version: 0.8.5
INFO: discovering wxpython==4.2.0
DEBUG: Downloading/building wheel for wxpython==4.2.0 into cache_dir /home/clement/.cache/pip/wheels/pipgrip
DEBUG: ['/tmp/test_pipgrip/.venv/bin/python', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/home/clement/.cache/pip/wheels/pipgrip', '--progress-bar=off', 'wxpython==4.2.0']
ERROR: Downloading/building wheel for wxpython==4.2.0 failed with output:
Collecting wxpython==4.2.0
  Using cached wxPython-4.2.0.tar.gz (71.0 MB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-wheel-f1dil03u/wxpython_d801786558cd4c0c93537ec795e53f1a/setup.py", line 27, in <module>
          from buildtools.config import Config, msg, opj, runcmd, canGetSOName, getSOName
        File "/tmp/pip-wheel-f1dil03u/wxpython_d801786558cd4c0c93537ec795e53f1a/buildtools/config.py", line 30, in <module>
          from attrdict import AttrDict
      ModuleNotFoundError: No module named 'attrdict'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Traceback (most recent call last):
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/pipgrip/pipper.py", line 258, in _download_wheel
    out = stream_bash_command(args)
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/pipgrip/pipper.py", line 91, in stream_bash_command
    raise subprocess.CalledProcessError(return_code, bash_command, output=out)
subprocess.CalledProcessError: Command '['/tmp/test_pipgrip/.venv/bin/python', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/home/clement/.cache/pip/wheels/pipgrip', '--progress-bar=off', 'wxpython==4.2.0']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/test_pipgrip/.venv/bin/pipgrip", line 8, in <module>
    sys.exit(main())
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/pipgrip/cli.py", line 430, in main
    source.root_dep(root_dependency)
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/pipgrip/package_source.py", line 160, in root_dep
    self.discover_and_add(req.__str__())
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/pipgrip/package_source.py", line 133, in discover_and_add
    to_create = discover_dependencies_and_versions(
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/pipgrip/pipper.py", line 414, in discover_dependencies_and_versions
    wheel_fname = _download_wheel(
  File "/tmp/test_pipgrip/.venv/lib/python3.10/site-packages/pipgrip/pipper.py", line 266, in _download_wheel
    raise RuntimeError("Failed to download/build wheel for {}".format(package))
RuntimeError: Failed to download/build wheel for wxpython==4.2.0

What you expected to happen

Don't get an exception and list correctly the dependencies

Step-by-step reproduction instructions

Just call pipgrip wxPython==4.2.0.

[ddelange]

Hi @Porkepix 👋

Will close this issue as duplicate of #40 (see #52 for another example of failing source build).

This is a downstream issue with wxPython not (yet) publishing wheels for your current environment (mix of cpu architecture, os, python version and implementation. See also first line of your command output above. You can confirm the issue by searching on pypi.org and inspecting the wheel filenames under downloads. In this case ( is probably a trivial fix in their source installation instruction set for linux.

To avoid source install (tar.gz), there are beautiful CI setups out there to get wheels uploaded for pretty much the whole matrix. Here's a package that can work with limited ABI, so there they can build py36 wheels that are upward compatible, and so a linux x86 py36 abi3 wheel will smoothly install in your particular py310 env. For example giampaolo/psutil#2103. Adopting such a solution, however, is up to the upstream maintainers & contributors.

Hope that helps!

[Porkepix]

@ddelange I understand the source of the issue and mentioned it in the issue you can see above, but publishing Linux wheels doesn't seems to be something planned for now. I understand it's kind of an issue to manage such cases and that #40 you pointed out tries to solve the issue. While it's not yet the case, maybe is it at least possible to have a clean error message (that might even point out to that issue!) rather than a harsh exception (not even visible when ran by Homebrew btw)?

EDIT: And thanks for the explanation, of course!

[ddelange]

not even visible when ran by Homebrew btw

definitely looking into that!

agreed that the error message could be more verbose and link to #40

[ChrisBarker-NOAA]

Just coming from the side here, but it certainly SHOULD be possible to determine dependencies without using platform specific wheels.

Maybe this is a still a gap in the toolchain(s) though -- if a package is built from old-style setup.py :-(

[ddelange]

Yes, it's a classic I'm afraid ref pypa/pip#1884

for instance, if there's no Mac wheel, it might be that the project does something like install_requires=['torchvision==0.10.0;sys_platform=="darwin"'], and so you will need to run through all of setup.py to find out the correct dependencies, as the linux wheel won't contain this dependency.

[ChrisBarker-NOAA]

Och! that's even worse than I thought.

It seems it wouldn't have been too hard to add a requirements specification to the sdist spec years ago, but ohh well.

-CHB

[ddelange]

Yes, py ecosystem can be a bit of a headache at times. Too many wasted hours was the trigger for me to publish this package in the first place... Ran into more walls in #45