Replace bundled pip and setuptools with a downloader in the ensurepip module

[webknjaz]

BPO	36608
Nosy	@ericvsmith, @ned-deily, @serhiy-storchaka, @dstufft, @webknjaz, @pradyunsg
PRs	gh-80789: Implement build-time pip bundling in ensurepip #12791

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-04-11.22:13:02.045>
labels = ['3.8', 'type-feature', 'library', '3.9']
title = 'Replace bundled pip and setuptools with a downloader in the ensurepip module'
updated_at = <Date 2021-07-05.23:28:08.628>
user = 'https://github.com/webknjaz'

bugs.python.org fields:

activity = <Date 2021-07-05.23:28:08.628>
actor = 'ned.deily'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-04-11.22:13:02.045>
creator = 'webknjaz'
dependencies = []
files = []
hgrepos = []
issue_num = 36608
keywords = ['patch']
message_count = 6.0
messages = ['339998', '340039', '340060', '340169', '340218', '347836']
nosy_count = 6.0
nosy_names = ['eric.smith', 'ned.deily', 'serhiy.storchaka', 'dstufft', 'webknjaz', 'pradyunsg']
pr_nums = ['12791']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue36608'
versions = ['Python 3.8', 'Python 3.9']

Linked PRs

• GH-80789: Bundle ensurepip wheels at build time #109130
• GH-80789: Get rid of the ensurepip infra for many wheels #109245

[webknjaz]

Hi,

I've noticed that there's an idea to not pollute Git tree with vendored blobs. In particular, ensurepip is one of the components doing this.

Such a wish was expressed here: https://bugs.python.org/issue35277#msg330098

So I thought I'd take a stab at it...

[ericvsmith]

ensurepip does not access the network, by design. We do not want it to start access the network without a lot of discussion.

And if it does access the network, it will need to be able to use alternate URLs. For example: where I deploy Python, it would not have access to the URLs in your PR, but instead would need to specify a different (internal) location. This is the same reason that pip install has --find-links, --no-index, --extra-index-url, etc.

I think this would need a lot of discussion (probably on distutils-sig), and probably a PEP.

[ericvsmith]

And I don't mean to sound like a total downer. I just think it's important that we recognize all of the use cases.

Thanks for your work on this.

[pradyunsg]

(Not sure how the Roundup handles email replies but I'm hoping this goes to
the right place)

I think it would be better if the downloading got invoked during the
interpreter build process -- to download the wheels and add them to the
final distribution. This lets us remove the wheels from the source
tree/version control and prevents needing to change the PEP for this change.

Functionally, I imagine having all the download logic in some sort of
ensurepip._bootstrap which has all the download logic and that getting
invoked in the build process. python -m ensurepip not hitting the
internet is a good invariant to keep.

On Fri, 12 Apr 2019 at 8:54 PM, Eric V. Smith <[email protected]>
wrote:

Eric V. Smith <[email protected]> added the comment:

And I don't mean to sound like a total downer. I just think it's important
that we recognize all of the use cases.

Thanks for your work on this.

----------

---

Python tracker <[email protected]>
<https://bugs.python.org/issue36608\>

---

[serhiy-storchaka]

I proposed to move bundled pip and setuptools to the external repository and download them at build time like Tcl and other dependencies on Windows.

[webknjaz]

Thanks for the feedback!

I've changed it a bit to have a separate command for downloading bundles to the source tree. It'd work as in python -m ensurepip.bundle (needs a better name/CLI args probably).

Does it sound better now?

[webknjaz]

Folks, I revamped the PR and need some help with the build process integration here: https://github.com/python/cpython/pull/12791/files#r1306308587. Could anybody take a look?

[webknjaz]

@pradyunsg should this issue get newer labels for 3.10, 3.11, 3.12 and 3.13?

[pradyunsg]

I discussed this at some length at the core developers sprint this week (including some of current RMs).

The main benefit of doing this is that we can avoid committing a ~2 MB wheel¹, on a quarterly basis for pip releases, due to a limitation of git + binary blobs. Given that we no longer vendor a setuptools wheel, which evolves at a faster cadence than pip, the size growth rate as well as the rate of change isn't particularly high.

The costs of this are (i) every one who tries to build CPython for the first time and run tests would need to perform this download, (ii) the CPython release process needs to be amended to include a downloaded wheel as part of the release tarball, (iii) it's unclear how/when this downloaded wheels should be deleted/removed and how that interacts with existing make clean/distclean etc, and (iv) it introduces an additional point of interaction with PyPI as part of the development/release cycle for CPython.

Based on the additional side-effects and work that this would cause, I'm gonna say that this isn't a big deal for now. We can revisit this if there is a strong(er) argument against committing the wheels into the repository to justify the additional side-effects on the development and release processes that it would have.

Footnotes

1. Or a few more, depending on bugfixes and whether pip's RM decides to place every wheel. ↩

[SpecLad]

Would it perhaps be viable to include pip as an (unpacked) sdist in the ensurepip source tree, and build it during the CPython build process? The repo bloat should be much smaller that way.

(You'd also need to include pip's build dependencies, but those don't need to be updated frequently.)

[webknjaz]

Would it perhaps be viable to include pip as an (unpacked) sdist in the ensurepip source tree, and build it during the CPython build process? The repo bloat should be much smaller that way.

@AA-Turner @pradyunsg WDYT about the suggestion above? Could also be an unpacked wheel (wheel unpack pip-*.whl output), which feels like a more slim solution, than an sdist.