Switch from get-pip to ensurepip

[edmorley]

The images in this repo currently use https://github.com/pypa/get-pip to bootstrap pip, however, all versions of Python supported here now include the ensurepip module (which was added in Python 3.4), which could be used instead (along with a suitable pip install -U command to upgrade to the chosen pinned pip etc):
https://docs.python.org/3/library/ensurepip.html

Switching to ensurepip would remove an external dependency and the script version bumping that comes with it (eg 1980d7b x8 for each upstream get-pip change).

Would you be open to a PR that made that switch?

[tianon]

Oh, very interesting proposal -- I can't remember why we explicitly disabled usage of ensurepip previously. 🤔

From what I can tell, in 3.12+ it effectively just installs itself (no downloading or other modules), so we could even stop scraping pip versions from it and just "trust" it to do the right thing, perhaps even just by removing our explicit flag that disables it during build/install. 🤔

Does it maybe install to a different path than the explicit/get-pip version? I'm trying to jog my memory as to why that fence is there, but I really am coming up blank. 😅

[edmorley]

There are two ways we could use ensurepip:

1. By continuing to disable ensurepip during the Python compile, but then running it explicitly in a separate layer afterwards (the same way that get-pip is run in a separate layer)
2. By re-enabling it during the Python build, and having pip be installed in the same layer as Python again

When I filed this issue I was thinking of (1).

Searching issues/PRs for ensurepip prior art, I found:

• Always run pip installer to get implicit deps #186
• Switch to unconditional invocation of "get-pip.py" #194

So it appears that:

• The use of the ensurepip (during the Python compile) for Python 3 images meant the implicit packages from get-pip were no longer installed.
• This was first fixed by running an extra pip install step after the Python compile, to install setuptools and wheel.
• However, to prevent layer invalidation of the Python layer every time a new pip/setuptools/wheel release occurred, this was then switched to disabling ensurepip during the build and calling get-pip in a later layer for all Python versions. In addition, the way get-pip was updated would mean the images would only get bumped for new pip releases and not also every time setuptools was released.

I think things have changed a fair bit since then:

• The images no longer default to latest pip/setuptools (and instead the versions that ship with that Python version), so the concern over too many setuptools updates doesn't apply. (xref How about updating setuptools, wheel, pip and installing setuptools-scm from git? #365)
• Python 2 images are no longer being built (where ensurepip doesn't exist in the stdlib), so ensurepip can be used unconditionally, rather than having the prior complexity of split get-pip/ensurepip codepaths.

As such, using ensurepip does seem viable now? The decision would be whether to use it via approach (1) or (2). Since the "too many setuptools releases" concern doesn't apply any more (given tracking bundled version), approach (2) does seem more viable than it was in the past, too - and would mean fewer layers (if that's still something that people care about).

[tianon]

Nice, thank you for the detailed digging! This is really helpful. ❤️

Looking at https://docs.python.org/3/library/ensurepip.html, --default-pip seems interesting (and something get-pip.py is currently doing by default). For Python itself, we're currently setting up some symlinks manually, but maybe there's a way we can pass through --default-pip to the Python install process directly so we don't have to add pip to that list? 👀

[edmorley]

I have a WIP of this locally, but will wait until #954 merged before continuing with it, to avoid conflicts.

[edmorley]

I've opened #955 for this, and chose approach (2) from #951 (comment).

There's no rush to look at that PR - I think it makes sense to wait until #954 has been released for a bit to reduce the chance of further churn :-)