32

As part of the compilation step for a new python version, I fetch and run get-pip.py, to have the latest pip installed next to the python executable:

$ /opt/python/3.7.0/bin/python --version Python 3.7.0 $ /opt/python/3.7.0/bin/pip --version pip 18.0 from /opt/python/3.7.0/lib/python3.7/site-packages/pip (python 3.7)

I have 25 such versions under /opt/python, although I mostly use the five latest versions of each major.minor version that is not EOL. To setup an invironment I used to run virtualenv or my virtualenvutils with the -p /opt/python/X.Y.Z/bin/python option to get a virtual environment with a specific version.

With Python 3.7 this gives the imp module deprecation warning:

$ virtualenv -p /opt/python/3.7.0/bin/python /tmp/py37virtualenv Running virtualenv with interpreter /opt/python/3.7.0/bin/python Using base prefix '/opt/python/3.7.0' /opt/util/virtualenvutils/lib/python3.6/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp New python executable in /tmp/py37virtualenv/bin/python Installing setuptools, pip, wheel...done.

I have little hope this will be solved in virtualenv, as this has had a PendingDeprecationWarning at least since 2014 (as can be seen from the output in this question)

While investigating replacing virtualenv with python -m venv in virtualenvutils, I first created a new venv based virtual environment by hand:

$ /opt/python/3.7.0/bin/python -m venv /tmp/py37venv $ /tmp/py37venv/bin/pip --version pip 10.0.1 from /tmp/py37venv/lib/python3.7/site-packages/pip (python 3.7)

That has an old pip version! If you use it, you'll get:

You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command

In the virtual environment created with virtualenv you immediately get the latest version:

$ /tmp/py37virtualenv/bin/pip --version pip 18.0 from /tmp/py37virtualenv/lib/python3.7/site-packages/pip (python 3.7)

I can run a post-creation step:

/tmp/py37venv/bin/pip install -U --disable-pip-version-check pip

which will take extra time. And if there was a some security update for pip, this would imply running the non-secure version to get a secure version, an ideal point of attack.

From virtualenvutils it is trivial to do the multiple steps to create a pip-less virtualenv and then add pip using get-pip.py. From the command-line this is not so simple:

$ /opt/python/3.7.0/bin/python -m venv --without-pip /tmp/py37venvnopip $ /tmp/py37venvnopip/bin/python -c "from urllib.request import urlopen; response = urlopen('https://bootstrap.pypa.io/get-pip'); open('/tmp/tmp_get_pip.py', 'w').write(response.read())" $ /opt/python/3.7.0/bin/python /tmp/tmp_get_pip.py ...... $ /opt/python/3.7.0/bin/pip --version

pip 18.0 from /opt/python/3.7.0/lib/python3.7/site-packages/pip (python 3.7)

What is causing /opt/python/3.7.0/bin/python -m venv to take that old pip version? Is that the version available when 3.7.0 was released?

How can I update my install under /opt/python/3.7.0 in some way so that using /opt/python/3.7.0/bin/python -m venv creates a virtualenv with the latest pip version without reverting to scripts, aliases or using multiple commands? Having the latest pip installed under /opt/python/3.7.0 obviously is not enough.

There are two bundled wheels:

/opt/python/3.7.0/lib/python3.7/ensurepip/_bundled/setuptools-39.0.1-py2.py3-none-any.whl /opt/python/3.7.0/lib/python3.7/ensurepip/_bundled/pip-10.0.1-py2.py3-none-any.whl

I suspect I need to update those. Is there a better way than updating those by hand? Some option for /some/python -m venv would be nice.

(And running /some/python -m ensurepip --upgrade doesn't do the trick)

Running the deprecated /opt/python/3.7.0/bin/pyvenv has the same old pip version problem.

Improve this question
3
  • 1
    It looks like bugs.python.org/issue30628 refers to this issue, but the bugs site is currently down. Commented Aug 7, 2018 at 7:27
  • 1
    To be able to proceed without warnings I ended up creating a single file utility that downloads new .whl files in _bundled and updates ensurepip/__init__.py. Also available on PyPI Commented Aug 7, 2018 at 16:40
  • 3
    To manually upgrade run python -m pip install --upgrade pip setuptools wheel Do not just run pip install as it will create a shebang to whatever default pip you have Commented Nov 23, 2018 at 11:04

5 Answers 5

Reset to default
21

The trick is not to install the bundled version of pip (which will almost always be out of date), but to use it to install the most current version from the internet.

Standard library venv offers a --without-pip flag that can help here. After creating the virtual environment without pip, you can then you can "execute" ensurepip's wheel directly thanks to Python's zip importer. This is both faster and less hacky than installing pip and then immediately using that same pip installation to uninstall itself and upgrade.

Code speaks louder than words, so here's an example bash function for the process I've described:

# in ~/.bashrc or wherever function ve() { local py="python3" if [ ! -d ./.venv ]; then echo "creating venv..." if ! $py -m venv .venv --prompt=$(basename $PWD) --without-pip; then echo "ERROR: Problem creating venv" >&2 return 1 else local whl=$($py -c "import pathlib, ensurepip; whl = list(pathlib.Path(ensurepip.__path__[0]).glob('_bundled/pip*.whl'))[0]; print(whl)") echo "boostrapping pip using $whl" .venv/bin/python $whl/pip install --upgrade pip setuptools wheel source .venv/bin/activate fi else source .venv/bin/activate fi }

If you prefer the older project virtualenv, it also offers --no-pip, --no-setuptools, and --no-wheel flags to achieve the same on Python 2.7.

Note: Python 3.9+ venv has an --upgrade-deps option to immediately upgrade the pip/setuptools versions after creating an environment, see https://bugs.python.org/issue34556 for more info about that. I don't use this option because it still goes through an unnecessary install/uninstall of the vendored versions, which is inferior to the method of creating an environment with the latest versions directly as shown above.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Funny enough, I have written a ve tool some months ago that does something like that. I might want to polish it, since apparently I am not the only one looking to get around this inconvenience of having to update pip in all virtual environments. Should do it before Python 3.9 comes out though...
3

I use upgrade-ensurepip to update those pip and setuptools wheel files that are part of the ensurepip package. It's not as elegant as being able to upgrade ensurepip via pip, but it's still preferable to doing it manually.

https://pypi.org/project/upgrade-ensurepip/

Improve this answer

2 Comments

I don't like the idea of user messing with the bundled pip versions (for example, they might be owned by root and installed from OS package manager, you will confuse your OS if you mess with those files)
Good point. I have switched to using conda and I don't have to deal with this issue now.
3

It is an expected behavior. python -m venv calls python -m ensurepip to install pip and This answer shows that ensurepip would only install the bundled version even with --upgrade option. There isn't any official option to update the bundled pip and setuptools.

Well I have also no good idea to fix this problem as it just is the designed behavior. I would like to give two suggestions:

  1. Use pipenv. It is really good! And it will be the next-generation official package manager in the future(Although there is a big problem related to current Pypi's structure. In short, a package manager can only decide the dependencies with downloading the whole package. This gives a huge difficulty to building dependencies graph.).

  2. Implement your custom EnvBuilder, actually there is an official example about this. And in the example, it also use get-pip.py to install the latest pip.

Improve this answer

6 Comments

Thanks for the envbuilder link, I'll look into that. pipenv has quite a bit of overlap with virtualenvutilsfor my use-case, but I would still have to make links to the actual executables in /usr/local/bin. The announced use of Pipfile by pipenv and the horror of yet another config file in every project's root (in the superfluous TOML format to boot) makes it rather uninteresting in the longer term.
pipenv does not just overlap with virtualenvutils but is a combination of pip and virtualenv which means it fully covers these two tools. And further, if you decide to use pipenv, you don't need requirements.txt any more :-) Actually the biggest reason I choose pipenv is that it can really manage dependencies instead of just list them as pip freeze > requirements.txt do. For example, you delete one top dependency in Pipfile and then run pipenv clean, it will automatically delete all unused sub dependencies. Well, after all, it depends on personal opinion :-)
virtualenvutils is not virtualenv, it does a bit more than that. As I already got rid of needing a requirements.txt (and setup.cfg, Makefile, tox.ini, setup.py, dist, build, .tox, etc.) anything pipenv puts there only clutters the development tree again. So pipenv (for me) is just extra work without benefit. Whenever I change dependencies that remove things, I save, then reinstall, a utility and its virtualenv from scratch (there is bound to be a new micro version of the used Python anyway), to get the right dependencies (and restore if that cannot be resolved).
Guess we have different usage scenarios.
I think pipenv should probably not be recommended these days. Extending EnvBuilder looks interesting though.
|
0

If you are using virtualenvwrapper you can put upgrade command in postmkvirtualenv script. It is in ~/.virtualenvs folder by default.

Each time you create a new virtual environment, pip would be upgraded. I know this is not perfect solution, but it's working.

# vim ~/.virtualenvs/postmkvirtualenv pip install --upgrade pip
Improve this answer

Comments

-1

I just use a bash script that does it for me (create_venv.sh):

#!/usr/bin/env bash python -m venv .venv source .venv/Scripts/activate echo "upgrading $(which pip)" python -m pip install --upgrade pip deactivate

Then I put in in my path and use that instead of directly using python -m venv .venv

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.