12

I am using pip 20.0.2 on Ubuntu, and installing a bunch of requirements from a requirements file. For some reason, pip is deciding to install idna==2.9(link), even though that is not a compatible version with one of my directly listed dependencies. So I used python -m pipdeptree -r within the virtualenv that I'm installing everything to, and I see this listed for idna:

idna==2.9 - cryptography==2.3.1 [requires: idna>=2.1] - requests==2.22.0 [requires: idna>=2.5,<2.9] - requests-oauthlib==1.3.0 [requires: requests>=2.0.0] - social-auth-core==3.2.0 [requires: requests-oauthlib>=0.6.1] - social-auth-app-django==2.1.0 [requires: social-auth-core>=1.2.0] - responses==0.10.9 [requires: requests>=2.0] - social-auth-core==3.2.0 [requires: requests>=2.9.1] - social-auth-app-django==2.1.0 [requires: social-auth-core>=1.2.0]

As we can see, my two direct dependencies (cryptography and requests), are what require idna. According to those, it looks like pip should decide to install 2.8, because it is the latest version that will fulfill the constraints.

Why is pip instead installing idna 2.9, as indicated by the top line of that output, and this error message when running pip install -r requirements.txt:

ERROR: requests 2.22.0 has requirement idna<2.9,>=2.5, but you'll have idna 2.9 which is incompatible.

EDIT: the contents of requirements.txt and it's children, as requested in the comments:

# requirements.txt -r requirements/requirements-base.txt -r requirements/requirements-testing.txt # requirements-base.txt cryptography~=2.3.1 pyjwt~=1.6.4 requests~=2.22.0 social-auth-app-django~=2.1.0 # requirements-testing.txt hypothesis~=3.87.0 pytest~=3.6.2 pytest-django~=3.3.2 pytest-cov~=2.5.1 responses~=0.10.5

Edit 2: I've created a minimally viable example. For this example, here is requirements.txt:

cryptography~=2.3.1 requests~=2.22.0

And here are the commands I ran from start to finish in a fresh directory:

virtualenv -p python3.6 -v venv source venv/bin/activate pip install -r requirements.txt --no-cache-dir

And the full output:

Collecting cryptography~=2.3.1 Downloading cryptography-2.3.1-cp34-abi3-manylinux1_x86_64.whl (2.1 MB) |████████████████████████████████| 2.1 MB 2.0 MB/s Collecting requests~=2.22.0 Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB) |████████████████████████████████| 57 kB 18.5 MB/s Collecting asn1crypto>=0.21.0 Downloading asn1crypto-1.3.0-py2.py3-none-any.whl (103 kB) |████████████████████████████████| 103 kB 65.4 MB/s Collecting idna>=2.1 Downloading idna-2.9-py2.py3-none-any.whl (58 kB) |████████████████████████████████| 58 kB 71.4 MB/s Collecting six>=1.4.1 Downloading six-1.14.0-py2.py3-none-any.whl (10 kB) Collecting cffi!=1.11.3,>=1.7 Downloading cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (399 kB) |████████████████████████████████| 399 kB 30.3 MB/s Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB) |████████████████████████████████| 125 kB 46.7 MB/s Collecting certifi>=2017.4.17 Downloading certifi-2019.11.28-py2.py3-none-any.whl (156 kB) |████████████████████████████████| 156 kB 65.1 MB/s Collecting chardet<3.1.0,>=3.0.2 Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB) |████████████████████████████████| 133 kB 60.8 MB/s Collecting pycparser Downloading pycparser-2.19.tar.gz (158 kB) |████████████████████████████████| 158 kB 25.0 MB/s Building wheels for collected packages: pycparser Building wheel for pycparser (setup.py) ... done Created wheel for pycparser: filename=pycparser-2.19-py2.py3-none-any.whl size=111031 sha256=030a1449dd5902f2f03e9e2f8f9cc6760503136a9243e965237a1ece1196502a Stored in directory: /tmp/pip-ephem-wheel-cache-c_dx8qi5/wheels/c6/6b/83/2608afaa57ecfb0a66ac89191a8d9bad71c62ca55ee499c2d0 Successfully built pycparser ERROR: requests 2.22.0 has requirement idna<2.9,>=2.5, but you'll have idna 2.9 which is incompatible. Installing collected packages: asn1crypto, idna, six, pycparser, cffi, cryptography, urllib3, certifi, chardet, requests Successfully installed asn1crypto-1.3.0 certifi-2019.11.28 cffi-1.14.0 chardet-3.0.4 cryptography-2.3.1 idna-2.9 pycparser-2.19 requests-2.22.0 six-1.14.0 urllib3-1.25.8
Improve this question
4
  • Ugh, I think I've spotted it, during the install: Collecting idna~=2.8 Using cached idna-2.9-py2.py3-none-any.whl (58 kB). That damn pip cache looks to have bitten me. It's so strange to me that pip will use a cached version, even though it doesn't properly fulfill the requirements. Commented Feb 19, 2020 at 15:31
  • Hm, nope. Even with pip install -r requirements.txt --no-cache-dir, it still collects the wrong version: Collecting idna>=2.1 Downloading idna-2.9-py2.py3-none-any.whl (58 kB) |████████████████████████████████| 58 kB 35.7 MB/s Commented Feb 19, 2020 at 15:34
  • To get this clear, what's the exact command that caused pip to install this version of idna? Was it installed as part of what's listed in requirements.txt? Then what exactly is listed in that file? Or did you install it separately from requirements.txt? Commented Feb 19, 2020 at 15:36
  • All I am ever using to install requirements is pip install -r requirements.txt, the contents of which I'll post above. Commented Feb 19, 2020 at 15:39

2 Answers 2

Reset to default
10

Pip does not have a dependency resolver. If you tell it to install package foo without any qualifications, you’re getting the newest version of foo, even if it conflicts with other packages you’ve already installed.

Other solutions like poetry exist which do have the logic to keep everything compatible. If you need this, consider using something like that instead of plain pip.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

I don't understand this answer - pip certainly does install the dependencies of the packages you tell it to install. For instance, in this very example I am telling it to install cryptography~=2.3.1 and requests~=2.22.0, and pip is fetching those, as well as their dependencies which includes idna. Both cryptography and requests specify specific version of idna which they support - they aren't simply calling for idna, but for specific versions thereof (as shown in the pipdeptree output).
Right, but let’s say pip installs cryptography first. It picks the newest versions of idna that cryptography can use. It doesn’t consider requests at all! Then when it tries to install requests, it sees that idna is already installed and doesn’t do anything more about it. That’s what poetry does differently: it considers that both cryptography and requests both require idna, and finds the best version that satisfies both of their requirements. If there’s no overlap (like one needs idna<2 and the other needs idna>=2) it will tell you so that you can fix it.
And what do you know, if I flip the order of dependencies in the minimal example, it installs correctly. Wow, I never knew this, I always assumed pip was doing some sort of dependency resolver. TIL. Thanks!
Awesome! Yeah, that’s why tools like poetry have become so popular: doing this stuff manually when you have a whole lot of packages can be a pain in the neck.
@fildred13 There is an ongoing work on implementing a dependency resolver for pip. Here is the related issue on Github: github.com/pypa/pip/issues/988
|
0

Starting with version 20.3, pip has a backtracking dependency resolver.

https://pip.pypa.io/en/stable/topics/dependency-resolution/

Current version as of 2023-05-08 is pip-23.1.2.

Updgrade:

pip install --upgrade pip
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.