24

I am trying to install the Scrapy package (among others) for python using pip. I have tried doing the installation using python 3 and python 2, I have installed/upgraded the setuptools like so: $ pip3 install --upgrade setuptools, I have tried to use the --trusted-host option like so: $ pip3 install --trusted-host pypi.python.org Scrapy. But I always get the same error message when I run $ pip3 install Scrapy. The complete output is this:

Collecting Scrapy Using cached Scrapy-1.3.2-py2.py3-none-any.whl Collecting PyDispatcher>=2.0.5 (from Scrapy) Using cached PyDispatcher-2.0.5.tar.gz Collecting service-identity (from Scrapy) Using cached service_identity-16.0.0-py2.py3-none-any.whl Collecting pyOpenSSL (from Scrapy) Using cached pyOpenSSL-16.2.0-py2.py3-none-any.whl Collecting w3lib>=1.15.0 (from Scrapy) Using cached w3lib-1.17.0-py2.py3-none-any.whl Collecting parsel>=1.1 (from Scrapy) Using cached parsel-1.1.0-py2.py3-none-any.whl Collecting queuelib (from Scrapy) Using cached queuelib-1.4.2-py2.py3-none-any.whl Requirement already satisfied: six>=1.5.2 in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages (from Scrapy) Collecting Twisted>=13.1.0 (from Scrapy) Using cached Twisted-17.1.0.tar.bz2 Complete output from command python setup.py egg_info: Download error on https://pypi.python.org/simple/incremental/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found! Couldn't find index page for 'incremental' (maybe misspelled?) Download error on https://pypi.python.org/simple/: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) -- Some packages may not be found! No local packages or working download links found for incremental>=16.10.1 Traceback (most recent call last): File "<string>", line 1, in <module> File "/private/var/folders/gy/5xt04_452z791v1qjs1yzxkh0000gn/T/pip-build-nkv4jozy/Twisted/setup.py", line 21, in <module> setuptools.setup(**_setup["getSetupArgs"]()) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/core.py", line 108, in setup _setup_distribution = dist = klass(attrs) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/dist.py", line 317, in __init__ self.fetch_build_eggs(attrs['setup_requires']) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/dist.py", line 372, in fetch_build_eggs replace_conflicting=True, File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 851, in resolve dist = best[req.key] = env.best_match(req, ws, installer) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1123, in best_match return self.obtain(req, installer) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1135, in obtain return installer(requirement) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/dist.py", line 440, in fetch_build_egg return cmd.easy_install(req) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 668, in easy_install raise DistutilsError(msg) distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('incremental>=16.10.1') ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/gy/5xt04_452z791v1qjs1yzxkh0000gn/T/pip-build-nkv4jozy/Twisted/

I am on a mac OS version 10.12.1 and am using python 3.6. Does anybody know a solution to this problem?

Improve this question
3

18 Answers 18

Reset to default
15

pip install --trusted-host pypi.python.org autopep8 (any package name)

This command will add pypi.python.org to the trusted sources and will install all the required package.

I ran into the error myself and typing this command helped me install all the pip packages of python.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

it should be noted that this is a VERY RISKY IDEA! this disables HTTPS certificate validation, and allows a man in the middle to install ARBITRARY CODE. curl calls this option --insecure (which is more accurate than "trusted"). the accepted answer about OpenSSL is correct
This is useful when behind a MITM corporate proxy.
13

As stated here https://bugs.python.org/issue28150 in previous versions of python Apple supplied the OpenSSL packages but does not anymore.

Running the command pip install certifi and then pip install Scrapy fixed it for me

Improve this answer

1 Comment

also possible to install OpenSSL via brew (or whatever package manager). on some Linux systems, you also may need to install ca-certificates (or whatever variation of that your distro has, but that's usually the correct name)
11

One note on the above answers: it is no longer sufficient to add just pypi.python.org to the trusted-hosts in the case where you are behind an HTTPS-intercepting proxy (we have zScaler).

I currently have the following in my pip.ini:

trusted-host = pypi.python.org pypi.org files.pythonhosted.org

Running pip -v install pkg will give you some hints as to which hosts might need to be added.

Improve this answer

3 Comments

If pip.ini does not exist, you have to also create the [global] section, then put in the "trusted-host = ..." line.
You can also supply multiple --trusted-host arguments to your pip install.
it should be noted that this is a VERY RISKY IDEA! this disables HTTPS certificate validation, and allows a man in the middle to install ARBITRARY CODE. curl calls this option --insecure (which is more accurate than "trusted"). the accepted answer about OpenSSL is correct
8

Something to try --- tell python to not use https with the index directive and a http:// address (not https://) 

pip install --index-url=http://pypi.python.org/simple/ --trusted-host pypi.python.org Scrapy

You may be behind a corporate firewall and Ive have experiences where even the above failed, though Im not going to pretend like I know enough about firewalls or SSL to understand why. In that case the only way I was able to get around that was to get a certificate file and pass it to python. See kenorb’s answer here for details.

Improve this answer

1 Comment

it worked for me, but I need to find an HTTP-mirror without redirecting to HTTPS, for example mirrors.sustech.edu.cn/pypi/simple
5

If you're using python3, you can try this too:

python3 -m pip install --upgrade Scrapy --trusted-host pypi.org --trusted-host files.pythonhosted.org
Improve this answer

1 Comment

Worked for me on Mac
3 
 pip3 install --trusted-host pypi.org --trusted-host files.pythonhosted.org <app>
Improve this answer

Comments

2

It seems that Scrapy fails because installing Twisted fails, which fails because incremental fails. Running pip install --upgrade pip && pip install --upgrade incremental fixed this for me.

Improve this answer

Comments

1

If adding pypi.python.org as a trusted host does not work, you try adding files.pythonhosted.org. For example

python -m pip install --upgrade --trusted-host files.pythonhosted.org <package-name>
Improve this answer

Comments

1

In Windows 10 / search the drive you have installed the conda or it should be in C:\Users\name\AppData\Roaming\pipright with your mouse right click and select edit with notepad leave the [global] and replace what ever you have in there with blow code, Ctrl+s and rerun the code. it should work.

trusted-host = pypi.python.org pypi.org files.pythonhosted.org
Improve this answer

Comments

1

Thank you for the solution. In my case the file %appdata%\pip\pip.ini was not present. I created it manually with this content:

[global] trusted-host = pypi.python.org files.pythonhosted.org pypi.org pypi.io
Improve this answer

1 Comment

In my case it needed to be created here: C:\Users\MY-USER\AppData\Local\Programs\Python\Python310\pip.ini
1

The following command worked for me (install a package using pip)

pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package_name>
Improve this answer

Comments

0

You can try sudo apt-get upgrade to get the latest packages. It fixed the issue on my machine.

Improve this answer

Comments

0

It looks like they are also using pypi.org now. I added the following to %appdata%\pip\pip.ini and was able to download my packages from behind an HTTPS-intercepting proxy:

trusted-host = pypi.python.org files.pythonhosted.org pypi.org

Improve this answer

Comments

0

I had same issue. I was trying to install mysqlclient for my Django project.

In my case the system date/time wasn't up-to date (Windows 8). That's causing the error. So, updated my system date time and ran the command pip install mysqlclient again. And it did the work.

Hope this would be helpful for those people who're executing all the commands out there (suggesting in other answers) without checking their system date/time.

Improve this answer

Comments

0

Pretty unique case here, but having Fiddler running (not even targeting the same process) gave me the same SSL errors. Running pip install with --verbose showed an error with Fiddler, closing Fiddler immediately fixed the issue.

Improve this answer

Comments

0

Strange that nobody came up with this in so many years?

I debugged a Python 2 legacy setup and ran into the same error, just with some other package:

Download error on http://www.pylonshq.com/download/: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed -- Some packages may not be found! Not found: ['http://www.pylonshq.com/download/', and here were also my internal server links with legacy packages in the list] ... Couldn't find index page for 'PasteDeploy' (maybe misspelled?) No local packages or download links found for PasteDeploy Traceback (most recent call last): File "setup.py", line 50, in """, File "/usr/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 239, in init self.fetch_build_eggs(attrs.pop('setup_requires')) File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 264, in fetch_build_eggs replace_conflicting=True File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in resolve dist = best[req.key] = env.best_match(req, ws, installer) File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in best_match return self.obtain(req, installer) # try and download/install File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in obtain return installer(requirement) File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 314, in fetch_build_egg return cmd.easy_install(req) File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 616, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 646, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 834, in install_eggs return self.build_and_install(setup_script, setup_base) File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1040, in build_and_install self.run_setup(setup_script, setup_base, args) File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1028, in run_setup raise DistutilsError("Setup script exited with %s" % (v.args[0],)) distutils.errors.DistutilsError: Setup script exited with error: Could not find suitable distribution for Requirement.parse('PasteDeploy')

Clearly, that download website is offline, and easy_install and eggs are deprecated. But you can still learn from this, I guess: you need to put the needed package in the setup_requires=[] list, not in the install_requires=[], and only then, it found the internal packages server with the legacy packages (you can also take an open packages server here).

The package was at

install_requires=["PasteDeploy>=1.0.0", ...],

but setuptools needed the entry instead at:

setup_requires=["PasteDeploy>=1.0.0, ...],

For some packages, it is not enough to put them in the install_requires=[] list since they are already needed during the setuptools setup which is needed before the install.

(Only setuptools should not be in there, see Should setuptools be in the setup_requires entry of setup.cfg files?).

Improve this answer

Comments

0

Below are the two ways to solve ssl certificate verification failed error-

(A) Method 1 (Secure method)-

  1. Find your python installation directory using "where python" cmd in Windows or "which python" cmd in Linux

  2. Add the below three locations in the Path Environment variable

    (a) ...\python-install-directory\ (it contains python.exe file)

    (b) ...\python-install-directory\Scripts (it contains pip.exe file)

    (c) ...\python-install-directory\Library\bin (it contains openssl.exe, libcrypto-1_1-x64.dll, libssl-1_1-x64.dll file)

Try if SSL certificate error is solved now, if it is coming even after following the above instructions then your firewall setting is messing up with the SSL certificate verification. Maybe you are behind a corporate firewall. Try the below steps to solve this-

  1. Download Wheel file of pip-system-certs (https://pypi.org/project/pip-system-certs/) and wrapt (https://pypi.org/project/wrapt/)
  2. pip install these wheel files. Installing these packages will instruct Python to use your System Certificate Store instead of the built-in store in the certifi package

SSL error may come intermittently, just try to pip install again

(B) Method 2 (Unsecure method)-

  1. If you are installing python packages using conda you can run below command before installing the packages

    conda config --set ssl_verify false

  2. If you are using pip to install the packages then use below command

    pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org <package_name>

Improve this answer

Comments

0

Another method is to point pip to your collection of root certificates using the command below. This is the preferred method instead of just simply trusting a hostname.

pip config set global.cert "c:/Temp/Zscaler.crt"

You can dump the certificate chain of a server using OpenSSL with this command:

openssl s_client -showcerts -connect stackoverflow.com:443
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.