7

I know that intall the scrapy should install the w3lib first,so I install the w3lib firstly,but when I import the scrapy in python ide,the program is crashed. the error:

creating Twisted.egg-info writing requirements to Twisted.egg-info\requires.txt writing Twisted.egg-info\PKG-INFO writing top-level names to Twisted.egg-info\top_level.txt writing dependency_links to Twisted.egg-info\dependency_links.txt writing manifest file 'Twisted.egg-info\SOURCES.txt' warning: manifest_maker: standard file '-c' not found reading manifest file 'Twisted.egg-info\SOURCES.txt' writing manifest file 'Twisted.egg-info\SOURCES.txt' copying twisted\internet\_sigchld.c -> build\lib.win-amd64-2.7\twisted\internet creating build\lib.win-amd64-2.7\twisted\internet\iocpreactor\iocpsupport copying twisted\internet/iocpreactor/iocpsupport\iocpsupport.c -> build\lib.win-amd64-2.7\twisted\internet/iocpreactor/i ocpsupport copying twisted\internet/iocpreactor/iocpsupport\winsock_pointers.c -> build\lib.win-amd64-2.7\twisted\internet/iocpreac tor/iocpsupport copying twisted\python\_epoll.c -> build\lib.win-amd64-2.7\twisted\python copying twisted\python\_initgroups.c -> build\lib.win-amd64-2.7\twisted\python copying twisted\python\sendmsg.c -> build\lib.win-amd64-2.7\twisted\python copying twisted\runner\portmap.c -> build\lib.win-amd64-2.7\twisted\runner copying twisted\test\raiser.c -> build\lib.win-amd64-2.7\twisted\test running build_ext

What's wrong?

Improve this question

3 Answers 3

Reset to default
13

This is how I installed scrapy on ubuntu:

sudo apt-get update sudo apt-get install python-pip build-essential python-dev libxslt-dev libxml2-dev sudo -H pip install Scrapy scrapy version

The important thing that solved my issues was sudo -H pip install Scrapy specifically the -H flag.

I also exited out of the terminal and started a new terminal to ensure the all the environment variables were set correctly

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

12

Make sure you had installed the Twisted, pyOpenSSL and pycrypto. These are my steps to install scrapy on ubuntu. 1.install gcc and lxml:

sudo apt-get install python-dev sudo apt-get install libevent-dev sudo apt-get install libxml2 libxml2-dev apt-get install libxml2-dev libxslt-dev apt-get install python-lxml

2.install twisted:

sudo apt-get install python-twisted python-libxml2 python-simplejson sudo apt-get install build-essential libssl-dev libffi-dev python-dev

3.install pyOpenSSL:

wget http://pypi.python.org/packages/source/p/pyOpenSSL/pyOpenSSL-0.13.tar.gz tar -zxvf pyOpenSSL-0.13.tar.gz cd pyOpenSSL-0.13 sudo python setup.py install

4.install pycrypto

wget http://pypi.python.org/packages/source/p/pycrypto/pycrypto-2.5.tar.gz tar -zxvf pycrypto-2.5.tar.gz cd pycrypto-2.5 sudo python setup.py install

5.install easy_install:(if you don't have easy_install)

wget http://peak.telecommunity.com/dist/ez_setup.py python ez_setup.py

6.install w3lib

sudo easy_install -U w3lib

7.install scrapy

sudo easy_install Scrapy

If you wanna know much,please goto my blog.

Improve this answer

2 Comments

I install the lxml failed when I run the command apt-get install python-lxml
oh,my god,I forget that you should use the sudo,for example:sudo apt-get install python-lxml.
0

First install system dependencies sudo apt-get install -y \ python-dev python-pip python-setuptools \ libffi-dev libxml2-dev libxslt1-dev \ libtiff5-dev libjpeg62-turbo-dev zlib1g-dev libfreetype6-dev \ liblcms2-dev libwebp-dev tcl8.5-dev tk8.5-dev python-tk

Then add followings in your requirements.txt

lxml pyOpenSSL Scrapy Pillow

And finally pip install -r requirements.txt

You can look around gist.github.com as well to resolve latest dependencies issues. I'm using docker to setup scrapy deps in a separate container. I've created one for mine needs here

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.