76

I found that pip only use single core when it compiles packages. Since some python packages takes some time to build using pip, I'd like to utilize multicore on the machine. When using Makefile, I can do that like following command:

make -j4

How can I achieve same thing for pip?

Improve this question
4
  • possible duplicate of Parallel Pip install Commented Oct 7, 2014 at 7:02
  • 6
    I don't think this is duplicated question. In "Parallel Pip install" thread, if I understood correctly, the author wants to run multiple pip install processes. In contrast, what I want to do is install a package using multiple cores. Commented Oct 7, 2014 at 7:10
  • 1
    as far as I know this is not implemented. The command build_ext has to be customized to achieve this. Commented Oct 23, 2014 at 18:47
  • 2
    Silly question; would it be possible to do export MAKEFLAGS=-j5 prior to running pip for pip packages that build C pieces or does pip not (generally) call out to make for those cases (I haven't dug deep enough to know) Commented Oct 30, 2014 at 14:48

5 Answers 5

Reset to default
51

The Ultimate Way to Resolve This Problem

Because all the c / cpp files would be compiled by using make commend, and make has an option which specify how many cpu cores shoule be used to compile the source code, we could do some tricks on make.

  1. Backup your original make command:

    sudo cp /usr/bin/make /usr/bin/make.bak

  2. write a "fake" make command, which will append --jobs=6 to its parameter list and pass them to the original make command make.bak:

    make.bak --jobs=6 $@

So after that, not even compile python with c libs, but also others contain c libs would speed up on compilation by 6 cores. Actually all files compiled by using make command will speed up.

And good luck.

Use: --install-option="--jobs=6" (pip docs).

pip3 install --install-option="--jobs=6" PyXXX

I have the same demand that use pip install to speed the compile progress. My target pkg is PySide. At first I use pip3 install pyside, it takes me nearly 30 minutes (AMD 1055T 6-cores, 10G RAM), only one core take 100% load.

There are no clues in pip3 --help, but I found lots of options like pip install -u pyXXX, but I didn't know what is '-u' and this parameter was not in pip --help too. I tried 'pip3 install --help' and there came the answer: --install-option.

I read the code of PySide's code and found another clue: OPTION_JOBS = has_option('jobs'), I put ipdb.set_trace() there and finally understand how to use multicore to compile by using pip install.

it took me about 6 minutes.

--------------------------update------------------------------

as comment below, I finally used tricks like this: cd /usr/bin sudo mv make make.bak touch make then edit make: vim make or other way you like and type this: make.bak --jobs=6 $* I'm not familiar with bash, so I'm not sure if this is the correcct bash code. I'm writing this comment in windows. The key is rename make into make.bak, and then create a new make, use this new make to call make.bak with added param --jobs=6

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

Seems to work -- the processor load increases on all of my cores when I run it this way.
I get warning UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.. It makes sense, but is it likely that the performance benefits of using multiple cores outweighs the performance hit of not using wheels? What does it depend on?
What about for pip (not pip3)? I'm getting: error: option --jobs not recognized
It appears that the parameter passed to --install-option is simply forwarded to the setup.py for the package you're installing. As such, the --jobs=6 is specific to PySide's build process. Other packages may have similar (or the same) build flag, but it's not a generic option.
There's no need to move system files or fool around with /usr/bin at all. Just add the script to /usr/local/bin/ (or ~/.local/bin/) and reference /usr/bin/make instead of /usr/bin/make.bak. I believe this is much cleaner and won't get nuked by system updates.
|
41

Tested this works https://stackoverflow.com/a/57014278/6147756

Single command:

MAKEFLAGS="-j$(nproc)" pip install xxx

Enable for all commands in a script:

export MAKEFLAGS="-j$(nproc)"
Improve this answer

4 Comments

Hi, Yushu. Please link to the answer you are referring to instead of the question. You can click “share” under an answer to obtain a direct link.
This is by far the simplest answer if you're using a script to call pip with.
This is a much better solution than the accepted solution. I would advise against replacing the system-wide make with a script that includes the argument --jobs=6. At the very least, this hack will stop working if a new version of make is included in an update.
As of 2023 with Python 3.10, this is what worked!
8

From what I can tell it does not look like pip has this ability but I may be mistaken.

To do multiprocessing in python you use the multiprocessing package, [here is a guide I found] (http://pymotw.com/2/multiprocessing/basics.html) about how to do it if you are interested and this is a link to the python docs that talk about it. I also found this question useful, Multiprocessing vs Threading Python, to make sure that multiprocessing did what I thought it did, being take advantage of multiple CPUs.

I have gone through the pip source code (available here) looking for a reference to the multiprocessing package and did not find any use of the package. This would mean that pip does not use/support multiprocessing. From what I can tell the /pip/commands/install.py file is the one of interest for your question as it is called when you run pip install <package>. For this file specifically the imports are 

from __future__ import absolute_import import logging import os import tempfile import shutil import warnings from pip.req import InstallRequirement, RequirementSet, parse_requirements from pip.locations import virtualenv_no_global, distutils_scheme from pip.basecommand import Command from pip.index import PackageFinder from pip.exceptions import ( InstallationError, CommandError, PreviousBuildDirError, ) from pip import cmdoptions from pip.utils.deprecation import RemovedInPip7Warning, RemovedInPip8Warning

which you can see does not have any reference to the multiprocessing package but I did check all of the other files just to be sure.

Furthermore, I checked the pip install documentation and found no reference to installing using multiple cores.

TL;DR: Pip doesn't do what you are asking. I may be wrong as I didn't look at the source that long but I'm pretty sure it just doesn't support it.

Improve this answer

3 Comments

That's a plausible argument although I'm a little bit disappointed that pip does not support this feature... Thank you very much for your analysis.
This answer is outdated. See Plasmatium's answer below.
GCC and make are not bound by the GIL anyway and never were.
0

On some shared hosting environments build cores are limited and so pip install uwsgi fails to build the package from sources. In that particular case of uWSGI build cores can be limited/set with CPUCOUNT env variable that is specific to a build system of uWSGI:

CPUCOUNT=1 pip install uwsgi
Improve this answer

Comments

0 
MAX_JOBS=1 pip install ... --verbose

You will see this line:

Using envvar MAX_JOBS (1) as the number of workers...

Only one job is not necessary. Usually, 4~8 is fine.

Too many jobs could lead to the error of Killed signal terminated program cc1plus.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.