48

I'm looking for a method to use pip or similiar to install a list of python packages to a custom target directory (ex./mypath/python/pkgs/ ), but also exclude/blacklist specific dependencies.

I want to exclude specific dependencies since they are already met from a different install path (e.g. an anaconda install). I don't have the privilege of adding packages to the default python installation (nor do I want to).

I'm currently use the -r and -t options of pip. But have not found a way to exclude specific packages.

A pip command like this is would be ideal:

pip install --log pip.log -r req.txt -t /mypath/pypkgs/ --exclude exclude.txt

--no-deps is not an option since I need some of the dependencies.

I'm currently pursuing a python script to do pip installs that include dependencies I don't need via:

pip install --log pip.log -r req.txt -t /mypath/python/pkgs/

and then (automatically) remove the unneeded dependencies after the pip install finishes.

I hoping some combination of pip commands can achieve what I'm looking for some straightforward away. I'm using pip 7.1.2. Thanks!

Similar, yet I'm not upgrading and want to specify a target path:

pip: upgrade package without upgrading particular dependency

Improve this question
2
  • So you want to use pip to install packages for an interpreter other than the one you're using to run it? Commented Oct 30, 2015 at 16:52
  • I'm not sure I'm answering your question @jonrsharpe: When I use pip from my default install (anaconda) with -t [targetdir] it appears to ignore the fact that I already it (For instance, It will install numpy again into [targetdir]). I want to add extra packages in my target directory without duplicate dependencies for my default install/interpreter. I plan to add the custom path to my PYTHONPATH. Commented Oct 30, 2015 at 17:26

5 Answers 5

Reset to default
33

Faced with a similar problem, and using a running bash shell I managed to exclude specific packages with

pip install $(grep -ivE "pkg1|pkg2|pkg3" requirements.txt)

where pkg1 and so on are the names of the packages to exclude.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

In case you made some comments then you would have to remove them as well: pip install $(grep -ivE "^pkg1|^pkg2|^#|^\s*$" requirements.txt | sed s/#.*//g)
But if your requirements.txt also includes specific version requirements it doesn't work. in you requirements.txt it will look like pip == 20.0.2 but when calling it with the above command it should look like pip install 'pip==20.0.2'. Any idea how to solve that problem?
I just found out this one works just fine! pip install -r <(sed '/^[pk1|pkg2]/d' requirements.txt)
This also doesn't work if requirements.txt includes another file ("-r morerequirements.txt").
19

I think this essentially can be achieved in several steps, assuming you're using virtualenv or similar...

  • If you first do a normal pip freeze > requirements.txt you'll get all of the transitive dependencies (e.g. not excluding anything.)

  • Now edit requirements.txt, remove the packages you want to exclude...

  • Finally, in a new environment do pip install -r requirements.txt -t ... --no-deps. Voila: you've installed the dependencies you wanted while excluding specific ones.

Improve this answer

3 Comments

Is it possible to share virtualenv with other users? My use case involves pointing colleagues to a path they can be appended to their PYTHONPATH environment variable, and we're done. The goal being a standard/default python environment we can all share (containing custom and 3rd party modules). This custom path will build upon a vanilla anaconda installation. (OS: Linux).
You mean like a network share with pre-installed binaries? That smells like a bad idea but I suppose it would work as long as everyone has access to the same path. What's wrong with doing pip install?
Right. We desire to use a shared network path so we have a standard environment(s) to share among the team - in addition to anaconda. We are in a firewalled environment - so pip install isn't an option (We'll use pip outside the firewall, and then transfer the environment inside the firewall so we can all share. That seems easier to manage than each potentially having a different environment). Going back to original question, because of anaconda we want to avoid duplicate dependencies (numpy) , however pip -t /path/ doesn't consider the current environment.
10

You can use pip install --no-deps with pip check to exclude specific packages. A real example of mine is when I installed paddleocr on Jetson Nano, pip kept installing python-opencv for me, which has already installed by myself without using pip, but pip cannot detect it. To stop pip installing python-opencv for me, here are the steps

  1. use pip install --no-deps paddleocr to install paddleocr without its dependencies
  2. use pip check to list unresolved dependencies, reformat the output so that it can be read by pip install -r, then remove the packages you don't want to install (it is python-opencv in my case), and save it to a file named fix-deps.txt
  3. use pip install --no-deps -r fix-deps.txt to install unresolved dependencies
  4. repeat steps 2 and 3 until the output of pip check only contains the packages you don't want to install.

pip install --no-deps and pip check are very useful commands that allow you to resolve the dependencies by yourself when pip cannot do it right for you. The shortcoming of this solution is the output of pip check is designed for humans, it cannot be used by pip install -r directly, so you have to reformat the output manually or use the awk command.

PR and issue have been created for the pip community to make the output of pip check suitable for the pip install to read, but for some reason, they don't think it is a good idea. so I guess this is the best we can do now.

Update:

You can also use johnnydep to list dependencies recursively. So the following one-liner should works

johnnydep paddleocr --output-format pinned | grep -v opencv > pkgs.txt && pip install --no-deps -r pkgs.txt

Refs:

  1. https://github.com/pypa/pip/pull/10108
  2. https://github.com/pypa/pip/issues/10066#issuecomment-872638361
Improve this answer

3 Comments

I needed to install pubnub but running pip install pubnub was not working with an error for pycryptodomex. It said that the hashes are incorrect. So I followed the above answer to install without dependendencies pip install --no-deps pubnub and then installed installed the missing dependencies found using pip check, except for pycryptodomex. For pycryptodomex I installed the previous stable version so that I don't get the same error with mismatched hashes. This answer worked for me. Thanks!
@Sashi Great to know it works for you, but maybe pip install pubnub pycryptodomex==x.y.z would be an easier solution for your issue.
Thanks! I miss the easy solutions so often! I will remember this for the next time I need to do something like this again.
6

An approach that takes into account sub-dependencies is to first install the exclude.txt environment, then the req.txt environment, then check the diff and finally install that into the target directory.

Example using virtualenv that will work on GNU/Linux:

virtualenv tmpenv source tmpenv/bin/activate pip install -r exclude.txt pip freeze > exclude-with-deps.txt pip install -r req.txt pip freeze > req-with-deps.txt comm -13 exclude-with-deps.txt req-with-deps.txt > final-req.txt pip install -r final-req.txt --no-deps -t pypkgs
Improve this answer

Comments

0

Quick Answer:
pip freeze | grep -vFxf pip_ignore.txt > requirements.txt
Will ignore whatever you save in pip_ignore.txt

This is good if you already know what you're going to ignore. However if you're about to install something that you know you'll want to ignore, you're going to want to be able to handle things a bit more easily. In that scenario what I do is pip freeze > before.txt then I install whatever packages it is, and then pip freeze | grep -vFxf before.txt > new_stuff_to_ignore.txt. You now have a list of the stuff you just installed. You can add it to your pip_ignore.txt and then finally do pip freeze | grep -vFxf pip_ignore.txt > requirements.txt

Be mindfudl that some packages in your requirements may share dependencies with packages you want to ignore. These dependencies might end up in your pip_ignore.txt.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.