0

There are two apps that have a few modules in common, like this:

/apps /app1 /.ve /requirements.txt /app2 /.ve /requirements.txt /modules /module1 /requirements.txt /module2 /requirements.txt /module3 /requirements.txt /module4 /requirements.txt
  • app1 depends on module1 and module3 and uses virtualenv
  • app2 depends on module2 and module3 and uses virtualenv
  • module3 depends on module4
  • modules are like 3rd party packages, but are under an active local development. They will only ever be executed by an application, so they do not have their own virtualenv
  • each app and each module has it's own requirements.txt

How do I "correctly" wire this thing up? The goal is to be able to install whole dependency graph for each application in /apps using a single command (like pip install -r requirements.txt, no custom scripts prefferably)

Idea #1:

  • add /modules to PYTHONPATH
  • use -r ../modules/module1/requirements.txt syntax in requirements.txt for each app

The problem: PIP will not work if any requirement is sepcified in both requirements.txt files (https://github.com/pypa/pip/issues/2367). If both app1 and module1 depends on django, I need to be able to make this explicit in their corresponding requirements.

Idea #2:

  • Create an egg/wheel/package/whatever for each module under /modules, and reference them in each app's requirements.txt

The problem: Creating eggs just for the sake of making pip behave properly is a lot of work. Plus development would be harder - I would have to reinstall each egg every time something changes.

Improve this question
8
  • Why do you need to do it with a single pip install? If you call pip install separately for each app it should just install what it needs, leaving any already-installed dependencies in place. Commented Feb 15, 2015 at 19:03
  • I mean I want to use a single pip install per application - one for app1 and one for app2. I updated my question to indicate that I want to put each app in a separate virtualenv Commented Feb 15, 2015 at 19:04
  • I guess I don't know enough about pip internals on this matter. Are you saying the requirement.txt in the apps directly reference the requirements.txt of the modules? If so, can you just instead specify the modules as normal dependencies and let pip install them normally? Commented Feb 15, 2015 at 19:10
  • I notice that the pip documentation says: "Requirement files are mostly flat. Maybe MyApp requires Framework, and Framework requires Library. I encourage you to still list all these in a single requirement file; it is the nature of Python programs that there are implicit bindings directly between MyApp and Library." This suggests that the "supported" way is to have the apps' requirements.txt directly reference any dependencies of the modules. Commented Feb 15, 2015 at 19:16
  • I can reference a module directly in requirements.txt but pip install will just copy the files - it makes the module development harder. Commented Feb 15, 2015 at 19:18

1 Answer 1

Reset to default
1

If you are developing these modules, give them a proper setup.py and install them in development mode to both virtual envs, i.e. with the module activated, python setup.py develop the required modules the application virtualenvs.

You could also install both applications in the same virtualenv, if they do not clash.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks! This is already something! Now, let's assume I add a new dependency to module3/requirements.txt - how do I install them? pip install --upgrade will remove .egg-link created by setuptools and just copy the module files; Of course I can just manually perform pip install dependency every time but I hope there is some automated way to do this
No, you should add them in the setup.py requirements and upgrade dependencies with setup.py develop.
pip.pypa.io/en/latest/user_guide.html#requirements-files - I consider the requirements.txt a secondary feature over the setup.py install_requires, used to freeze the whole virtual env contents and package conflicts.
Okay one last issue with that - how do I explicitly specify module4 as a dependency of module3? install_requires + dependency_links doesn't exactly work even with --process-dependency-links
I guess I can do setup.py develop on each dependency for each virtualenv, but this way I become a dependency solver. There must be a better way

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.