I've just started working with setuptools and virtualenv. My package requires the latest python-gearman that is only available from GitHub. The python-gearman version that's on PyPI is an old one. The Github source is setuptools-compatible, i.e. has setup.py, etc. Is there a way to make setuptools download and install the new version instead of looking for it on PyPI and installing the old one?
FYI, the new python-gearman is http://github.com/mtai/python-gearman
The key is to tell easy_install where the package can be downloaded. In this particular case, it can be found at the url http://github.com/mtai/python-gearman/tarball/master. However, that link by itself won't work, because easy_install can't tell just by looking at the URL what it's going to get.
By changing it to http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta instead, easy_install will be able to identify the package name and its version.
The final step is to add the URL to your package's dependency_links, e.g.:
setup( ... dependency_links = ['http://github.com/mtai/python-gearman/tarball/master#egg=gearman-2.0.0beta'] ) Now, when YOUR package is being installed, easy_install will discover that there is a "gearman 2.0.0beta" available for download from that URL, and happily pick it over the one on PyPI, if you specify "gearman>=2.0.0beta" in your dependencies..
(Normally, the way this sort of thing is done is to include a link on one's PyPI page to the downloadable source; in this case, if the author of the gearman package had included a link like the above, you'd be already set. Typically, people mark the development version with 'myproject-dev' and then people use a requirement of 'myproject>=somever,==dev', so that if there isn't a package of somever or higher, easy_install will try to check out or download the release.)
You'll need to specify --process-dependency-links when using pip. Note that dependency links processing has been deprecated and will be removed in a future release.
beta suffix on an existing version on PyPI, it will still install the package from PyPI instead of the one defined in dependency_links. If you try to set a higher version than what exists on PyPI with #egg=package-version, the setup tool will complain with a Could not find a version that satisfies the requirement error and a list of all versions available on PyPI. Note that I'm trying to build my package with sdist, then installing it with pip install http://url/to/my/generated/tar.easy_install http://url/to/my/generated/tar, everything works like expected... Any idea why ?
You can use the pip install protocol+location[@tag][#egg=Dependency] format to install directly from source using pip.
pip install git+https://github.com/username/repo.git pip install git+https://github.com/username/repo.git@MyTag pip install git+https://github.com/username/repo.git@MyTag#egg=ProjectName pip install hg+https://hg.myproject.org/MyProject/ pip install svn+svn://svn.myproject.org/svn/MyProject pip install bzr+http://bzr.myproject.org/MyProject/trunk The following protocols are supported: [+git, +svn, +hg, +bzr]
@tag lets you specify a specific version/tag to check out.
#egg=name lets you specify what the project is as a dependency for others.
The order must always be @tag#egg=name.
You can also install from private repositories by changing the protocol to SSH (ssh://) and adding an appropriate user (git@):
git+ssh://[email protected]/username/my_private_repo You can also install from private repositories with a username / password.
git+https://<username>:<password>@github.com/<user>/<repo>.git Github provides the ability to create personal OAuth tokens which can be cycled
git+https://<oauth token>:[email protected]/<user>/<repo>.git requirements.txt is used to specify project dependencies:
requirements.txt
package1 package2==1.0.2 package3>=0.0.4 git+https://github.com/username/repo.git These are not installed automatically with the package and must be installed with the command pip -r requirements.txt.
Requirements files can include other requirements files:
requirements-docs.txt
sphinx -r requirements-dev.txt requirements-dev.txt
some-dev-tool -r requirements.txt requirements.txt
package1 package2==1.0.2 package3>=0.0.4 git+https://github.com/username/repo.git Requirements files can install dependencies specified in setup.py with the following command:
-e . setup.py can also install from repositories using the same syntax as above, but using the dependency_links value as mentioned in this answer.
https://pip.pypa.io/en/latest/user_guide.html#installing-packages https://pip.pypa.io/en/latest/reference/pip_install.html
extras_require={'all': [repo @ git+https://github.com/username/repo.git]}As I just had to do the same thing, I found another way to do this as pip's --process-dependency-links are scheduled to be removed in pip 19.0 according to this comment.
pip 18.1 includes the following feature
Allow PEP 508 URL requirements to be used as dependencies.
From the description of PEP 508, the syntax for such URL dependencies looks like:
A minimal URL based lookup:
pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
So in your setup.py it would look like
setup( ... install_requires = [ ... 'python-gearman @ https://github.com/mtai/python-gearman/archive/master.zip' ... ] ) Notice, the link is an archive file and could also be a specific release or branch of a repository as described in this answer. Also, see that answer for working with other repository hosts.
To the best of my knowledge, the easiest way to update the dependency is by using pip install -I . when installing your package from its directory.
pip install supports, such as git URLs, #subdirectory=... etc? Or did they come up with a brand new syntax with different features exposed in a different and incompatible way?
distutils.core.setup and pip 19.1.1 requirements.txt, however, is not compatible with pip install -r requirments.txtrequirments.txt are you referring to?Vanilla setuptools does not support downloading directly from a git repository but you can use one of the Download Source links from that page, like:
easy_install http://github.com/mtai/python-gearman/tarball/master 
pip or buildout which have more sophisticated requirements management. See, for example: pip.openplans.org/#requirements-files
python setup.py installin the source directory?easy_installorpipto install it straight from Github. But there's also another solution, have you considered adding the package to PyPI?buildout? It has a couple of ready-made Git plugins.