109

I am creating a setup.py file for a project which depends on private GitHub repositories. The relevant parts of the file look like this:

from setuptools import setup setup(name='my_project', ..., install_requires=[ 'public_package', 'other_public_package', 'private_repo_1', 'private_repo_2', ], dependency_links=[ 'https://github.com/my_account/private_repo_1/master/tarball/', 'https://github.com/my_account/private_repo_2/master/tarball/', ], ..., )

I am using setuptools instead of distutils because the latter does not support the install_requires and dependency_links arguments per this answer.

The above setup file fails to access the private repos with a 404 error - which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can't figure out how to make setuptools authenticate.

Here are some things I've tried:

  1. Use git+ssh:// instead of https:// in dependency_links as I would if installing the repo with pip. This fails because setuptools doesn't recognize this protocol ("unknown url type: git+ssh"), though the distribute documentation says it should. Ditto git+https and git+http.

  2. https://<username>:<password>@github.com/... - still get a 404. (This method doesn't work with curl or wget from the command line either - though curl -u <username> <repo_url> -O <output_file_name> does work.)

  3. Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice.

Currently I just have setup.py print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there's something obvious that I'm missing, but can't think what it might be. :)

Duplicate-ish question with no answers here.

Improve this question
0

10 Answers 10

Reset to default
107

I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508 standard should be used, from [2] I retrieved an example which actually does work (at least for my case).

Please note; this is with pip 20.0.2 on Python 3.7.4

setup( name='<package>', ... install_requires=[ '<normal_dependency>', # Private repository '<dependency_name> @ git+ssh://[email protected]/<user>/<repo_name>@<branch>', # Public repository '<dependency_name> @ git+https://github.com/<user>/<repo_name>@<branch>', ], )

After specifying my package this way installation works fine (also with -e settings and without the need to specify --process-dependency-links).

References [1] https://github.com/pypa/pip/issues/4187 [2] https://github.com/pypa/pip/issues/5566

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

If you use ssh:// and run into Could not resolve hostname change the : to / in your clone url. I had this error with gitlab.
That doesn't seem to work anymore as setuptools seems to look for a package on PyPi with the dependency name: Reading https://pypi.org/simple/some-fake-name/, and then Couldn't find index page for 'some_fake_name' (maybe misspelled?). In the end, the last error displayed is error: Could not find suitable distribution for Requirement.parse('some_fake_name@ git+ssh://[email protected]/cglacet/quadtree.git')
Emphasis that <dependency_name> is not the same as the <repo_name> AND you may replace 'ssh://git@' with 'https://' if it is a public repo
@Anusha Sorry, I am not able to reproduce cglacet's error. His/her repository is not private and I expect Phil's suggestion to resolve it. Have you tried Phil's suggestion?
@TomHemmes I am trying to install a local package, so in install_requires I have <package-name> @ file://localhost/lib/<package-name>/<package_name>.version.whl and I get the same error as @cglacet
|
45

Here's what worked for me:

 install_requires=[ 'private_package_name==1.1', ], dependency_links=[ 'git+ssh://[email protected]/username/private_repo.git#egg=private_package_name-1.1', ]

Note that you have to have the version number in the egg name, otherwise it will say it can't find the package.

Improve this answer

8 Comments

Hi vadimg - Which version of setuptools/distribute are you using? I get "Unknown url type: git+ssh" using distribute 0.7.3 (the latest version).
This did not work for me 3 years later... Got: Could not find a version that satisfies the requirement
As of Oct 28, 2016, this approach seems broken
With modern pip's you need to include an option, something like pip install --process-dependency-links ...
--process-dpendency-links is deprecated, see my answer using PEP508 url specification
|
11

I couldn't find any good documentation on this, but came across the solution mainly through trial & error. Further, installing from pip & setuptools have some subtle differences; but this way should work for both.

GitHub don't (currently, as of August 2016) offer an easy way to get the zip / tarball of private repos. So you need to point setuptools to tell setuptools that you're pointing to a git repo:

from setuptools import setup import os # get deploy key from https://help.github.com/articles/git-automation-with-oauth-tokens/ github_token = os.environ['GITHUB_TOKEN'] setup( # ... install_requires='package', dependency_links = [ 'git+https://{github_token}@github.com/user/{package}.git/@{version}#egg={package}-0' .format(github_token=github_token, package=package, version=master) ]

A couple of notes here:

  • For private repos, you need to authenticate with GitHub; the simplest way I found is to create an oauth token, drop that into your environment, and then include it with the URL
  • You need to include some version number (here is 0) at the end of the link, even if there's no package on PyPI. This has to be a actual number, not a word.
  • You need to preface with git+ to tell setuptools it's to clone the repo, rather than pointing at a zip / tarball
  • version can be a branch, a tag, or a commit hash
  • You need to supply --process-dependency-links if installing from pip
Improve this answer

2 Comments

I am getting a cannot find tag or branch message. Despite the fact that the private repo I am attempting to clone does have a tag.
figured out what the problem was. The tag had a v prepended to it in github. So I needed to use v1.1.0 instead of 1.1.0 in my setup.py script.
5

I found a (hacky) workaround:

#!/usr/bin/env python from setuptools import setup import os os.system('pip install git+https://github-private.corp.com/user/repo.git@master') setup( name='original-name' , ... , install_requires=['repo'] )

I understand that there are ethical issues with having a system call in a setup script, but I can't think of another way to do this.

Improve this answer

3 Comments

yes this was also an ugly workaround for us due to the following: github.com/pypa/pip/issues/2822
This is the only way I could get it to work, although I went with import pip. Neither @vadimg's answer or this suggestion in pypa/pip worked.
This will install a dependency even if running something unrelated to installation like python setup.py --version.
4

Via Tom Hemmes' answer I found this is the only thing that worked for me:

 install_requires=[ '<package> @ https://github.com/<username>/<package>/archive/<branch_name>.zip']
Improve this answer

Comments

1

Using archive URL from github works for me, for public repositories. E.g.

dependency_links = [ 'https://github.com/username/reponame/archive/master.zip#egg=eggname-version', ]
Improve this answer

1 Comment

How do you find the eggname-version?
1

With pip 20.1.1, this works for me

install_requires=[ "packson3@https://tracinsy.ewi.tudelft.nl/pubtrac/Utilities/export/138/packson3/dist/packson3-1.0.0.tar.gz"],

in setup.py

Improve this answer

Comments

0

Here is another way (free):

  1. Clone GitHub Repo to GitLab Repo (free account) link
  2. Add GitLab Repo Access Key (read only) [repository -> settings -> access tokens]
  3. Use as dependency: https://oauth2:GITLAB_ACCESS_TOKEN>@gitlab.com/<GITLAB_USERNAME>/<PROJECT_NAME>.git

Q&A:

Q: Why through GitLab? A: Because GitHub only has Account Access Keys, GitLab has Repository Access Keys!

Q: Access Tokens in git history? A: Yes, because the whole point is to use a privat repo in privat context, the token is read-only, and the token expires after maximum of 1 year.

Q: Why is the GitLab Repository not synced automatically with the GitHub Repo? A: Mirroring is a GitLab premium feature, but you can do it for free if you use GitHub Actions, here is a tutorial.

Improve this answer

Comments

-1

Edit: This appears to only work with public github repositories, see comments.

dependency_links=[ 'https://github.com/my_account/private_repo_1/tarball/master#egg=private_repo_1', 'https://github.com/my_account/private_repo_2/tarball/master#egg=private_repo_2', ],

Above syntax seems to work for me with setuptools 1.0. At the moment at least the syntax of adding "#egg=project_name-version" to VCS dependencies is documented in the link you gave to distribute documentation.

Improve this answer

3 Comments

I still get the same 404 error. Are the repos in your case private? I am aware of the #egg= syntax but don't know that it affects authentication.
Sorry, it of course was a public repo project. There probably is not any way to do this with current setuptools if https//<username>:<password>@... tarball URLs don't work with githubs private repositories as this question seems also to indicate: having-trouble-downloading-git-archive-tarballs-from-private-repo
You could try also development version of setuptools and/or check the code. The relevant code seems to be: package_index:fetch_distribution():534 package_index.py:_download_url():736
-2

This work for our scenario:

  1. package is on github in a private repo
  2. we want to install it into site-packages (not into ./src with -e)
  3. being able to use pip install -r requirements.txt
  4. being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt

https://github.com/pypa/pip/issues/3610#issuecomment-356687173

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.