34

I am using numpy / scipy / pynest to do some research computing on Mac OS X. For performance, we rent a 400-node cluster (with Linux) from our university so that the tasks could be done parallel. The problem is that we are NOT allowed to install any extra packages on the cluster (no sudo or any installation tool), they only provide the raw python itself.

How can I run my scripts on the cluster then? Is there any way to integrate the modules (numpy and scipy also have some compiled binaries I think) so that it could be interpreted and executed without installing packages?

Improve this question
9
  • I'm not really clear on exactly what you're asking--most Python modules are just Python code that you call import on. If that's the issue, and you can't use pip or easy_install, you can just download the packages and stick them right in your Python site-packages, and then just import them. Commented Jan 6, 2013 at 6:39
  • @jdotjdot there're not only python code, but also compiled binary in the modules. Commented Jan 6, 2013 at 6:42
  • 3
    For numpy and scipy that is true. But what do you mean by "can't install packages"? What exactly is it that you're not permitted to install/put on the cluster? Commented Jan 6, 2013 at 6:43
  • @jdotjdot we're not permitted to sudo, and there's not any installation tool either. Commented Jan 6, 2013 at 6:51
  • thanks, that's what I was looking for. Looks like David provided a good answer already. Commented Jan 6, 2013 at 7:15

4 Answers 4

Reset to default
45

You don't need root privileges to install packages in your home directory. You can do that with a command such as

pip install --user numpy

or from source

python setup.py install --user

See https://stackoverflow.com/a/7143496/284795

The first alternative is much more convenient, so if the server doesn't have pip or easy_install, you should politely ask the admins to add it, explaining the benefit to them (they won't be bothered anymore by requests for individual packages).

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

politely ask the admins to add it, why didn't I think of that! OMG, we geeks suck...
@Skyler: asking the admins is definitely the best option! Based on your question I thought they might be opposed to it. (I do still recommend using virtualenvs once pip is installed, which incidentally might also serve as a solution to this other question of yours).
This should have been python setup.py install --user. See pythonhosted.org/setuptools/easy_install.html#id37
12

You could create a virtual environment through the virtualenv package.

This creates a folder (say venv) with a new copy of the Python executable and a new site-packages directory, into which you can "install" any number of packages without needing any kind of administrative access at all. Thus, activating the environment through source venv/bin/activate will give Python an environment that's equivalent to having those packages installed.

I know this works for SGE clusters, although how the virtual environment is activated might depend on your cluster's configuration.

You can try installing virtualenv on your cluster within your own site-packages directory using the following steps:

  1. Download virtualenv from here, put it on your cluster

  2. Install it using setup.py to a specific, local directory to serve as your own site-packages:

    python setup.py build python setup.py install --install-base /path/to/local-site-packages

  3. Add that directory to your PYTHONPATH:

    export PYTHONPATH="/path/to/local-site-packages:${PYTHONPATH}"

  4. Create a virtualenv:

    virtualenv venv
Improve this answer

11 Comments

Thank you! How to install virtualenv without sudo or pip or easy_install?
@Skyler: You can install virtualenv on a system where you do have control, create the virtualenv there, and then transfer the files to the cluster (unless I'm mistaken, and I could be, you don't even need virtualenv to activate the environment, only to create it- the environments is activated using bash scripts). However, that works only if the system you are creating the virtualenv on is of the same OS and architecture as the cluster- you might need to do a lot of customizing.
@Skyler: On second thought, it might be easier to install virtualenv in a local site-packages directory (basically bootstrapping your way up based on what you can do). Try the approach in my edit (I haven't tested it at all so let me know what problems you run into on the way).
Hi David, the step 2 gives error: install-base or install-platbase supplied, but installation scheme is incomplete. Did I miss something?
@Skyler: Sorry- try replacing --install-base with --home
|
1

You can import a module from an arbitrary path by calling:

sys.path.append()

Improve this answer

Comments

1

The Python Distribution Anaconda solves many of the issues discussed in this questions. Anaconda does not require Admin or root access and is able to install to your home directory. Anaconda comes with many of the packages in question (scipy, numpy, sklearn, etc...) as well as the conda installer to install additional packages should additional ones be necessary.

It can be downloaded from https://www.continuum.io/downloads

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.