30

I am pulling my hair out trying to figure this out because I had it working until last week and somehow it broke.

When I setup a virtualenv for a Google App Engine app and start the app with dev_appserver.py, I get errors importing the standard library (like "ImportError: No module named base64").

Here's what I'm doing:

(Using the system Python)

virtualenv --python=python2.5 --no-site-packages ~/.virtualenv/foobar

Then I add the a gae.pth file to ~/.virtualenv/foobar/lib/python2.5/site-packages/ containing the Google App Engine libraries:

/usr/local/google_appengine /usr/local/google_appengine/lib/antlr3 /usr/local/google_appengine/lib/cacerts /usr/local/google_appengine/lib/django /usr/local/google_appengine/lib/fancy_urllib /usr/local/google_appengine/lib/ipaddr /usr/local/google_appengine/lib/webob_1_1_1 /usr/local/google_appengine/lib/yaml/lib

(That's based on this answer.)

Then I source my "foobar" virtualenv and try to start my app with dev_appserver.py.

The server starts but the first request errors out with the aforementioned "ImportError: No module named base64". If I visit the admin console I get "ImportError: No module named cgi".

If I start up python, I can load these modules. 

>>> import base64 >>> base64.__file__ '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/base64.py'

It seems that the SDK's sandboxing is preventing these libraries from getting loaded. But like I said, I had this working until last week...something changed or I inadvertently broke my virtualenv and I can't figure out how I got it working in the first place.

Software versions:

Google App Engine SDK 1.3.7
Mac OS X Snow Leopard 10.6.4
virtualenv 1.5.1

Update: In response to Alan Franzoni's questions:

I am using the system Python that came with Mac OS X. I installed virtualenv via easy_install. I upgraded to virtualenv 1.5.1 today to try to fix the problem.

If I run python /usr/local/bin/dev_appserver.py with the virtualenv python, the problem persists. If I deactivate the virtualenv and run that command with the system python2.5, it works. (Also, I can use the GoogleAppEngineLauncher to start my app.)

Here is a full stack trace (this one uses the Kay framework, but the problem is the same with webapp):

Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 3206, in _HandleRequest self._Dispatch(dispatcher, self.rfile, outfile, env_dict) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 3149, in _Dispatch base_env_dict=env_dict) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 525, in Dispatch base_env_dict=base_env_dict) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2402, in Dispatch self._module_dict) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2312, in ExecuteCGI reset_modules = exec_script(handler_path, cgi_path, hook) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2208, in ExecuteOrImportScript exec module_code in script_module.__dict__ File "/Users/look/myapp/kay/main.py", line 17, in <module> kay.setup() File "/Users/look/myapp/kay/__init__.py", line 122, in setup from google.appengine.ext import db File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate return func(self, *args, **kwargs) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1937, in load_module return self.FindAndLoadModule(submodule, fullname, search_path) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate return func(self, *args, **kwargs) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1839, in FindAndLoadModule description) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate return func(self, *args, **kwargs) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1790, in LoadModuleRestricted description) File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/db/__init__.py", line 81, in <module> import base64 ImportError: No module named base64
Improve this question
5
  • Could you please cut&paste the full output for such error? If you use "python /usr/local/bin/dev_appserver.py" does the error stay the same? Also, how did you install virtualenv, via setuptools? And are you using system python2.5, macports one, any combo of those? Commented Oct 4, 2010 at 22:05
  • Thanks for your response. I've updated the post to answer your questions. Commented Oct 4, 2010 at 23:21
  • Can you type "which python" to ensure you are running the virtualenv Python? Commented Oct 5, 2010 at 4:48
  • and if you run the virtualenv's python interactively, does import base64 succeed? (I've gotten GAE working in a virtualenv on OSX with svn trunk source of GAE, not installing the .dmg. I can post my steps as an answer if it would help, but I just have the helloworld tutorial app right now, not any kind of complex app.) Commented Oct 19, 2010 at 6:15
  • SDK ≥ 1.6.1 + gae.pth is a great way for keeping the SDK in sync on multi-virtualenv setup. By adding setup.py + MANIFEST.in I managed to make it pip installable = 100% automated + made the SDK live in virtualenv no patching required. Commented Mar 25, 2012 at 8:38

5 Answers 5

Reset to default
15

It's an issue 4339 with the GAE SDK, it's confirmed and there are two slightly different patches available in the bug entry that make it work.

What happens is dev_appserver.py sets up a restricted python environment by disallowing access to any non-system-python modules and it does that by calculating the system python folder from the location of the os module. In a virtualenv instance the os.py gets symlinked into the virtualenv but gets compiled straight into virtualenv, and this is the path that dev_appserver uses, effectively blocking access to any module from the system python library that is not linked by virtualend, which is most of them. The solution is to "bless" both paths.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Issue 4339 was just closed as Fixed for SDK version 1.6.1. I tried it and it works for me again!
2

Google AppEngine SDK makes a lot of trick in order to pull its install into sys.path, and these tricks rely on actual file's path. I think there might be a lot of various reasons why it fails. SDK doesn't install itself as a real python package, virtualenv doesn't do complete sandboxing, it just sets up an environment (obviously) and changes sys.path. And GAE SDK does this too, they both intefere, SDK is being developing rapidly and changing often, so this is extremely bumpy road to go.

Probably, it would be better if you'd explain what are you trying to achieve. My guess is that you're trying to create a clean environment to ensure that no 3rd-party module is available to application. If this guess is correct, I'd go with installing GAE SDK into virtualenv via requirements files as described here.

Improve this answer

Comments

1

I think, since you've setup virtualenv with the --no-site-packages option, you need to install the SDK into the environment. --no-site-packages seperates the dev environment you're configuring from any other Python installation on your computer so, as you seem to have it configured you're calling a module that doesn't exist (in the environment) which is why it works with the env deactivated (which is then running Python from the default OS installation). Try setting up the dev env without the --no-site-packages option if you want to be able to access modules outside of the env.

Improve this answer

1 Comment

it's common to have --no-site-packages with virtualenv to avoid contamination
1

I'm a little late to the conversation, but I was just having the same issue and I stumbled across gae_installer, which you can install in the usual way with pip install gae_installer. This will put the google app engine (gae) sdk directly into your python path. Hope others find this useful.

Improve this answer

1 Comment

did you do this inside or outside the virtual env?
0

Same answer as bozzo. Here's instructions:

This is described in Issue 4339 for GAE. Here's how to fix it:

  1. Download patch here: patch
  2. Move the patch to google_appengine/google/appengine/tools/
  3. Change your working directory to the same path as above
  4. Type: patch -p0 < dev_appserver.patch
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.