3

I want to mock out calls to urllib.request.urlopen in a module. It works when it's a single file, but when I put it in a package and import the module in the package's __init__.py, I cannot mock it out anymore.

Reproduction

Imagine I have two modules in a test package:

  • module.py

    from urllib.request import urlopen def do_it(): print(urlopen.__module__) urlopen('test')

  • module_test.py

    from unittest import mock from .module import do_it def test_not_working(): with mock.patch('urllib.request.urlopen', lambda url: print(url)): do_it() def test_plain(): with mock.patch('test.module.urlopen', lambda url: print(url)): do_it()

test_not_working prints urllib.request as the module for urlopen, since I patched the local urlopen function in the test module, and the test fails because test is not a valid URL.
test_plain prints .module, because I succesfully patched urlopen, and the test succeeds and prints test.

My issue is that I have moved .module into a package, because I wanted to group multiple files I created. It now looks like this:

  • module

    • __init__.py

      from .module import do_it
    • module.py (same as module.py before)

  • test_module.py

    from unittest import mock from .module import do_it def test_packaged_fails(): with mock.patch('test.module.urlopen', lambda url: print(url)): do_it() def test_packaged_works(): with mock.patch('test.module.module.urlopen', lambda url: print(url)): do_it()

The first two tests stay the same, but test_packaged prints urllib.request, and fails as the first test because the URL test is invalid.

I understand that I failed to mock urlopen, because it evidently doesn't use test.module.urlopen but test.module.module.urlopen.

Restrictions

Actual Project

I don't know how to fix this issue, because of the module is just one of many on an open source project (OpenMensa Parsers). The Aachener parser is a package containing multiple files, instead of a single file module like the other parsers.
The issue mentioned above occurs when I want to upgrade snapshots for our regression tests. It is supposed to cache all requests that a parser makes, so that the test can be reproduced later, even if the website changes.

List

I have the following restrictions:

  • I cannot hard code the path to the submodule (test.module.module.urlopen), because I want to use the snapshot generation for the other parsers as well.
  • I can only use libraries that are available as Debian Wheezy packages, because of the build system and deployment. See the currently installed dependencies.
  • I would like to stay with urllib.request.urlopen for consistency with the other parsers.
  • I actually do not call do_it() directly. I call a function from another module that dynamically imports do_it() from different parser modules and then calls it. I will leave the minimal example above as is for simplicity.

Question

How can I patch urlopen in the package, if I do not know the subpackage it is called in?

Improve this question

2 Answers 2

Reset to default
1

A workaround is to modify the import statement in module/module.py:

from urllib import request def do_it(): print(urlopen.__module__) request.urlopen('test')

This makes it possible to mock it using:

with mock.patch('urllib.request.urlopen', lambda url: print(url)): do_it()

However this is a workaround that requires changing the module under test. I am still interested in a way find the submodule that imported urlopen directly and mock it out there.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

well this is called writing code for test-ability I guess.
0

If you know what function uses urlopen, and you know it accesses urlopen through a module-level from urllib.request import urlopen, then you can perform the patch through the function's global variable dict:

with mock.patch.dict(do_it.__globals__, urlopen=whatever): do_whatever_with(do_it)

Ultimately, mocking in Python will still require a lot of implementation detail knowledge about the tested code's imports.

Improve this answer

3 Comments

This doesn't appear to work for me, because the package module/ does not have an attribute __globals__. I would have to add it myself, and I'm not sure how to do that. While this would be definitely cleaner, the root problem is that I want to know how to programmatically find the function inside some package's module, and save myself from adapting that module or the package.
@J0hj0h: The package doesn't have a __globals__ attribute. The function has a __globals__ attribute.
Thanks for your clarification! In the actual code I do not have direct access to that function, since I call it indirectly. I have updated the restrictions to point to this. Nevertheless, your answer is interesting. I learned something! :)

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.