I need to test a function that needs to query a page on an external server using urllib.urlopen (it also uses urllib.urlencode). The server could be down, the page could change; I can't rely on it for a test.
What is the best way to control what urllib.urlopen returns?
Another simple approach is to have your test override urllib's urlopen() function. For example, if your module has
import urllib def some_function_that_uses_urllib(): ... urllib.urlopen() ... You could define your test like this:
import mymodule def dummy_urlopen(url): ... mymodule.urllib.urlopen = dummy_urlopen Then, when your tests invoke functions in mymodule, dummy_urlopen() will be called instead of the real urlopen(). Dynamic languages like Python make it super easy to stub out methods and classes for testing.
See my blog posts at http://softwarecorner.wordpress.com/ for more information about stubbing out dependencies for tests.
I am using Mock's patch decorator:
from mock import patch [...] @patch('urllib.urlopen') def test_foo(self, urlopen_mock): urlopen_mock.return_value = MyUrlOpenMock() 
Did you give Mox a look? It should do everything you need. Here is a simple interactive session illustrating the solution you need:
>>> import urllib >>> # check that it works >>> urllib.urlopen('http://www.google.com/') <addinfourl at 3082723820L ...> >>> # check what happens when it doesn't >>> urllib.urlopen('http://hopefully.doesnotexist.com/') #-- snip -- IOError: [Errno socket error] (-2, 'Name or service not known') >>> # OK, let's mock it up >>> import mox >>> m = mox.Mox() >>> m.StubOutWithMock(urllib, 'urlopen') >>> # We can be verbose if we want to :) >>> urllib.urlopen(mox.IgnoreArg()).AndRaise( ... IOError('socket error', (-2, 'Name or service not known'))) >>> # Let's check if it works >>> m.ReplayAll() >>> urllib.urlopen('http://www.google.com/') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.5/site-packages/mox.py", line 568, in __call__ raise expected_method._exception IOError: [Errno socket error] (-2, 'Name or service not known') >>> # yay! now unset everything >>> m.UnsetStubs() >>> m.VerifyAll() >>> # and check that it still works >>> urllib.urlopen('http://www.google.com/') <addinfourl at 3076773548L ...> 
HTTPretty works in the exact same way that FakeWeb does. HTTPretty works in the socket layer, so it should work intercepting any python http client libraries. It's battle tested against urllib2, httplib2 and requests
import urllib2 from httpretty import HTTPretty, httprettified @httprettified def test_one(): HTTPretty.register_uri(HTTPretty.GET, "http://yipit.com/", body="Find the best daily deals") fd = urllib2.urlopen('http://yipit.com') got = fd.read() fd.close() assert got == "Find the best daily deals" 
In case you don't want to even load the module:
import sys,types class MockCallable(): """ Mocks a function, can be enquired on how many calls it received """ def __init__(self, result): self.result = result self._calls = [] def __call__(self, *arguments): """Mock callable""" self._calls.append(arguments) return self.result def called(self): """docstring for called""" return self._calls class StubModule(types.ModuleType, object): """ Uses a stub instead of loading libraries """ def __init__(self, moduleName): self.__name__ = moduleName sys.modules[moduleName] = self def __repr__(self): name = self.__name__ mocks = ', '.join(set(dir(self)) - set(['__name__'])) return "<StubModule: %(name)s; mocks: %(mocks)s>" % locals() class StubObject(object): pass And then:
>>> urllib = StubModule("urllib") >>> import urllib # won't actually load urllib >>> urls.urlopen = MockCallable(StubObject()) >>> example = urllib.urlopen('http://example.com') >>> example.read = MockCallable('foo') >>> print(example.read()) 'foo' 
Probably the best way to handle this is to split up the code, so that logic that processes the page contents is split from the code that fetches the page.
Then pass an instance of the fetcher code into the processing logic, then you can easily replace it with a mock fetcher for the unit test.
e.g.
class Processor(oject): def __init__(self, fetcher): self.m_fetcher = fetcher def doProcessing(self): ## use self.m_fetcher to get page contents class RealFetcher(object): def fetchPage(self, url): ## get real contents class FakeFetcher(object): def fetchPage(self, url): ## Return whatever fake contents are required for this test The simplest way is to change your function so that it doesn't necessarily use urllib.urlopen. Let's say this is your original function:
def my_grabber(arg1, arg2, arg3): # .. do some stuff .. url = make_url_somehow() data = urllib.urlopen(url) # .. do something with data .. return answer Add an argument which is the function to use to open the URL. Then you can provide a mock function to do whatever you need:
def my_grabber(arg1, arg2, arg3, urlopen=urllib.urlopen): # .. do some stuff .. url = make_url_somehow() data = urlopen(url) # .. do something with data .. return answer def test_my_grabber(): my_grabber(arg1, arg2, arg3, urlopen=my_mock_open) 
Adding onto Clint Miller's answer, to do this I had to create a fake class that implements a read method like this:
class FakeURL: def read(foo): return '{"some":"json_text"}' Then to stub out urllib2.open:
# Stub out urllib2.open. def dummy_urlopen(foo, bar, baz): return FakeURL() urllib2.urlopen = dummy_urlopen 
requests: How can I mock requests and the response?