Python HTTP mocking

I need to do a better job mocking out HTTP requests to allow me to test all this web API code I’ve written. Right now I rolled my own bad mocking around a wrapper for all the REST calls, but I think it makes more sense just to mock at the HTTP level.

A quick Google search reveals three modern libraries: requests-mock, responses, and httpretty. After literally tens of minutes I’m left with the opinion any would work. responses-mock looks like the most mature and complete product. I like responses for its simplicity. The main differences between the three are the mechanism they use to intercept HTTP calls. I believe httpretty is the only one that works with an HTTP library other than Requests. OTOH why would you use any other HTTP library in Python?

Update: since doing this exercise I started using requests-mock. I’m quite happy with it.

Some detailed notes below, then my test code.

requests-mock: mocks out Requests by installing a Transport Adapter. (I’m endeared, because I came up with a similar technique on my own for Java back in 2000 or so.) Has detailed support for matching full URLs or parts of URLs or specific query strings or URL regexps. Supports all HTTP methods. Responses can be text, JSON, byte strings, or callbacks. Has a way to inspect which URLs were called. Has great docs.

Responses: mocks out Requests using the unittest.mock library, patching requests’ send function. Supports matching on regex, but no special support for query strings in URLs. Responses can be text or JSON or callbacks. Supports all HTTP methods. Optionally throws an error if not all URLs were accessed. No fancy docs, but the code is easy to read. Motivation in this blog post.

httpretty: works by monkey patching sockets. that’s terrifying, but kinda neat because it can mock out HTTP from sources other than Requests. Oddly they don’t have official Python 3 support (it seems to work though). I didn’t look at all its features in detail.

(BTW, unittest.mock was new to me, a very nice standard library for monkey patching any Python library for unit test mocking. That could come in handy in lots of places.)

Here’s my test / sample code.

#!/usr/bin/env python3

import requests, unittest
import requests_mock, responses, httpretty

def fetch():
    "The method we'll be mocking"
    r = requests.get('https://www.somebits.com/robots.txt')
    return r.text

class TestStuff(unittest.TestCase):
    def test_real(self):
        self.assertEqual(107, len(fetch()))

    def test_requests_mock(self):
        with requests_mock.Mocker() as m:
            m.get('https://www.somebits.com/robots.txt', text='nelson-requests-mock')
            self.assertEqual('nelson-requests-mock', fetch())

    def test_responses(self):
        with responses.RequestsMock() as r:
            r.add(responses.GET, 'https://www.somebits.com/robots.txt', 'nelson-responses-mock')
            self.assertEqual('nelson-responses-mock', fetch())

    def test_httpretty(self):
         httpretty.enable()
         httpretty.register_uri(httpretty.GET, "https://www.somebits.com/robots.txt", 'nelson-response-httpretty')
         self.assertEqual('nelson-response-httpretty', fetch())
         httpretty.disable()

if __name__ == '__main__':
    unittest.main(exit=False)

    # Verify the environment is left clean
    assert 107 == len(fetch())
    print("Real requests are working again.")