These days, every codebase I work on runs tests under some form of third-party continuous integration. This is great, because people are lazy and don't always run all of their tests before pushing. But it's not so great when trying to debug why a test failed on the CI server but not locally. Even moreso when the test is a Selenium test; even a live SSH build on the CI server is of limited use, especially for an intermittent failure.

So, I had the idea to automatically take a screenshot of the current browser state whenever a selenium test failed or encountered an exception. That's easier said than done. Unless you are building your own test runner from scratch, hooking into unittest can be tricky. After much research and pain, this is what I came up with:

class SeleniumTestCase(StaticLiveServerTestCase):  # StaticLiveServerTestCase from django.contrib.staticfiles
    live_server_url = 'http://localhost:8081'

    @classmethod
    def setUpClass(cls):
        '''
        One example of creating the test browser.  You could do it per-function as well, in setUp() instead.
        We'll be using cls.browser or self.browser later to get at the screenshot-saving mechanism.
        I'm omitting the corresponding tearDownClass or tearDown, which needs to close the browser.
        '''
        cls.browser = Browser()

    def run(self, result=None):
        '''
        Run is called on every test function, and an aggregate TestResult object
        is maintained across every test function.
        '''
        self.currentResult = result # remember result for use in tearDown
        super(SeleniumTestCase, self).run(result)

    def tearDown(self):
        '''
        tearDown is called after every test function.
        unittest.TestCase.run() handles errors and failures by appending each
        to the aggregate TestResult lists (TestResult.errors and TestResult.failures)
        so after a test just failed or errored, it'll be the last one in the list.
        '''
        result = self.currentResult  # currentResult contains running results for every test in a TestCase
        if result.errors:
            last_error = result.errors[-1]
            last_error_case, _ = last_error  # extract test method from tuple
            if last_error_case.id() == self.id():
                self._save_screenshot(last_error)
        elif result.failures:
            last_fail = result.failures[-1]
            last_fail_case, _ = last_fail  # extract test method from tuple
            if last_fail_case.id() == self.id():
                self._save_screenshot(last_fail)

        return super(SeleniumTestCase, self).tearDown()

    def _save_screenshot(self, error=None):
        '''
        At this point, we can use the browser object to save a screenshot.
        We can also do this directly from test functions.
        Screenshots will be saved with the python path to the test method
        as a filename.
        '''
        path = settings.SELENIUM_SCREENSHOTS_PATH
        import os
        if not os.path.exists(path):
            os.makedirs(path)
        filename = "%s.jpg" % self.id()
        full_path = os.path.join(path, filename)
        self.browser.save_screenshot(full_path)

That tearDown() method is really gross, but the best approach I could find. It's also only compatible with Python 2.6 and 2.7, as unittest changes after 3.x., but see the sources below for some 3.x-compatible approaches.

It's also probably not compatible with Nose.

If I could wave a magic wand and update the unittest API, it would be far easier to do this if there were a post-failure hook, or if TestCase.run() returned the last test function's result.

Sources:

http://stackoverflow.com/questions/12290336/how-to-execute-code-only-on-test-failures-with-python-unittest2

http://stackoverflow.com/questions/4414234/getting-pythons-unittest-results-in-a-teardown-method


Using Tastypie Inside Django

Wed 21 November 2012 by George Dorn

Make use of a Tastypie's API from within Django

Tastypie is an excellent way to generate a REST API with minimal coding. But often it exists as a separate means of accessing your data, with its own implementation of your business logic, while your views also implement business logic ...

read more

Legacy BooleanField in Django

Mon 29 November 2010 by George Dorn

A legacy BooleanField supporting all kinds of antiquated ways of storing boolean values.

Django's inspectdb is pretty good at providing models that will at least read and write from your legacy database. But to get real power out of the ORM, you may need to provide some custom mapping ...

read more

Using Django Auth with a legacy app

Tue 09 November 2010 by George Dorn

One strategy for integrating a legacy python application with Django.

At work, we're planning to switch to Django. Rather than doing a complete feature freeze for six months while we rewrite the site in Django, the decision has been made to run two codebases and migrate features to Django ...

read more

Django vs SQAlchemy (vs PyDO) Speed Tests

Fri 29 October 2010 by George Dorn

Speed tests for SQLAlchemy vs Django vs PyDO

At work, we're preparing to move away from a 6-year-old homebrew web framework to Django. In the process, I figured I'd do some speed tests of the old ORM (PyDO, version one, last updated in 2004 and with an author ...

read more

Hygienic unit testing with Solr.

Mon 13 September 2010 by George Dorn

Python unittest hygiene and Solr.

A sane methodology for testing an application that uses a database looks like this:

  1. Prop up an empty database, preferably using the same engine as production.
  2. Import some fixture data to play with (optional).
  3. Run tests that manipulate data in this database.
  4. Roll back the ...
read more

uWSGI + lighttpd: Serving static files.

Thu 08 July 2010 by George Dorn

Serving both static and dynamic files with lighttpd + uWSGI.

A common situation: you have one domain and one application, but the app contains both static files (images, js, etc) and dynamic code (e.g. python modules). Lighttpd's configuration is not trivial, and there's an added complication from how ...

read more

Werkzeug DebuggedApplication with uWSGI under lighttpd.

Thu 08 July 2010 by George Dorn

uWSGI is a fairly awesome app server; it's lightweight and fast, yet highly stable.

Unfortunately, mod_uwsgi for lighttpd is somewhat half-baked. Aside from yet another app server using the equivalent of urllib.unquote instead of urllib.unquote_plus, it also behaves differently than most app servers in dealing with query ...

read more