Functional Tests

So far all the testing we’ve done has been unit testing, where we carefully isolate some piece of code and make assertions about how it (and only it) works.

It’s easy to get yourself into a situation where you mock yourself into placation, and the code doesn’t actually work anymore when you glue the pieces together even though all the unit tests pass. To avoid this situation, there are functional (or “system”) tests.

Adding a UI to Our Portfolio

Our portfolio code is, right now, essentially a library. It is a bit of code that is meant to be imported and used by other people. Let’s give it a UI and turn it in to something that normal people might see. In other words, let’s write an application that uses our Portfolio object.

Install Pyramid, pyramid_chameleon, waitress, and selenium:

$ pip install pyramid pyramid_chameleon waitress selenium

Add a views.py to our project (next to portfolio.py):

from pyramid.view import view_config

from .portfolio import Portfolio

@view_config(renderer='index.pt')
def index(request):
    p = Portfolio()
    p.buy('IBM', 100, 176.48)
    p.buy('HPQ', 100, 36.15)
    return {'portfolio':p}

Add an index.pt to our project (next to views.py):

<html>
<head></head>
<body>
<ul tal:define="current portfolio.current_prices()">
<li tal:repeat="stock portfolio.stocks">
  ${stock[0]} ${stock[1]} ${stock[2]} ${current[stock[0]]}
</li>
</ul>
</body>
</html>

Change our __init__.py so that it looks like this:

from waitress import serve
from pyramid.config import Configurator

def main():
    """Entry point for the application script"""
    config = Configurator()
    config.include('pyramid_chameleon')
    config.scan()
    app = config.make_wsgi_app()
    serve(app)

If we rerun setup.py develop, and run sample, we’ll see something like:

$ sample
serving on http://0.0.0.0:8080

We have now turned our library into an application. The application shows each stock’s name, the number of shares we own, its buying price, and its current value, one per line. We can make assertions about this output using selenium, a functional testing system.

Adding Functional Tests

Add a directory named functests to the sample project (next to tests). Add a file named __init__.py within it. Give it these contents:

# -*- coding: utf-8 -*-

import unittest

browser = None

# we create a global browser object for speed reasons.
# not ideal.. it breaks test isolation.

def setUpModule():
    from selenium.webdriver import Firefox
    global browser
    browser = Firefox()
    return browser

def tearDownModule():
    browser.quit()

class PageTests(unittest.TestCase):
    def test_render_default(self):
        browser.get('http://localhost:8080/')
        self.assertTrue('IBM 100 176.48' in browser.page_source)

Run nosetests sample.functests:

$ nosetests sample.functests
.
----------------------------------------------------------------------
Ran 1 test in 4.400s

OK

When you run these tests you will see the Firefox browser launched on your system briefly. Selenium is a system that executes browsers to ensure that web applications are operating properly.

We are using the nosetests script that is installed with nose for convenience. We could have changed things around such that setup.py nosetests executed the functional tests too, but we explicitly don’t want them to run when we run unit tests because when lots of test cases are added, they’re too slow, and people don’t run slow test suites.

Improving the Application

The template we wrote demonstrates that our Portfolio API is a little lame. Let’s improve it and evolve the template along with it, as well as adding a few more tests.

A Real World Test Suite

Let’s take a look at the Deformdemo application to get a sense of a real-world test suite for an application. https://github.com/Pylons/deformdemo

Follow the instructions in its README.rst to be able to run its test suite. Do this is a new shell, unrelated to the portfolio demo.

(Where it says virtualenv2.7 instead use virtualenv -ppy27).

Take a look at the tests defined in deformdemo/test.py (https://github.com/Pylons/deformdemo/blob/master/deformdemo/test.py).

Note that you can:

  • Click buttons.
  • Fill in form fields.
  • Make assertions about the contents of HTML elements.
  • Find specific elements using XPath, CSS, or by unique HTML id.
  • Send keystrokes.
  • Test the result of executing JavaScript (such as drag and drop).

Selenium can be driven using Python, and I tend to use it that way. It can also be driven via a UI (Selenium-UI), or by bindings for most other dynamic languages.

What About Integration Tests?

Integration tests, to me, are the worst of both worlds. They have all the fragility of unit tests and all of the slowness of system tests. I avoid them where possible. Sometimes they’re required such as when you test code that integrates with something that, for practical reasons, cannot be mocked, such as a database. But otherwise, they are not very useful, IMO.