Growing A Project With Tests

Let’s test some real code. Note that some of these code examples are courtesty Ned Batchelder.

Add a file to your “sample project” within sampleproject/sample/portfolio.py. Use the following contents:

# portfolio3.py

import urllib
import csv

class Portfolio(object):
    """A simple stock portfolio

    >>> p = Portfolio()
    >>> p.cost()
    0.0

    >>> p.buy("IBM", 100, 176.48)
    >>> p.cost()
    17648.0

    >>> p.buy("HPQ", 100, 36.15)
    >>> p.cost()
    21263.0

    """
    def __init__(self):
        # stocks is a list of lists:
        #   [[name, shares, price], ...]
        self.stocks = []

    def buy(self, name, shares, price):
        """Buy `name`: `shares` shares at `price`."""
        self.stocks.append([name, shares, price])

    def cost(self):
        """What was the total cost of this portfolio?"""
        amt = 0.0
        for name, shares, price in self.stocks:
            amt += shares * price
        return amt

    def sell(self, name, shares):
        """Sell some number of shares of `name`."""
        for holding in self.stocks:
            if holding[0] == name:
                if holding[1] < shares:
                    raise ValueError("Not enough shares")
                holding[1] -= shares
                break
        else:
            raise ValueError("You don't own that stock")

    #(((value)))
    def current_prices(self):
        """Return a dict mapping names to current prices."""
        url = "http://finance.yahoo.com/d/quotes.csv?f=sl1&s="
        url += ",".join(sorted(s[0] for s in self.stocks))
        data = urllib.urlopen(url)
        return { sym: float(last) for sym, last in csv.reader(data) }

    def value(self):
        """Return the current value of the portfolio."""
        prices = self.current_prices()
        total = 0.0
        for name, shares, _ in self.stocks:
            total += shares * prices[name]
        return total
    #(((end)))

Code Description

Our code under test is a simple stock portfolio class. It simply stores the lots of stocks purchased: each is a stock name, a number of shares, and the price it was bought at. We have a method to buy a stock, and a method that tells us the total cost of the portfolio.

How do we test it?

Let’s try it interactively. Invoke python and do the following:

>>> from sample.portfolio import Portfolio
>>> p = Portfolio()
>>> p.cost()
0.0
>>> p.buy("IBM", 100, 176.48)
>>> p.cost()
17648.0
>>> p.buy("HPQ", 100, 36.15)
>>> p.cost()
21263.0

We can see that an empty portfolio has a cost of zero. We buy oen stock and the cost is the price times the shares. Then we buy another, and the cost has risen as it should.

Good:

• We’re testing the code.

Bad:

• Not repeatable
• Labor intensive
• Is the result right?

We could do mre adhoc testing, and we could even script our testing into a simple script that makes assertions about the output of the methods we’re calling.

from sample.portfolio import Portfolio

p = Portfolio()
print "Empty portfolio cost: %s, should be 0.0" % p.cost()
assert p.cost() == 0.0
p.buy("IBM", 100, 176.48)
print "With 100 IBM @ 176.48: %s, should be 17648.0" % p.cost()
assert p.cost() == 17648.0
p.buy("HPQ", 100, 36.15)
print "With 100 HPQ @ 36.15: %s, should be 21263.0" % p.cost()
assert p.cost() == 21263.0

That would be OK. But it would mean that if the tests started to fail, they would stop on the first error. For example, if we broke the code for the cost method, we might run our test script and see the following:

Empty portfolio cost: 0.0, should be 0.0
With 100 IBM @ 176.48: 17648.0, should be 17600.0
Traceback (most recent call last):
  File "porttest3_broken.py", line 9, in <module>
    assert p.cost() == 17600.0
AssertionError

That’s not great, because as we add more tests, we won’t want things to stop at the first error. Instead, we’ll want all of our tests to run, and we’ll want the system to display all of the errors it finds without stopping at the first one.

Let’s use what we learned earlier about running unit tests to fix the bad parts better.

Adding Unit Tests

Add a file to the sampleproject/sample/tests directory named test_portfolio.py with the following contents:

import unittest
from sample.portfolio import Portfolio

class PortfolioTest(unittest.TestCase):
    def test_buy_one_stock(self):
        p = Portfolio()
        p.buy("IBM", 100, 176.48)
        assert p.cost() == 17648.0

Now let’s use what we learned about the mechanics of running tests to run this test case against our codebase.

$ python setup.py test -q

Here’s the output we should see:

running test
running egg_info
writing requirements to sample.egg-info/requires.txt
writing sample.egg-info/PKG-INFO
writing top-level names to sample.egg-info/top_level.txt
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'
running build_ext
F.
======================================================================
FAIL: test_failure (sample.tests.test_simple.TestSimple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_simple.py", line 9, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (failures=1)

What?! A failing test?! Oh, it’s just the test_simple.py file we forgot to remove. We don’t need that anymore. Remove sampleproject/sample/tests/test_simple.py (and its accompanying .pyc file that is created) and rerun the test suite.:

$ rm sample/tests/sample.py{,c}
$ python setup.py test -q

running test
running egg_info
writing requirements to sample.egg-info/requires.txt
writing sample.egg-info/PKG-INFO
writing top-level names to sample.egg-info/top_level.txt
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'
running build_ext
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Much better. We now have a level of confidence that buying a single stock works as we expect.

Let’s add a few more tests for edge cases. Change the test_portfolio.py file to look like this:

import unittest
from sample.portfolio import Portfolio

class PortfolioTest(unittest.TestCase):
    def test_empty(self):
        p = Portfolio()
        self.assertEqual(p.cost(), 0.0)

    def test_buy_one_stock(self):
        p = Portfolio()
        p.buy("IBM", 100, 176.48)
        self.assertEqual(p.cost(), 17648.0)

    def test_buy_two_stocks(self):
        p = Portfolio()
        p.buy("IBM", 100, 176.48)
        p.buy("HPQ", 100, 36.15)
        self.assertEqual(p.cost(), 21263.0)

Here we add a simpler test, test_empty, and a more complicated test, test_buy_two_stocks. Each test is another test method in our PortfolioTest class. Each one creates the portfolio object it needs, performs the manipulations it wants, and makes assertions about the outcome.

Let’s run the test suite again.

$ python setup.py test -q

running test
running egg_info
writing requirements to sample.egg-info/requires.txt
writing sample.egg-info/PKG-INFO
writing top-level names to sample.egg-info/top_level.txt
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'
running build_ext
...
----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Under The Covers

unittest runs our test suite as if we had typed this:

testcase = PortfolioTest()

try:
    testcase.test_buy_one_stock()
except AssertionError:
    # [record failure]
else:
    # [record success]

testcase = PortfolioTest()

try:
    testcase.test_buy_two_stocks()
except AssertionError:
    # [record failure]
else:
    # [record success]

testcase = PortfolioTest()

try:
    testcase.test_empty()
except AssertionError:
    # [record failure]
else:
    # [record success]

(Again, tests run in alphabetical order.)

A key thing to note here is that a new instance of PortfolioTest is created for each test method. This helps to guarantee an important property of good tests: isolation.

Test isolation means that each of your tests is unaffected by every other test. This is good because it makes your tests more repeatable, and they are clearer about what they are testing. It also means that if a test fails, you don’t have to think about all the conditions and data created by earlier tests: running just that one test will reproduce the failure.

Earlier we demonstrated a problem where one test failing prevented the other tests from running. Here unittest is running each test independently, so if one fails, the rest will run, and will run just as if the earlier test had succeeded.

What Failure Looks Like

Let’s break a test and see what happens. Change the test_portfolio.py file to look like this:

import unittest
from sample.portfolio import Portfolio

class PortfolioTest(unittest.TestCase):
    def test_empty(self):
        p = Portfolio()
        assert p.cost() == 0.0

    def test_buy_one_stock(self):
        p = Portfolio()
        p.buy("IBM", 100, 176)      # this is wrong, to make the test fail!
        assert p.cost() == 17648.0

    def test_buy_two_stocks(self):
        p = Portfolio()
        p.buy("IBM", 100, 176.48)
        p.buy("HPQ", 100, 36.15)
        assert p.cost() == 21263.0

$ python setup.py test -q

F..
======================================================================
FAIL: test_buy_one_stock (sample.tests.test_portfolio.PortfolioTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_portfolio.py", line 12, in test_buy_one_stock
    assert p.cost() == 17648.0
AssertionError

----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (failures=1)

(I’ve omitted the boilerplate stuff emitted by setuptools at the top, I’ll keep omitting it from here on in).

Note that we see two dots and a failure represented by the “F”. The traceback shows us where the failure occured. Remember: when your tests pass, you don’t have to do anything, you can go on with other work, so passing tests, while a good thing, should not cause a lot of noise. It’s the failing tests we need to think about.

It’s great that the traceback shows what assertion failed, but notice that it doesn’t tell us what bad value was returned. We can see that we expected it to be 17648.0, but we don’t know what the actual value was.

Let’s use unittest helper methods to fix that.

Using Unittest Helper Methods

Change the test_portfolio.py file to look like this:

import unittest
from sample.portfolio import Portfolio

class PortfolioTest(unittest.TestCase):
    def test_empty(self):
        p = Portfolio()
        self.assertEqual(p.cost(), 0.0)

    def test_buy_one_stock(self):
        p = Portfolio()
        p.buy("IBM", 100, 176)      # this is wrong, to make the test fail!
        self.assertEqual(p.cost(), 17648.0)

    def test_buy_two_stocks(self):
        p = Portfolio()
        p.buy("IBM", 100, 176.48)
        p.buy("HPQ", 100, 36.15)
        self.assertEqual(p.cost(), 21263.0)

In this test, we’ve used self.assertEqual instead of the built-in assert statement. The benefit of the method is that it can print both the actual and expected values if the test fails.

$ python setup.py test -q

F..
======================================================================
FAIL: test_buy_one_stock (sample.tests.test_portfolio.PortfolioTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_portfolio.py", line 12, in test_buy_one_stock
    self.assertEqual(p.cost(), 17648.0)
AssertionError: 17600.0 != 17648.0

----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (failures=1)

Here the assertion error message shows both values. The best tests will give you information you can use to diagnose and debug the problem. Here, when comparing two values for equality, we are told not only that they are not equal, but also what each value was.

Assert helpers:

assertEqual(first, second)
assertNotEqual(first, second)
assertTrue(expr)
assertFalse(expr)
assertIn(first, second)
assertNotIn(first, second)
assertIs(first, second)
assertIsNot(first, second)
assertAlmostEqual(first, second)
assertNotAlmostEqual(first, second)
assertGreater(first, second)
assertLess(first, second)
assertRegexpMatches(first, second)
assertRaises(exc_class, func, ...)
assertSequenceEqual(seq1, seq2)
assertItemsEqual(seq1, seq2)

# ... etc ...

The unittest.TestCase class provides many assertion methods for testing many conditions. The unittest documentation at http://docs.python.org/2/library/unittest.html shows all of them.

An additional benefit of using assertion methods rather than the assert Pytyhon builtin: assert statements can be compiled out in certain circumstances (the -O flag to Python) and may not even execute as a result (a questionable, surprising feature which was added to Python early in its lifetime).

To avoid needing to think about any of this, always use assertion methods rather than the assert builtin.

Seeing Errors

Let’s break our code in one additional way, by injecting an exception. We’ll change the test_buy_one_stock method such that we try to call a nonexistent method named buyXX instead of buy.

import unittest
from sample.portfolio import Portfolio

class PortfolioTest(unittest.TestCase):
    def test_empty(self):
        p = Portfolio()
        self.assertEqual(p.cost(), 0.0)

    def test_buy_one_stock(self):
        p = Portfolio()
        p.buyXX("IBM", 100, 176.48)      # this is wrong, to make the test fail!
        self.assertEqual(p.cost(), 17648.0)

    def test_buy_two_stocks(self):
        p = Portfolio()
        p.buy("IBM", 100, 176.48)
        p.buy("HPQ", 100, 36.15)
        self.assertEqual(p.cost(), 21263.0)

If we run the tests again we’ll see this output:

E..
======================================================================
ERROR: test_buy_one_stock (sample.tests.test_portfolio.PortfolioTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_portfolio.py", line 11, in test_buy_one_stock
    p.buyXX("IBM", 100, 176.48)      # this is wrong, to make the test fail!
AttributeError: 'Portfolio' object has no attribute 'buyXX'

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (errors=1)

There are actually three possible outcomes for a test: passing, failure, and error. Error means that the test raised an exception other than AssertionError. Well-written tests should either succeed, or should fail in an assertion. If another exception happens, it could either mean that the test is broken, or that the code is broken, but either way, it isn’t what you intended. This condition is marked with an “E”, and the error stacktrace is displayed.

assertRaises

The assertRaises helper method is a bit complex. The easiest way to use it is via a with statement.

Add the following to the PortfolioTest class:

def test_bad_input(self):
    p = Portfolio()
    with self.assertRaises(TypeError):
        p.buy("IBM")

This test will pass. It asserts that the buy method will raise a TypeError when called with too few arguments. This is normal Python behvaior, and doesn’t really need to be tested; this is just an example for purposes of explaining the helper method.

Adding Coverage Reporting

We can use the coverage tool to add objective metrics that help us evaluate the usefulness of our test suite.

To do so, we’ll switch our test runner. So far we’ve been using the built-in setuptools test runner. To easily get coverage statistics, instead, we’re going to use the nose test runner.

Let’s change our setup.py to depend, in its test extra, on nose.

Change it from this:

extras_require={
    'dev': ['check-manifest'],
    'test': ['coverage'],
},

To this:

extras_require={
    'dev': ['check-manifest'],
    'test': ['coverage', 'nose'],
},

Then, we’ll change our setup.cfg file to create an alias, so that when we type setup.py dev, it will install both nose and coverage for us. Add the following stanza to setup.cfg:

[aliases]
dev = develop easy_install sample[test]

Now that we’ve made those two changes, type python setup.py dev in your shell, and see that both nose and coverage get installed:

# ...
creating /home/chrism/projects/carscom/unittest_training/lib/python2.7/site-packages/coverage-4.0b2-py2.7-linux-x86_64.egg
Extracting coverage-4.0b2-py2.7-linux-x86_64.egg to /home/chrism/projects/carscom/unittest_training/lib/python2.7/site-packages
Adding coverage 4.0b2 to easy-install.pth file
Installing coverage2 script to /home/chrism/projects/carscom/unittest_training/bin
Installing coverage-2.7 script to /home/chrism/projects/carscom/unittest_training/bin
Installing coverage script to /home/chrism/projects/carscom/unittest_training/bin

Installed /home/chrism/projects/carscom/unittest_training/lib/python2.7/site-packages/coverage-4.0b2-py2.7-linux-x86_64.egg
# ...

Running Tests via nosetests

Now that we have nose installed, we can use its features.

Let’s first run python setup.py nosetests. See that it produces output similar to that of running python setup.py test.

Once we verify that, we’ll run it with the --with-coverage flag:

python setup.py nosetests --with-coverage --cover-package=sample

.F
======================================================================
FAIL: test_failure (sample.tests.test_simple.TestSimple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_simple.py", line 9, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

Name                  Stmts   Miss  Cover   Missing
---------------------------------------------------
sample.py                 2      1    50%   4
sample/portfolio.py      31     16    48%   40-47, 52-55, 59-63
---------------------------------------------------
TOTAL                    33     17    48%
----------------------------------------------------------------------
Ran 2 tests in 0.011s

FAILED (failures=1)

(Your output will differ slightly).

The things we want to pay attention to are the Cover column and the Missing column.

Cover is a percentage of statements in the module covered. A “statement” for purposes of this discussion is a single line of code.

Missing is a list of lines that were not executed when the tests ran. Above, the sample.py file actually means the __init__.py file in the sample project. Its line 4 was not executed. The sample/portfolio.py file had unexecuted lines 40-47, 52-55, 59-63.

Our goal will be to get rid of all the output in the Missing column and make the Cover column display 100% for each module.

Testing the Portfolio.sell() method

Let’s test the Portfolio.sell() method in real time, using setup.py nosetests to check our work. Coverage helps us see what we missed.

• How many test methods should we use to try to test it (hint: it has a couple of conditions in it)?
• What does for: else: mean?
• pragma: no cover.
• --cover-html
• Extra credit: branch coverage (--cover-branches).

Uh Oh. Code That Integrates With Another System

The code in our Portfolio.current_prices() method calls urllib.urlopen, asking finance.yahoo.com what the value is of each stock that we own. The Portfolio.value() method calls the current_prices() method so it depends on the same external system.

• What happens when finance.yahoo.com is unavailable? Maybe it’s not even down; maybe you’re on a plane.
• Should we be able to test this code without the yahoo site being available? Yes!
• Do we care that the yahoo site actually works in this context? No!
• Unit tests test units of code. In this case we’re concerned about the code in the value and current_prices methods. We are only concerned about testing that code, no other code. In our little unit testing world, we will assume that urllib.urlopen works, we assume that finance.yahoo.com works as requested and that it returns something that can be parsed by the CSV module.
• Another reason to think of things this way: the yahoo site might be slow, which will make your tests slow, which makes it hard to write lots of tests that will be run frequently.
• External dependencies are also unpredictable, which makes testing code which depends on one hard. For instance, in our case, the finance site might return values that differ every few minutes.

Solution? Test doubles.

Types of doubles:

• Stubs.
• Mocks.

Stubs

We’ll add a stub in a new test case to help us avoid depending on the “real” current_prices implementation. Add this class to the test_portfolio.py module:

class PortfolioValueTest(unittest.TestCase):
    def stub_current_prices(self):
        return {'IBM':140.0, 'HPQ':32.0}

    def _makePortfolio(self):
        p = Portfolio()
        p.buy('IBM', 100, 120.0)
        p.buy('HPQ', 100, 30.0)
        p.current_prices = self.stub_current_prices
        return p

    def test_value(self):
        p = self._makePortfolio()
        self.assertEqual(p.value(), 17200)

The magic of Python allows us to replace the method named current_prices on the portfolio we return to the test_value` method with one which always return a static dictionary, therefore the external system is avoided.

But we can see if we run our coverage tests again that we haven’t tested all the code. How do we fix that?

Stubbing urllib

Change portfolio.py to add the following line to the class near the top:

urllib = urllib

And change the code in current_price to use self.urllib instead of urllib.

import urllib
# ....

class Portfolio(object):
    # ...
    urllib = urllib
    # ...
    def current_prices(self):
        """Return a dict mapping names to current prices."""
        url = "http://finance.yahoo.com/d/quotes.csv?f=sl1&s="
        url += ",".join(sorted(s[0] for s in self.stocks))
        data = self.urllib.urlopen(url)
        return { sym: float(last) for sym, last in csv.reader(data) }

Then in our test code we can change it so we do this:

from StringIO import StringIO

class FakeUrllib(object):
    def urlopen(self, url):
        return StringIO('"IBM",140\n"HPQ",32\n')

class PortfolioValueTest(unittest.TestCase):
    def _makePortfolio(self):
        p = Portfolio()
        p.buy('IBM', 100, 120.0)
        p.buy('HPQ', 100, 30.0)
        p.urllib = FakeUrllib()
        return p

    def test_value(self):
        p = self._makePortfolio()
        self.assertEqual(p.value(), 17200)

Now most of our code is tested as we can see from coverage reports.

But we needed to change our code to be able to stubify it. Let’s try it with a concept named mocks instead.

Mocks

Get rid of the urllib = urllib line in the Portfolio class and change things back so that we do data = urllib.urlopen.. instead of data= self.urllib.urlopen. Now let’s use a mock in our test.

import mock

from sample.portfolio import Portfolio

class PortfolioValueTest(unittest.TestCase):
    def _makePortfolio(self):
        p = Portfolio()
        p.buy('IBM', 100, 120.0)
        p.buy('HPQ', 100, 30.0)
        return p

    def test_value(self):
        p = self._makePortfolio()
        with mock.patch('urllib.urlopen') as urlopen:
            fake_yahoo = StringIO('"IBM",140\n"HPQ",32\n')
            urlopen.return_value = fake_yahoo
            self.assertEqual(p.value(), 17200)
            urlopen.assert_called_with('http://finance.yahoo.com/d/quotes.csv?f=sl1&s=HPQ,IBM')

Now we didn’t need to mutate our code to make it testable. We were also able to make assertions about the value that was passed to urlopen pretty easily.

Mocks or Stubs?

• Yes.
• Mocks have some downsides that we’ll cover in a future section. In particular, they can make the reader of the testing code need to understand obscure bits of the Mock API, making the tests less understandbale.
• Stubs have the downside that you often need to change your code to use them, and writing them isn’t much fun.

Discussion

• What happens when code starts using requests instead of urllib?
• What kind of tradeoff could we choose to make in order to make our tests less fragile?

Coda: setUp and tearDown

unittest.TestCase subclasses can have setUp and TearDown methods. The code in each of these runs before/after each test that runs. I don’t use these much because they’re often used to manage globals, and I try not to have globals. Another thing they’re used for is to get rid of boilerplate.

• An example which gets rid of p = Portfolio() in each test by adding self.p = Portfolio() to setUp.

Under the covers of setUp and tearDown:

testcase = PortfolioTest()
try:
    testcase.setUp()
except:
    # [ record error ]
else:
    try:
        testcase.test_method()
    except AssertionError:
        # [ record failure ]
    except:
        # [ record error ]
    else:
        # [ record success ]
    finally:
        try:
            testcase.tearDown()
        except:
            # [ record error ]

• A helper method, _makePortfolio() instead.
• I like the helper method instead.