Growing A Project With Tests ============================ Let's test some real code. Note that some of these code examples are courtesty Ned Batchelder. Add a file to your "sample project" within ``sampleproject/sample/portfolio.py``. Use the following contents: .. literalinclude:: sampleproject/sample/portfolio.py :linenos: Code Description ---------------- Our code under test is a simple stock portfolio class. It simply stores the lots of stocks purchased: each is a stock name, a number of shares, and the price it was bought at. We have a method to buy a stock, and a method that tells us the total cost of the portfolio. How do we test it? Let's try it interactively. Invoke ``python`` and do the following: .. code-block:: python >>> from sample.portfolio import Portfolio >>> p = Portfolio() >>> p.cost() 0.0 >>> p.buy("IBM", 100, 176.48) >>> p.cost() 17648.0 >>> p.buy("HPQ", 100, 36.15) >>> p.cost() 21263.0 We can see that an empty portfolio has a cost of zero. We buy oen stock and the cost is the price times the shares. Then we buy another, and the cost has risen as it should. Good: - We're testing the code. Bad: - Not repeatable - Labor intensive - Is the result right? We could do mre adhoc testing, and we could even script our testing into a simple script that makes assertions about the output of the methods we're calling. .. literalinclude:: portadhoc.py :linenos: That would be OK. But it would mean that if the tests started to fail, they would stop on the first error. For example, if we broke the code for the ``cost`` method, we might run our test script and see the following: .. code-block:: python Empty portfolio cost: 0.0, should be 0.0 With 100 IBM @ 176.48: 17648.0, should be 17600.0 Traceback (most recent call last): File "porttest3_broken.py", line 9, in assert p.cost() == 17600.0 AssertionError That's not great, because as we add more tests, we won't want things to stop at the first error. Instead, we'll want *all* of our tests to run, and we'll want the system to display all of the errors it finds without stopping at the first one. Let's use what we learned earlier about running unit tests to fix the bad parts better. Adding Unit Tests ----------------- Add a file to the ``sampleproject/sample/tests`` directory named ``test_portfolio.py`` with the following contents: .. literalinclude:: sampleproject/sample/tests/test_portfolio.py :linenos: Now let's use what we learned about the mechanics of running tests to run this test case against our codebase. .. code-block:: bash $ python setup.py test -q Here's the output we should see: .. code-block:: text running test running egg_info writing requirements to sample.egg-info/requires.txt writing sample.egg-info/PKG-INFO writing top-level names to sample.egg-info/top_level.txt writing dependency_links to sample.egg-info/dependency_links.txt writing entry points to sample.egg-info/entry_points.txt reading manifest file 'sample.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sample.egg-info/SOURCES.txt' running build_ext F. ====================================================================== FAIL: test_failure (sample.tests.test_simple.TestSimple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_simple.py", line 9, in test_failure self.assertTrue(False) AssertionError: False is not true ---------------------------------------------------------------------- Ran 2 tests in 0.000s FAILED (failures=1) What?! A failing test?! Oh, it's just the test_simple.py file we forgot to remove. We don't need that anymore. Remove ``sampleproject/sample/tests/test_simple.py`` (and its accompanying .pyc file that is created) and rerun the test suite.: .. code-block:: bash $ rm sample/tests/sample.py{,c} $ python setup.py test -q .. code-block:: text running test running egg_info writing requirements to sample.egg-info/requires.txt writing sample.egg-info/PKG-INFO writing top-level names to sample.egg-info/top_level.txt writing dependency_links to sample.egg-info/dependency_links.txt writing entry points to sample.egg-info/entry_points.txt reading manifest file 'sample.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sample.egg-info/SOURCES.txt' running build_ext . ---------------------------------------------------------------------- Ran 1 test in 0.000s OK Much better. We now have a level of confidence that buying a single stock works as we expect. Let's add a few more tests for edge cases. Change the ``test_portfolio.py`` file to look like this: .. code-block:: python import unittest from sample.portfolio import Portfolio class PortfolioTest(unittest.TestCase): def test_empty(self): p = Portfolio() self.assertEqual(p.cost(), 0.0) def test_buy_one_stock(self): p = Portfolio() p.buy("IBM", 100, 176.48) self.assertEqual(p.cost(), 17648.0) def test_buy_two_stocks(self): p = Portfolio() p.buy("IBM", 100, 176.48) p.buy("HPQ", 100, 36.15) self.assertEqual(p.cost(), 21263.0) Here we add a simpler test, test_empty, and a more complicated test, test_buy_two_stocks. Each test is another test method in our PortfolioTest class. Each one creates the portfolio object it needs, performs the manipulations it wants, and makes assertions about the outcome. Let's run the test suite again. .. code-block:: bash $ python setup.py test -q .. code-block:: text running test running egg_info writing requirements to sample.egg-info/requires.txt writing sample.egg-info/PKG-INFO writing top-level names to sample.egg-info/top_level.txt writing dependency_links to sample.egg-info/dependency_links.txt writing entry points to sample.egg-info/entry_points.txt reading manifest file 'sample.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sample.egg-info/SOURCES.txt' running build_ext ... ---------------------------------------------------------------------- Ran 3 tests in 0.000s OK Under The Covers ---------------- ``unittest`` runs our test suite as if we had typed this: .. code-block:: python testcase = PortfolioTest() try: testcase.test_buy_one_stock() except AssertionError: # [record failure] else: # [record success] testcase = PortfolioTest() try: testcase.test_buy_two_stocks() except AssertionError: # [record failure] else: # [record success] testcase = PortfolioTest() try: testcase.test_empty() except AssertionError: # [record failure] else: # [record success] (Again, tests run in alphabetical order.) A key thing to note here is that a new instance of PortfolioTest is created for each test method. This helps to guarantee an important property of good tests: isolation. Test isolation means that each of your tests is unaffected by every other test. This is good because it makes your tests more repeatable, and they are clearer about what they are testing. It also means that if a test fails, you don't have to think about all the conditions and data created by earlier tests: running just that one test will reproduce the failure. Earlier we demonstrated a problem where one test failing prevented the other tests from running. Here unittest is running each test independently, so if one fails, the rest will run, and will run just as if the earlier test had succeeded. What Failure Looks Like ----------------------- Let's break a test and see what happens. Change the test_portfolio.py file to look like this: .. code-block:: python import unittest from sample.portfolio import Portfolio class PortfolioTest(unittest.TestCase): def test_empty(self): p = Portfolio() assert p.cost() == 0.0 def test_buy_one_stock(self): p = Portfolio() p.buy("IBM", 100, 176) # this is wrong, to make the test fail! assert p.cost() == 17648.0 def test_buy_two_stocks(self): p = Portfolio() p.buy("IBM", 100, 176.48) p.buy("HPQ", 100, 36.15) assert p.cost() == 21263.0 .. code-block:: bash $ python setup.py test -q .. code-block:: text F.. ====================================================================== FAIL: test_buy_one_stock (sample.tests.test_portfolio.PortfolioTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_portfolio.py", line 12, in test_buy_one_stock assert p.cost() == 17648.0 AssertionError ---------------------------------------------------------------------- Ran 3 tests in 0.000s FAILED (failures=1) (I've omitted the boilerplate stuff emitted by setuptools at the top, I'll keep omitting it from here on in). Note that we see two dots and a failure represented by the "F". The traceback shows us where the failure occured. Remember: when your tests pass, you don't have to do anything, you can go on with other work, so passing tests, while a good thing, should not cause a lot of noise. It's the failing tests we need to think about. It's great that the traceback shows what assertion failed, but notice that it doesn't tell us what bad value was returned. We can see that we expected it to be 17648.0, but we don't know what the actual value was. Let's use unittest helper methods to fix that. Using Unittest Helper Methods ----------------------------- Change the test_portfolio.py file to look like this: .. code-block:: python import unittest from sample.portfolio import Portfolio class PortfolioTest(unittest.TestCase): def test_empty(self): p = Portfolio() self.assertEqual(p.cost(), 0.0) def test_buy_one_stock(self): p = Portfolio() p.buy("IBM", 100, 176) # this is wrong, to make the test fail! self.assertEqual(p.cost(), 17648.0) def test_buy_two_stocks(self): p = Portfolio() p.buy("IBM", 100, 176.48) p.buy("HPQ", 100, 36.15) self.assertEqual(p.cost(), 21263.0) In this test, we've used self.assertEqual instead of the built-in assert statement. The benefit of the method is that it can print both the actual and expected values if the test fails. .. code-block:: bash $ python setup.py test -q .. code-block:: text F.. ====================================================================== FAIL: test_buy_one_stock (sample.tests.test_portfolio.PortfolioTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_portfolio.py", line 12, in test_buy_one_stock self.assertEqual(p.cost(), 17648.0) AssertionError: 17600.0 != 17648.0 ---------------------------------------------------------------------- Ran 3 tests in 0.000s FAILED (failures=1) Here the assertion error message shows both values. The best tests will give you information you can use to diagnose and debug the problem. Here, when comparing two values for equality, we are told not only that they are not equal, but also what each value was. Assert helpers: .. code-block:: text assertEqual(first, second) assertNotEqual(first, second) assertTrue(expr) assertFalse(expr) assertIn(first, second) assertNotIn(first, second) assertIs(first, second) assertIsNot(first, second) assertAlmostEqual(first, second) assertNotAlmostEqual(first, second) assertGreater(first, second) assertLess(first, second) assertRegexpMatches(first, second) assertRaises(exc_class, func, ...) assertSequenceEqual(seq1, seq2) assertItemsEqual(seq1, seq2) # ... etc ... The unittest.TestCase class provides many assertion methods for testing many conditions. The unittest documentation at http://docs.python.org/2/library/unittest.html shows all of them. An additional benefit of using assertion methods rather than the ``assert`` Pytyhon builtin: ``assert`` statements can be compiled out in certain circumstances (the ``-O`` flag to Python) and may not even execute as a result (a questionable, surprising feature which was added to Python early in its lifetime). To avoid needing to think about any of this, always use assertion methods rather than the ``assert`` builtin. Seeing Errors ------------- Let's break our code in one additional way, by injecting an exception. We'll change the ``test_buy_one_stock`` method such that we try to call a nonexistent method named ``buyXX`` instead of ``buy``. .. code-block:: python import unittest from sample.portfolio import Portfolio class PortfolioTest(unittest.TestCase): def test_empty(self): p = Portfolio() self.assertEqual(p.cost(), 0.0) def test_buy_one_stock(self): p = Portfolio() p.buyXX("IBM", 100, 176.48) # this is wrong, to make the test fail! self.assertEqual(p.cost(), 17648.0) def test_buy_two_stocks(self): p = Portfolio() p.buy("IBM", 100, 176.48) p.buy("HPQ", 100, 36.15) self.assertEqual(p.cost(), 21263.0) If we run the tests again we'll see this output: .. code-block:: text E.. ====================================================================== ERROR: test_buy_one_stock (sample.tests.test_portfolio.PortfolioTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_portfolio.py", line 11, in test_buy_one_stock p.buyXX("IBM", 100, 176.48) # this is wrong, to make the test fail! AttributeError: 'Portfolio' object has no attribute 'buyXX' ---------------------------------------------------------------------- Ran 3 tests in 0.001s FAILED (errors=1) There are actually three possible outcomes for a test: passing, failure, and error. Error means that the test raised an exception other than ``AssertionError``. Well-written tests should either succeed, or should fail in an assertion. If another exception happens, it could either mean that the test is broken, or that the code is broken, but either way, it isn't what you intended. This condition is marked with an "E", and the error stacktrace is displayed. assertRaises ------------ The assertRaises helper method is a bit complex. The easiest way to use it is via a ``with`` statement. Add the following to the PortfolioTest class: .. code-block:: python def test_bad_input(self): p = Portfolio() with self.assertRaises(TypeError): p.buy("IBM") This test will pass. It asserts that the ``buy`` method will raise a TypeError when called with too few arguments. This is normal Python behvaior, and doesn't really need to be tested; this is just an example for purposes of explaining the helper method. Adding Coverage Reporting ------------------------- We can use the ``coverage`` tool to add objective metrics that help us evaluate the usefulness of our test suite. To do so, we'll switch our *test runner*. So far we've been using the built-in ``setuptools`` test runner. To easily get coverage statistics, instead, we're going to use the ``nose`` test runner. Let's change our ``setup.py`` to depend, in its ``test`` extra, on ``nose``. Change it from this: .. code-block:: python extras_require={ 'dev': ['check-manifest'], 'test': ['coverage'], }, To this: .. code-block:: python extras_require={ 'dev': ['check-manifest'], 'test': ['coverage', 'nose'], }, Then, we'll change our ``setup.cfg`` file to create an *alias*, so that when we type ``setup.py dev``, it will install both nose and coverage for us. Add the following stanza to ``setup.cfg``: .. code-block:: ini [aliases] dev = develop easy_install sample[test] Now that we've made those two changes, type ``python setup.py dev`` in your shell, and see that both ``nose`` and ``coverage`` get installed: .. code-block:: text # ... creating /home/chrism/projects/carscom/unittest_training/lib/python2.7/site-packages/coverage-4.0b2-py2.7-linux-x86_64.egg Extracting coverage-4.0b2-py2.7-linux-x86_64.egg to /home/chrism/projects/carscom/unittest_training/lib/python2.7/site-packages Adding coverage 4.0b2 to easy-install.pth file Installing coverage2 script to /home/chrism/projects/carscom/unittest_training/bin Installing coverage-2.7 script to /home/chrism/projects/carscom/unittest_training/bin Installing coverage script to /home/chrism/projects/carscom/unittest_training/bin Installed /home/chrism/projects/carscom/unittest_training/lib/python2.7/site-packages/coverage-4.0b2-py2.7-linux-x86_64.egg # ... Running Tests via ``nosetests`` ------------------------------- Now that we have ``nose`` installed, we can use its features. Let's first run ``python setup.py nosetests``. See that it produces output similar to that of running ``python setup.py test``. Once we verify that, we'll run it with the ``--with-coverage`` flag: .. code-block:: bash python setup.py nosetests --with-coverage --cover-package=sample .F ====================================================================== FAIL: test_failure (sample.tests.test_simple.TestSimple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_simple.py", line 9, in test_failure self.assertTrue(False) AssertionError: False is not true Name Stmts Miss Cover Missing --------------------------------------------------- sample.py 2 1 50% 4 sample/portfolio.py 31 16 48% 40-47, 52-55, 59-63 --------------------------------------------------- TOTAL 33 17 48% ---------------------------------------------------------------------- Ran 2 tests in 0.011s FAILED (failures=1) (Your output will differ slightly). The things we want to pay attention to are the ``Cover`` column and the ``Missing`` column. ``Cover`` is a percentage of statements in the module covered. A "statement" for purposes of this discussion is a single line of code. ``Missing`` is a list of lines that were not executed when the tests ran. Above, the ``sample.py`` file actually means the ``__init__.py`` file in the ``sample`` project. Its line 4 was not executed. The ``sample/portfolio.py`` file had unexecuted lines 40-47, 52-55, 59-63. Our goal will be to get rid of all the output in the ``Missing`` column and make the ``Cover`` column display ``100%`` for each module. Testing the ``Portfolio.sell()`` method --------------------------------------- Let's test the ``Portfolio.sell()`` method in real time, using ``setup.py nosetests`` to check our work. Coverage helps us see what we missed. - How many test methods should we use to try to test it (hint: it has a couple of conditions in it)? - What does ``for: else:`` mean? - ``pragma: no cover``. - ``--cover-html`` - Extra credit: branch coverage (``--cover-branches``). Uh Oh. Code That Integrates With Another System ------------------------------------------------ The code in our ``Portfolio.current_prices()`` method calls ``urllib.urlopen``, asking ``finance.yahoo.com`` what the value is of each stock that we own. The ``Portfolio.value()`` method calls the ``current_prices()`` method so it depends on the same external system. - What happens when ``finance.yahoo.com`` is unavailable? Maybe it's not even down; maybe you're on a plane. - Should we be able to test this code without the yahoo site being available? Yes! - Do we care that the yahoo site actually works in this context? No! - Unit tests test *units* of code. In this case we're concerned about the code in the ``value`` and ``current_prices`` methods. We are *only* concerned about testing that code, no other code. In our little unit testing world, we will assume that urllib.urlopen works, we assume that finance.yahoo.com works as requested and that it returns something that can be parsed by the CSV module. - Another reason to think of things this way: the yahoo site might be slow, which will make your tests slow, which makes it hard to write lots of tests that will be run frequently. - External dependencies are also unpredictable, which makes testing code which depends on one hard. For instance, in our case, the finance site might return values that differ every few minutes. Solution? Test doubles. Types of doubles: - Stubs. - Mocks. Stubs ------ We'll add a stub in a new test case to help us avoid depending on the "real" ``current_prices`` implementation. Add this class to the ``test_portfolio.py`` module: .. code-block:: python class PortfolioValueTest(unittest.TestCase): def stub_current_prices(self): return {'IBM':140.0, 'HPQ':32.0} def _makePortfolio(self): p = Portfolio() p.buy('IBM', 100, 120.0) p.buy('HPQ', 100, 30.0) p.current_prices = self.stub_current_prices return p def test_value(self): p = self._makePortfolio() self.assertEqual(p.value(), 17200) The magic of Python allows us to replace the method named ``current_prices`` on the portfolio we return to the `test_value`` method with one which always return a static dictionary, therefore the external system is avoided. But we can see if we run our coverage tests again that we haven't tested all the code. How do we fix that? Stubbing ``urllib`` -------------------- Change portfolio.py to add the following line to the class near the top: .. code-block:: python urllib = urllib And change the code in ``current_price`` to use ``self.urllib`` instead of ``urllib``. .. code-block:: python import urllib # .... class Portfolio(object): # ... urllib = urllib # ... def current_prices(self): """Return a dict mapping names to current prices.""" url = "http://finance.yahoo.com/d/quotes.csv?f=sl1&s=" url += ",".join(sorted(s[0] for s in self.stocks)) data = self.urllib.urlopen(url) return { sym: float(last) for sym, last in csv.reader(data) } Then in our test code we can change it so we do this: .. code-block:: python from StringIO import StringIO class FakeUrllib(object): def urlopen(self, url): return StringIO('"IBM",140\n"HPQ",32\n') class PortfolioValueTest(unittest.TestCase): def _makePortfolio(self): p = Portfolio() p.buy('IBM', 100, 120.0) p.buy('HPQ', 100, 30.0) p.urllib = FakeUrllib() return p def test_value(self): p = self._makePortfolio() self.assertEqual(p.value(), 17200) Now most of our code is tested as we can see from coverage reports. But we needed to change our code to be able to stubify it. Let's try it with a concept named ``mocks`` instead. Mocks ----- Get rid of the ``urllib = urllib`` line in the Portfolio class and change things back so that we do ``data = urllib.urlopen..`` instead of ``data= self.urllib.urlopen``. Now let's use a mock in our test. .. code-block:: python import mock from sample.portfolio import Portfolio class PortfolioValueTest(unittest.TestCase): def _makePortfolio(self): p = Portfolio() p.buy('IBM', 100, 120.0) p.buy('HPQ', 100, 30.0) return p def test_value(self): p = self._makePortfolio() with mock.patch('urllib.urlopen') as urlopen: fake_yahoo = StringIO('"IBM",140\n"HPQ",32\n') urlopen.return_value = fake_yahoo self.assertEqual(p.value(), 17200) urlopen.assert_called_with('http://finance.yahoo.com/d/quotes.csv?f=sl1&s=HPQ,IBM') Now we didn't need to mutate our code to make it testable. We were also able to make assertions about the value that was passed to urlopen pretty easily. Mocks or Stubs? --------------- - Yes. - Mocks have some downsides that we'll cover in a future section. In particular, they can make the reader of the testing code need to understand obscure bits of the Mock API, making the tests less understandbale. - Stubs have the downside that you often need to change your code to use them, and writing them isn't much fun. Discussion ---------- - What happens when code starts using ``requests`` instead of ``urllib``? - What kind of tradeoff could we choose to make in order to make our tests less fragile? Coda: setUp and tearDown ------------------------ ``unittest.TestCase`` subclasses can have setUp and TearDown methods. The code in each of these runs before/after each test that runs. I don't use these much because they're often used to manage globals, and I try not to have globals. Another thing they're used for is to get rid of boilerplate. - An example which gets rid of ``p = Portfolio()`` in each test by adding ``self.p = Portfolio()`` to setUp. Under the covers of setUp and tearDown: .. code-block:: python testcase = PortfolioTest() try: testcase.setUp() except: # [ record error ] else: try: testcase.test_method() except AssertionError: # [ record failure ] except: # [ record error ] else: # [ record success ] finally: try: testcase.tearDown() except: # [ record error ] - A helper method, ``_makePortfolio()`` instead. - I like the helper method instead.