Mechanics of Running Project Tests

So far we’ve run “toy” tests where 1) we’re not actually testing any external code and 2) all of the test code is defined in a single file.

In the real world, it’s almost certain that you’ll be running the tests of a multifile project. Let’s see how you might do that.

Checking Out the Repository

Check out the repository of the PyPA “sampleproject” into your virtual environment directory:

$ cd $VENV
$ git clone https://github.com/pypa/sampleproject.git

The result should be a sampleproject directory. Change directory into that directory:

$ cd sampleproject

Installing the Project

Let’s then install the project into our virtualenv:

$ python setup.py develop

Running The Tests

Now that we have it installed, let’s run its tests:

$ cd $VENV/sampleproject
$ python -m unittest discover -s ./tests

Here’s the output we see:

F
======================================================================
FAIL: test_failure (test_simple.TestSimple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/carscom/training/tmp/sampleproject/tests/test_simple.py", line 9, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

If we take a look at the file within sampleproject/tests/test_simple.py, we see indeed that there is a test defined within it, and it’s the test that got run. It failed because it’s making an incorrect assertion. So if you see this, it worked.

What Did I Just Do?

You ran the tests of a more “real-world” project than ones we’ve created so far, although it’s realness is still pretty sparse.

The project we checked out is the official “sample” project of the Python Packaging Authority (see https://packaging.python.org/en/latest/distributing.html). It is filled with comments in all of its files. Python packaging is a complex topic, but for our purposes, we just need to know that the setup.py file is important. It’s what we use to install our package.

The projects that you work on in your real job will also likely have a setup.py and will follow a similar directory layout.

Alternate Way of Running The Tests

I like to be able to run tests a slightly different way, because I don’t like having to remember the python -m unittest command line switches. To do so I change the project’s setup.py.

At line 111, after the trailing comma after entry_points={}, add a carriage return and this line:

test_suite="tests",

The result should look something like this:

# ...

    entry_points={
        'console_scripts': [
            'sample=sample:main',
        ],
    },

    test_suite="tests",
)

After we’ve done this, it is possible to run the tests like this:

$ python setup.py test

And the output you’ll see is something like this:

running test
running egg_info
writing requirements to sample.egg-info/requires.txt
writing sample.egg-info/PKG-INFO
writing top-level names to sample.egg-info/top_level.txt
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'
running build_ext
test_failure (tests.test_simple.TestSimple) ... FAIL

======================================================================
FAIL: test_failure (tests.test_simple.TestSimple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/pytraining/sampleproject/tests/test_simple.py", line 9, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

As you notice, the output is slightly different than the output of the tests executed via python -m unittest. We can make them more similar by passing the -q flag.

python setup.py test -q

Now we see similar output to python -m unittest.

running test
running egg_info
writing requirements to sample.egg-info/requires.txt
writing sample.egg-info/PKG-INFO
writing top-level names to sample.egg-info/top_level.txt
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'
running build_ext
F
======================================================================
FAIL: test_failure (sample.tests.test_simple.TestSimple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_simple.py", line 9, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

Which Way?

I personally prefer to stick a test_suite into my setup.py so I just need to remember, for whatever project I’m working on, that I can always run python setup.py test -q. It’s just easier to remember than the unittest command line switches. That said, it really makes very little difference, and alternate Python testing frameworks that we’ll discuss later provide additional ways to do it that are sometimes even easier.

Directory Layout of Test-Related Files

Let’s examine the directory layout of the sampleproject project:

sampleproject/--\
                |
                |-- setup.py
                |
                |-- sample/--\
                |            |
                |            |-- __init__.py
                |            |
                |            |-- # files we dont care about yet
                |
                |-- tests/--\
                             |
                             |-- __init__.py
                             |
                             |-- test_simple.py

This irritates me because:

1. We have two Python packages present within the project. One is named sample, and the other is named tests.
2. Python packages are importable by name. When we do setup.py develop or setup.py install of this project, we’ll be able to do import sample.
3. When we install the package, it won’t install a top-level package named tests, even though it exists within the project.

Do you think it’s a good idea to have a top-level package around named tests for people to trip over and get confused about? Me either. Let’s fix it.

Fixing It

Move the directory named tests from the sampleproject directory into the sampleproject/sample directory.

$ mv tests/ sample/

However, when we try to run the tests again:

$ python -m unittest discover -s ./tests/

We get this output:

Traceback (most recent call last):
  File "/home/chrism/opt/Python-2.7.10/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/chrism/opt/Python-2.7.10/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/__main__.py", line 12, in <module>
    main(module=None)
  File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 94, in __init__
    self.parseArgs(argv)
  File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 113, in parseArgs
    self._do_discovery(argv[2:])
  File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 214, in _do_discovery
    self.test = loader.discover(start_dir, pattern, top_level_dir)
  File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/loader.py", line 204, in discover
    raise ImportError('Start directory is not importable: %r' % start_dir)
ImportError: Start directory is not importable: './tests/'

We have to change the command to point to the right path, which is now sample/tests:

$ python -m unittest discover -s sample/tests

Similarly when we run setup.py test we get an (inscrutable) error:

$ python setup.py test

 running test
 running egg_info
 writing requirements to sample.egg-info/requires.txt
 writing sample.egg-info/PKG-INFO
 writing top-level names to sample.egg-info/top_level.txt
 writing dependency_links to sample.egg-info/dependency_links.txt
 writing entry points to sample.egg-info/entry_points.txt
 reading manifest file 'sample.egg-info/SOURCES.txt'
 reading manifest template 'MANIFEST.in'
 writing manifest file 'sample.egg-info/SOURCES.txt'
 running build_ext
 Traceback (most recent call last):
   File "setup.py", line 111, in <module>
     test_suite="tests",
   File "/home/chrism/opt/Python-2.7.10/lib/python2.7/distutils/core.py", line 151, in setup
     dist.run_commands()
   File "/home/chrism/opt/Python-2.7.10/lib/python2.7/distutils/dist.py", line 953, in run_commands
     self.run_command(cmd)
   File "/home/chrism/opt/Python-2.7.10/lib/python2.7/distutils/dist.py", line 972, in run_command
     cmd_obj.run()
   File "/home/chrism/projects/carscom/training/lib/python2.7/site-packages/setuptools/command/test.py", line 142, in run
     self.with_project_on_sys_path(self.run_tests)
   File "/home/chrism/projects/carscom/training/lib/python2.7/site-packages/setuptools/command/test.py", line 122, in with_project_on_sys_path
     func()
   File "/home/chrism/projects/carscom/training/lib/python2.7/site-packages/setuptools/command/test.py", line 163, in run_tests
     testRunner=self._resolve_as_ep(self.test_runner),
   File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 94, in __init__
     self.parseArgs(argv)
   File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 149, in parseArgs
     self.createTests()
   File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 158, in createTests
     self.module)
   File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames
     suites = [self.loadTestsFromName(name, module) for name in names]
   File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName
     module = __import__('.'.join(parts_copy))
 ImportError: No module named tests

We need to change setup.py. Change this line:

test_suite="tests",

To:

test_suite="sample.tests",

Note that there’s a dot instead of a slash there. It’s because we’re referring to a module instead of a disk path in setup.py’s test_suite.

Now when we run the tests, we get expected output:

running test
running egg_info
writing requirements to sample.egg-info/requires.txt
writing sample.egg-info/PKG-INFO
writing top-level names to sample.egg-info/top_level.txt
writing dependency_links to sample.egg-info/dependency_links.txt
writing entry points to sample.egg-info/entry_points.txt
reading manifest file 'sample.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'sample.egg-info/SOURCES.txt'
running build_ext
test_failure (sample.tests.test_simple.TestSimple) ... FAIL

======================================================================
FAIL: test_failure (sample.tests.test_simple.TestSimple)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/chrism/projects/carscom/training/tmp/sampleproject/sample/tests/test_simple.py", line 9, in test_failure
    self.assertTrue(False)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=1)

Conclusions

• Put tests inside the package you’re working on, not outside.
• Reasonable people disagree about this. The people who put tests outside claim that it’s better because the tests don’t ship with the code when it’s distributed to production. I don’t agree that this is better.
• The test_suite line in a setup.py refers to a module, while the command line path as an argument to -s in python -m unittest discover -s foo refers to a disk path. Yes, it’s not sane.