Mechanics of Running Project Tests ================================== So far we've run "toy" tests where 1) we're not actually testing any external code and 2) all of the test code is defined in a single file. In the real world, it's almost certain that you'll be running the tests of a multifile project. Let's see how you might do that. Checking Out the Repository --------------------------- Check out the repository of the PyPA "sampleproject" into your virtual environment directory: .. code-block:: bash $ cd $VENV $ git clone https://github.com/pypa/sampleproject.git The result should be a ``sampleproject`` directory. Change directory into that directory: .. code-block:: bash $ cd sampleproject Installing the Project ---------------------- Let's then install the project into our virtualenv: .. code-block:: bash $ python setup.py develop Running The Tests ----------------- Now that we have it installed, let's run its tests: .. code-block:: bash .. code-block:: bash $ cd $VENV/sampleproject $ python -m unittest discover -s ./tests Here's the output we see: .. code-block:: text F ====================================================================== FAIL: test_failure (test_simple.TestSimple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/projects/carscom/training/tmp/sampleproject/tests/test_simple.py", line 9, in test_failure self.assertTrue(False) AssertionError: False is not true ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (failures=1) If we take a look at the file within ``sampleproject/tests/test_simple.py``, we see indeed that there is a test defined within it, and it's the test that got run. It failed because it's making an incorrect assertion. So if you see this, it worked. What Did I Just Do? ------------------- You ran the tests of a more "real-world" project than ones we've created so far, although it's realness is still pretty sparse. The project we checked out is the official "sample" project of the Python Packaging Authority (see https://packaging.python.org/en/latest/distributing.html). It is filled with comments in all of its files. Python packaging is a complex topic, but for our purposes, we just need to know that the ``setup.py`` file is important. It's what we use to install our package. The projects that you work on in your real job will also likely have a ``setup.py`` and will follow a similar directory layout. Alternate Way of Running The Tests ---------------------------------- I like to be able to run tests a slightly different way, because I don't like having to remember the ``python -m unittest`` command line switches. To do so I change the project's ``setup.py``. At line 111, after the trailing comma after ``entry_points={},`` add a carriage return and this line: .. code-block:: python test_suite="tests", The result should look something like this: .. code-block:: python # ... entry_points={ 'console_scripts': [ 'sample=sample:main', ], }, test_suite="tests", ) After we've done this, it is possible to run the tests like this: .. code-block:: bash $ python setup.py test And the output you'll see is something like this: .. code-block:: text running test running egg_info writing requirements to sample.egg-info/requires.txt writing sample.egg-info/PKG-INFO writing top-level names to sample.egg-info/top_level.txt writing dependency_links to sample.egg-info/dependency_links.txt writing entry points to sample.egg-info/entry_points.txt reading manifest file 'sample.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sample.egg-info/SOURCES.txt' running build_ext test_failure (tests.test_simple.TestSimple) ... FAIL ====================================================================== FAIL: test_failure (tests.test_simple.TestSimple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/pytraining/sampleproject/tests/test_simple.py", line 9, in test_failure self.assertTrue(False) AssertionError: False is not true ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (failures=1) As you notice, the output is slightly different than the output of the tests executed via ``python -m unittest``. We can make them more similar by passing the ``-q`` flag. .. code-block:: bash python setup.py test -q Now we see similar output to ``python -m unittest``. .. code-block:: bash running test running egg_info writing requirements to sample.egg-info/requires.txt writing sample.egg-info/PKG-INFO writing top-level names to sample.egg-info/top_level.txt writing dependency_links to sample.egg-info/dependency_links.txt writing entry points to sample.egg-info/entry_points.txt reading manifest file 'sample.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sample.egg-info/SOURCES.txt' running build_ext F ====================================================================== FAIL: test_failure (sample.tests.test_simple.TestSimple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/projects/carscom/unittest_training/realworld/sampleproject/sample/tests/test_simple.py", line 9, in test_failure self.assertTrue(False) AssertionError: False is not true ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (failures=1) Which Way? ---------- I personally prefer to stick a ``test_suite`` into my ``setup.py`` so I just need to remember, for whatever project I'm working on, that I can always run ``python setup.py test -q``. It's just easier to remember than the unittest command line switches. That said, it really makes very little difference, and alternate Python testing frameworks that we'll discuss later provide additional ways to do it that are sometimes even easier. Directory Layout of Test-Related Files --------------------------------------- Let's examine the directory layout of the ``sampleproject`` project: .. code-block:: text sampleproject/--\ | |-- setup.py | |-- sample/--\ | | | |-- __init__.py | | | |-- # files we dont care about yet | |-- tests/--\ | |-- __init__.py | |-- test_simple.py This irritates me because: #. We have *two* Python packages present within the project. One is named ``sample``, and the other is named ``tests``. #. Python packages are importable by name. When we do ``setup.py develop`` or ``setup.py install`` of this project, we'll be able to do ``import sample``. #. When we install the package, it *won't* install a top-level package named ``tests``, even though it exists within the project. Do you think it's a good idea to have a top-level package around named ``tests`` for people to trip over and get confused about? Me either. Let's fix it. Fixing It --------- Move the directory named ``tests`` from the ``sampleproject`` directory into the ``sampleproject/sample`` directory. .. code-block:: bash $ mv tests/ sample/ However, when we try to run the tests again: .. code-block:: bash $ python -m unittest discover -s ./tests/ We get this output: .. code-block:: text Traceback (most recent call last): File "/home/chrism/opt/Python-2.7.10/lib/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/home/chrism/opt/Python-2.7.10/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/__main__.py", line 12, in main(module=None) File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 94, in __init__ self.parseArgs(argv) File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 113, in parseArgs self._do_discovery(argv[2:]) File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 214, in _do_discovery self.test = loader.discover(start_dir, pattern, top_level_dir) File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/loader.py", line 204, in discover raise ImportError('Start directory is not importable: %r' % start_dir) ImportError: Start directory is not importable: './tests/' We have to change the command to point to the right path, which is now ``sample/tests``: .. code-block:: bash $ python -m unittest discover -s sample/tests Similarly when we run ``setup.py test`` we get an (inscrutable) error: .. code-block:: bash $ python setup.py test running test running egg_info writing requirements to sample.egg-info/requires.txt writing sample.egg-info/PKG-INFO writing top-level names to sample.egg-info/top_level.txt writing dependency_links to sample.egg-info/dependency_links.txt writing entry points to sample.egg-info/entry_points.txt reading manifest file 'sample.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sample.egg-info/SOURCES.txt' running build_ext Traceback (most recent call last): File "setup.py", line 111, in test_suite="tests", File "/home/chrism/opt/Python-2.7.10/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/home/chrism/opt/Python-2.7.10/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/home/chrism/opt/Python-2.7.10/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/chrism/projects/carscom/training/lib/python2.7/site-packages/setuptools/command/test.py", line 142, in run self.with_project_on_sys_path(self.run_tests) File "/home/chrism/projects/carscom/training/lib/python2.7/site-packages/setuptools/command/test.py", line 122, in with_project_on_sys_path func() File "/home/chrism/projects/carscom/training/lib/python2.7/site-packages/setuptools/command/test.py", line 163, in run_tests testRunner=self._resolve_as_ep(self.test_runner), File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 94, in __init__ self.parseArgs(argv) File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 149, in parseArgs self.createTests() File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/main.py", line 158, in createTests self.module) File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/loader.py", line 130, in loadTestsFromNames suites = [self.loadTestsFromName(name, module) for name in names] File "/home/chrism/opt/Python-2.7.10/lib/python2.7/unittest/loader.py", line 91, in loadTestsFromName module = __import__('.'.join(parts_copy)) ImportError: No module named tests We need to change setup.py. Change this line: .. code-block:: text test_suite="tests", To: .. code-block:: text test_suite="sample.tests", Note that there's a dot instead of a slash there. It's because we're referring to a *module* instead of a disk path in setup.py's ``test_suite``. Now when we run the tests, we get expected output: .. code-block:: text running test running egg_info writing requirements to sample.egg-info/requires.txt writing sample.egg-info/PKG-INFO writing top-level names to sample.egg-info/top_level.txt writing dependency_links to sample.egg-info/dependency_links.txt writing entry points to sample.egg-info/entry_points.txt reading manifest file 'sample.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'sample.egg-info/SOURCES.txt' running build_ext test_failure (sample.tests.test_simple.TestSimple) ... FAIL ====================================================================== FAIL: test_failure (sample.tests.test_simple.TestSimple) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/chrism/projects/carscom/training/tmp/sampleproject/sample/tests/test_simple.py", line 9, in test_failure self.assertTrue(False) AssertionError: False is not true ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (failures=1) Conclusions ----------- - Put tests *inside* the package you're working on, not outside. - Reasonable people disagree about this. The people who put tests outside claim that it's better because the tests don't ship with the code when it's distributed to production. I don't agree that this is better. - The ``test_suite`` line in a setup.py refers to a *module*, while the command line path as an argument to ``-s`` in ``python -m unittest discover -s foo`` refers to a disk path. Yes, it's not sane.