502

I have a directory that contains my Python unit tests. Each unit test module is of the form test_*.py. I am attempting to make a file called all_test.py that will, you guessed it, run all files in the aforementioned test form and return the result. I have tried two methods so far; both have failed. I will show the two methods, and I hope someone out there knows how to actually do this correctly.

For my first valiant attempt, I thought "If I just import all my testing modules in the file, and then call this unittest.main() doodad, it will work, right?" Well, turns out I was wrong.

import glob import unittest testSuite = unittest.TestSuite() test_file_strings = glob.glob('test_*.py') module_strings = [str[0:len(str)-3] for str in test_file_strings] if __name__ == "__main__": unittest.main()

This did not work, the result I got was:

$ python all_test.py ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK

For my second try, I though, ok, maybe I will try to do this whole testing thing in a more "manual" fashion. So I attempted to do that below:

import glob import unittest testSuite = unittest.TestSuite() test_file_strings = glob.glob('test_*.py') module_strings = [str[0:len(str)-3] for str in test_file_strings] [__import__(str) for str in module_strings] suites = [unittest.TestLoader().loadTestsFromName(str) for str in module_strings] [testSuite.addTest(suite) for suite in suites] print testSuite result = unittest.TestResult() testSuite.run(result) print result #Ok, at this point I have a result #How do I display it as the normal unit test command line output? if __name__ == "__main__": unittest.main()

This also did not work, but it seems so close!

$ python all_test.py <unittest.TestSuite tests=[<unittest.TestSuite tests=[<unittest.TestSuite tests=[<test_main.TestMain testMethod=test_respondes_to_get>]>]>]> <unittest.TestResult run=1 errors=0 failures=0> ---------------------------------------------------------------------- Ran 0 tests in 0.000s OK

I seem to have a suite of some sort, and I can execute the result. I am a little concerned about the fact that it says I have only run=1, seems like that should be run=2, but it is progress. But how do I pass and display the result to main? Or how do I basically get it working so I can just run this file, and in doing so, run all the unit tests in this directory?

Improve this question
2
  • did you ever try running the tests from an test instance object? Commented Jun 24, 2017 at 23:16
  • See this answer for a solution with an example file structure. Commented Jun 28, 2017 at 23:48

18 Answers 18

Reset to default
747

With Python 2.7 and higher you don't have to write new code or use third-party tools to do this; recursive test execution via the command line is built-in. Put an __init__.py in your test directory and:

python -m unittest discover <test_directory> # or python -m unittest discover -s <directory> -p '*_test.py'

You can read more in the python 2.7 or python 3.x unittest documentation.

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

problems include: ImportError: Start directory is not importable:
At least with Python 2.7.8 on Linux neither command line invocation gives me recursion. My project has several subprojects whose unit tests live in respective "unit_tests/<subproject>/python/" directories. If I specify such a path the unit tests for that subproject are run, but with just "unit_tests" as test directory argument no tests are found (instead of all tests for all subprojects, as I hoped). Any hint?
About recursion: The first command without a <test_directory> defaults to "." and recurses to submodules. That is, all tests directories you want discovered needs to have a init.py. If they do, they will get found by the discover command. Just tried it, it worked.
did you ever try running the tests from an test instance object?
This works for me even if I don't specify -s and -p args. Just write python -m unittest discover
|
307

In python 3, if you're using unittest.TestCase:

  • You must have an empty (or otherwise) __init__.py file in your test directory (must be named test/)
  • Your test files inside test/ match the pattern test_*.py.
    • They can be inside a subdirectory under test/. Those subdirs can be named as anything, but they all need to have an __init__.py file in them
  • Your test function names should start with test_. Ex: test_text_matcher

Then, you can run all the tests with:

python -m unittest

Done! A solution less than 100 lines. Hopefully another python beginner saves time by finding this.

Improve this answer

11 Comments

Note that by default it only searches for tests in filenames beginning with "test"
That's correct, the original question referred to the fact that "Each unit test module is of the form test_*.py.", so this answer in direct reply. I've now updated the answer to be more explicit
Thanks, that what was missing for me to use Travis Bear's answer.
I also needed to add the init.py file to each subfolder for it to work, otherwise great. Thanks!
Can you update your answer to include that the subdirectories need to be packages as well, so that you need to add an init.py file to the subdirectories inside the test directory?
|
112

You could use a test runner that would do this for you. nose is very good for example. When run, it will find tests in the current tree and run them.

Updated:

Here's some code from my pre-nose days. You probably don't want the explicit list of module names, but maybe the rest will be useful to you.

testmodules = [ 'cogapp.test_makefiles', 'cogapp.test_whiteutils', 'cogapp.test_cogapp', ] suite = unittest.TestSuite() for t in testmodules: try: # If the module defines a suite() function, call it to get the suite. mod = __import__(t, globals(), locals(), ['suite']) suitefn = getattr(mod, 'suite') suite.addTest(suitefn()) except (ImportError, AttributeError): # else, just load all the test cases from the module. suite.addTest(unittest.defaultTestLoader.loadTestsFromName(t)) unittest.TextTestRunner().run(suite)
Improve this answer

5 Comments

Is the advantage of this approach over just explicitly importing all of your test modules in to one test_all.py module and calling unittest.main() that you can optionally declare a test suite in some modules and not in others?
I tried out nose and it works perfectly. It was easy to install and run in my project. I was even able to automate it with a few lines of script, running inside a virtualenv. +1 for nose!
Not always doable: sometimes importing structure of the project can lead to nose getting confused if it tries to run the imports on modules.
Note that nose has been "in maintenance mode for the past several years" and it is currently advised to use nose2, pytest, or just plain unittest / unittest2 for new projects.
did you ever try running the tests from an test instance object?
96

This is now possible directly from unittest: unittest.TestLoader.discover.

import unittest loader = unittest.TestLoader() start_dir = 'path/to/your/test/files' suite = loader.discover(start_dir) runner = unittest.TextTestRunner() runner.run(suite)
Improve this answer

6 Comments

I have tried this method also, have couple tests, but works perfectly. Excellent!!! But I'm curious I have only 4 tests. Together they run 0.032s, but when I use this method to run them all, i get result .... ---------------------------------------------------------------------- Ran 4 tests in 0.000s OK Why? The difference, where it comes from?
I am having trouble running a file that looks like this from the command line. How should it be invoked?
python file.py
Worked flawlessly! Just set it in your test/ dir and then set the start_id = "./" . IMHO, this answer is now (Python 3.7) the accepted way!
You can change the last line to ´res = runner.run(suite); sys.exit(0 if res.wasSuccessful() else 1)´ if you want a correct exit code
|
33

Well by studying the code above a bit (specifically using TextTestRunner and defaultTestLoader), I was able to get pretty close. Eventually I fixed my code by also just passing all test suites to a single suites constructor, rather than adding them "manually", which fixed my other problems. So here is my solution.

import glob import unittest test_files = glob.glob('test_*.py') module_strings = [test_file[0:len(test_file)-3] for test_file in test_files] suites = [unittest.defaultTestLoader.loadTestsFromName(test_file) for test_file in module_strings] test_suite = unittest.TestSuite(suites) test_runner = unittest.TextTestRunner().run(test_suite)

Yeah, it is probably easier to just use nose than to do this, but that is besides the point.

Improve this answer

3 Comments

good, it works fine for the current directory, how to invoke the sub-directly ?
Larry, see the new answer (stackoverflow.com/a/24562019/104143) for recursive test discovery
did you ever try running the tests from an test instance object?
29

If you want to run all the tests from various test case classes and you're happy to specify them explicitly then you can do it like this: 

from unittest import TestLoader, TextTestRunner, TestSuite from uclid.test.test_symbols import TestSymbols from uclid.test.test_patterns import TestPatterns if __name__ == "__main__": loader = TestLoader() tests = [ loader.loadTestsFromTestCase(test) for test in (TestSymbols, TestPatterns) ] suite = TestSuite(tests) runner = TextTestRunner(verbosity=2) runner.run(suite)

where uclid is my project and TestSymbols and TestPatterns are subclasses of TestCase.

Improve this answer

3 Comments

From the unittest.TestLoader docs: "Normally, there is no need to create an instance of this class; the unittest module provides an instance that can be shared as unittest.defaultTestLoader." Also since TestSuite accepts an iterable as an argument, you can construct said iterable in a loop to avoid repeating loader.loadTestsFromTestCase.
@Two-Bit Alchemist your second point in particular is nice. I'd change the code to include but I can't test it. (First mod would make it look like too much like Java for my liking.. though I realize I'm being irrational (screw them an their camel case variable names)).
This is my fav, very clean. Was able to package this and make it an argument in my regular command line.
15

I have used the discover method and an overloading of load_tests to achieve this result in a (minimal, I think) number lines of code:

def load_tests(loader, tests, pattern): ''' Discover and load all unit tests in all files named ``*_test.py`` in ``./src/`` ''' suite = TestSuite() for all_test_suite in unittest.defaultTestLoader.discover('src', pattern='*_tests.py'): for test_suite in all_test_suite: suite.addTests(test_suite) return suite if __name__ == '__main__': unittest.main()

Execution on fives something like

Ran 27 tests in 0.187s OK
Improve this answer

4 Comments

this is available for python2.7 only, I guess
@larrycai Maybe, I am usually on Python 3, sometimes Python 2.7. The question was not tied to a specific version.
I'm on Python 3.4 and discover returns a suite, making the loop redundant.
For future Larry's: "Many new features were added to unittest in Python 2.7, including test discovery. unittest2 allows you to use these features with earlier versions of Python."
9

I tried various approaches but all seem flawed or I have to makeup some code, that's annoying. But there's a convinient way under linux, that is simply to find every test through certain pattern and then invoke them one by one.

find . -name 'Test*py' -exec python '{}' \;

and most importantly, it definitely works.

Improve this answer

Comments

9

In case of a packaged library or application, you don't want to do it. setuptools will do it for you.

To use this command, your project’s tests must be wrapped in a unittest test suite by either a function, a TestCase class or method, or a module or package containing TestCase classes. If the named suite is a module, and the module has an additional_tests() function, it is called and the result (which must be a unittest.TestSuite) is added to the tests to be run. If the named suite is a package, any submodules and subpackages are recursively added to the overall test suite.

Just tell it where your root test package is, like:

setup( # ... test_suite = 'somepkg.test' )

And run python setup.py test.

File-based discovery may be problematic in Python 3, unless you avoid relative imports in your test suite, because discover uses file import. Even though it supports optional top_level_dir, but I had some infinite recursion errors. So a simple solution for a non-packaged code is to put the following in __init__.py of your test package (see load_tests Protocol).

import unittest from . import foo, bar def load_tests(loader, tests, pattern): suite = unittest.TestSuite() suite.addTests(loader.loadTestsFromModule(foo)) suite.addTests(loader.loadTestsFromModule(bar)) return suite
Improve this answer

Comments

5

This is an old question, but what worked for me now (in 2019) is:

python -m unittest *_test.py

All my test files are in the same folder as the source files and they end with _test.

Improve this answer

Comments

4

I use PyDev/LiClipse and haven't really figured out how to run all tests at once from the GUI. (edit: you right click the root test folder and choose Run as -> Python unit-test

This is my current workaround:

import unittest def load_tests(loader, tests, pattern): return loader.discover('.') if __name__ == '__main__': unittest.main()

I put this code in a module called all in my test directory. If I run this module as a unittest from LiClipse then all tests are run. If I ask to only repeat specific or failed tests then only those tests are run. It doesn't interfere with my commandline test runner either (nosetests) -- it's ignored.

You may need to change the arguments to discover based on your project setup.

Improve this answer

1 Comment

The names of all test files and test methods should start with "test_". Otherwise the command "Run as -> Python unit test" wont find them.
3

Based on the answer of Stephen Cagle I added support for nested test modules.

import fnmatch import os import unittest def all_test_modules(root_dir, pattern): test_file_names = all_files_in(root_dir, pattern) return [path_to_module(str) for str in test_file_names] def all_files_in(root_dir, pattern): matches = [] for root, dirnames, filenames in os.walk(root_dir): for filename in fnmatch.filter(filenames, pattern): matches.append(os.path.join(root, filename)) return matches def path_to_module(py_file): return strip_leading_dots( \ replace_slash_by_dot( \ strip_extension(py_file))) def strip_extension(py_file): return py_file[0:len(py_file) - len('.py')] def replace_slash_by_dot(str): return str.replace('\\', '.').replace('/', '.') def strip_leading_dots(str): while str.startswith('.'): str = str[1:len(str)] return str module_names = all_test_modules('.', '*Tests.py') suites = [unittest.defaultTestLoader.loadTestsFromName(mname) for mname in module_names] testSuite = unittest.TestSuite(suites) runner = unittest.TextTestRunner(verbosity=1) runner.run(testSuite)

The code searches all subdirectories of . for *Tests.py files which are then loaded. It expects each *Tests.py to contain a single class *Tests(unittest.TestCase) which is loaded in turn and executed one after another.

This works with arbitrary deep nesting of directories/modules, but each directory in between needs to contain an empty __init__.py file at least. This allows the test to load the nested modules by replacing slashes (or backslashes) by dots (see replace_slash_by_dot).

Improve this answer

Comments

3

I just created a discover.py file in my base test directory and added import statements for anything in my sub directories. Then discover is able to find all my tests in those directories by running it on discover.py

python -m unittest discover ./test -p '*.py' 
# /test/discover.py import unittest from test.package1.mod1 import XYZTest from test.package1.package2.mod2 import ABCTest ... if __name__ == "__main__" unittest.main()
Improve this answer

Comments

3

Encountered the same issue.

The solution is to add an empty __init__.py to each folder and uses python -m unittest discover -s

Project Structure

tests/ __init__.py domain/ value_object/ __init__.py test_name.py __init__.py presentation/ __init__.py test_app.py

And running the command

python -m unittest discover -s tests/domain

To get the expected outcome

. ---------------------------------------------------------------------- Ran 1 test in 0.007s
Improve this answer

Comments

1

Because Test discovery seems to be a complete subject, there is some dedicated framework to test discovery :

More reading here : https://wiki.python.org/moin/PythonTestingToolsTaxonomy

Improve this answer

Comments

1

This BASH script will execute the python unittest test directory from ANYWHERE in the file system, no matter what working directory you are in: its working directory always be where that test directory is located.

ALL TESTS, independent $PWD

unittest Python module is sensitive to your current directory, unless you tell it where (using discover -s option).

This is useful when staying in the ./src or ./example working directory and you need a quick overall unit test:

#!/bin/bash this_program="$0" dirname="`dirname $this_program`" readlink="`readlink -e $dirname`" python -m unittest discover -s "$readlink"/test -v

SELECTED TESTS, independent $PWD

I name this utility file: runone.py and use it like this:

runone.py <test-python-filename-minus-dot-py-fileextension> 
#!/bin/bash this_program="$0" dirname="`dirname $this_program`" readlink="`readlink -e $dirname`" (cd "$dirname"/test; python -m unittest $1)

No need for a test/__init__.py file to burden your package/memory-overhead during production.

Improve this answer

Comments

-1

I have no package and as mentioned on this page, this is creating issue while issing dicovery. So, I used the following solution. All the test result will be put in a given output folder.

RunAllUT.py:

""" The given script is executing all the Unit Test of the project stored at the path %relativePath2Src% currently fixed coded for the given project. Prerequired: - Anaconda should be install - For the current user, an enviornment called "mtToolsEnv" should exists - xmlrunner Library should be installed """ import sys import os import xmlrunner from Repository import repository relativePath2Src="./../.." pythonPath=r'"C:\Users\%USERNAME%\.conda\envs\YourConfig\python.exe"' outputTestReportFolder=os.path.dirname(os.path.abspath(__file__))+r'\test-reports' #subfolder in current file path class UTTesting(): """ Class tto run all the UT of the project """ def __init__(self): """ Initiate instance Returns ------- None. """ self.projectRepository = repository() self.UTfile = [] #List all file def retrieveAllUT(self): """ Generate the list of UT file in the project Returns ------- None. """ print(os.path.realpath(relativePath2Src)) self.projectRepository.retriveAllFilePaths(relativePath2Src) #self.projectRepository.printAllFile() #debug for file2scan in self.projectRepository.devfile: if file2scan.endswith("_UT.py"): self.UTfile.append(file2scan) print(self.projectRepository.devfilepath[file2scan]+'/'+file2scan) def runUT(self,UTtoRun): """ Run a single UT Parameters ---------- UTtoRun : String File Name of the UT Returns ------- None. """ print(UTtoRun) if UTtoRun in self.projectRepository.devfilepath: UTtoRunFolderPath=os.path.realpath(os.path.join(self.projectRepository.devfilepath[UTtoRun])) UTtoRunPath = os.path.join(UTtoRunFolderPath, UTtoRun) print(UTtoRunPath) #set the correct execution context & run the test os.system(" cd " + UTtoRunFolderPath + \ " & " + pythonPath + " " + UTtoRunPath + " " + outputTestReportFolder ) def runAllUT(self): """ Run all the UT contained in self The function "retrieveAllUT" sjould ahve been performed before Returns ------- None. """ for UTfile in self.UTfile: self.runUT(UTfile) if __name__ == "__main__": undertest=UTTesting() undertest.retrieveAllUT() undertest.runAllUT()

In my project specific, I have a class that I used in other script. This might be an overkill for your usecase.

Repository.py

import os class repository(): """ Class that decribed folder and file in a repository """ def __init__(self): """ Initiate instance Returns ------- None. """ self.devfile = [] #List all file self.devfilepath = {} #List all file paths def retriveAllFilePaths(self,pathrepo): """ Retrive all files and their path in the class Parameters ---------- pathrepo : Path used for the parsin Returns ------- None. """ for path, subdirs, files in os.walk(pathrepo): for file_name in files: self.devfile.append(file_name) self.devfilepath[file_name] = path def printAllFile(self): """ Display all file with paths Parameters ---------- def printAllFile : TYPE DESCRIPTION. Returns ------- None. """ for file_loop in self.devfile: print(self.devfilepath[file_loop]+'/'+file_loop)

In your test files, you need to have a main like this:

if __name__ == "__main__": import xmlrunner import sys if len(sys.argv) > 1: outputFolder = sys.argv.pop() #avoid conflic with unittest.main else: outputFolder = r'test-reports' print("Report will be created and store there: " + outputFolder) unittest.main(testRunner=xmlrunner.XMLTestRunner(output=outputFolder))
Improve this answer

Comments

-3

Here is my approach by creating a wrapper to run tests from the command line:

#!/usr/bin/env python3 import os, sys, unittest, argparse, inspect, logging if __name__ == '__main__': # Parse arguments. parser = argparse.ArgumentParser(add_help=False) parser.add_argument("-?", "--help", action="help", help="show this help message and exit" ) parser.add_argument("-v", "--verbose", action="store_true", dest="verbose", help="increase output verbosity" ) parser.add_argument("-d", "--debug", action="store_true", dest="debug", help="show debug messages" ) parser.add_argument("-h", "--host", action="store", dest="host", help="Destination host" ) parser.add_argument("-b", "--browser", action="store", dest="browser", help="Browser driver.", choices=["Firefox", "Chrome", "IE", "Opera", "PhantomJS"] ) parser.add_argument("-r", "--reports-dir", action="store", dest="dir", help="Directory to save screenshots.", default="reports") parser.add_argument('files', nargs='*') args = parser.parse_args() # Load files from the arguments. for filename in args.files: exec(open(filename).read()) # See: http://codereview.stackexchange.com/q/88655/15346 def make_suite(tc_class): testloader = unittest.TestLoader() testnames = testloader.getTestCaseNames(tc_class) suite = unittest.TestSuite() for name in testnames: suite.addTest(tc_class(name, cargs=args)) return suite # Add all tests. alltests = unittest.TestSuite() for name, obj in inspect.getmembers(sys.modules[__name__]): if inspect.isclass(obj) and name.startswith("FooTest"): alltests.addTest(make_suite(obj)) # Set-up logger verbose = bool(os.environ.get('VERBOSE', args.verbose)) debug = bool(os.environ.get('DEBUG', args.debug)) if verbose or debug: logging.basicConfig( stream=sys.stdout ) root = logging.getLogger() root.setLevel(logging.INFO if verbose else logging.DEBUG) ch = logging.StreamHandler(sys.stdout) ch.setLevel(logging.INFO if verbose else logging.DEBUG) ch.setFormatter(logging.Formatter('%(asctime)s %(levelname)s: %(name)s: %(message)s')) root.addHandler(ch) else: logging.basicConfig(stream=sys.stderr) # Run tests. result = unittest.TextTestRunner(verbosity=2).run(alltests) sys.exit(not result.wasSuccessful())

For sake of simplicity, please excuse my non-PEP8 coding standards.

Then you can create BaseTest class for common components for all your tests, so each of your test would simply look like:

from BaseTest import BaseTest class FooTestPagesBasic(BaseTest): def test_foo(self): driver = self.driver driver.get(self.base_url + "/")

To run, you simply specifying tests as part of the command line arguments, e.g.:

./run_tests.py -h http://example.com/ tests/**/*.py
Improve this answer

1 Comment

most of this answer has nothing to do with test discovery (i.e logging, etc). Stack Overflow is for answering questions, not showing off unrelated code.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.