JUnit test with dynamic number of tests

JUnit 3

public class XTest extends TestCase { public File file; public XTest(File file) { super(file.toString()); this.file = file; } public void testX() { fail("Failed: " + file); } } public class XTestSuite extends TestSuite { public static Test suite() { TestSuite suite = new TestSuite("XTestSuite"); File[] files = new File(".").listFiles(); for (File file : files) { suite.addTest(new XTest(file)); } return suite; } } 

JUnit 4

import org.junit.Test; import org.junit.runner.RunWith; import org.junit.runners.Parameterized; import org.junit.runners.Parameterized.Parameters; @RunWith(Parameterized.class) public class TestY { @Parameters public static Collection<Object[]> getFiles() { Collection<Object[]> params = new ArrayList<Object[]>(); for (File f : new File(".").listFiles()) { Object[] arr = new Object[] { f }; params.add(arr); } return params; } private File file; public TestY(File file) { this.file = file; } @Test public void testY() { fail(file.toString()); } } 

Junit 5 Parameterized Tests

JUnit 5 parameterized tests support this by allowing the use of a method as data source:

@ParameterizedTest @MethodSource("fileProvider") void testFile(File f) { // Your test comes here } static Stream<File> fileProvider() { return Arrays.asList(new File(".").list()).stream(); } 

JUnit 5 DynamicTests

JUnit 5 also supports this through the notion of a DynamicTest, which is to be generated in a @TestFactory, by means of the static method dynamicTest.

import org.junit.jupiter.api.DynamicTest; import org.junit.jupiter.api.TestFactory; import static org.junit.jupiter.api.DynamicTest.dynamicTest; import java.util.stream.Stream; @TestFactory public Stream<DynamicTest> testFiles() { return Arrays.asList(new File(".").list()) .stream() .map((file) -> dynamicTest( "Test for file: " + file, () -> { /* Your test comes here */ })); } 

The tests run in your IDE (IntelliJ here) will be displayed like this:

Should be possible in JUnit 3 by inheriting from TestSuite and overriding the tests() method to list the files and for each return an instance of a subclass of TestCase that takes the filename as constructor parameter and has a test method that tests the file given in the constructor.

In JUnit 4 it might be even easier.

You could consider using JUnitParams library, so you would have a few more (cleaner) options:

@org.junit.runner.RunWith(junitparams.JUnitParamsRunner.class) public class ParameterizedTest { @org.junit.Test @junitparams.Parameters(method = "data") public void test1(File file) throws Exception { } @org.junit.Test @junitparams.Parameters(method = "data") public void test2(File file) throws Exception { } public static File[] data() { return new File[] { new File("path1"), new File("path2") }; } } 

@org.junit.runner.RunWith(junitparams.JUnitParamsRunner.class) public class ParameterizedTest { @org.junit.Test @junitparams.Parameters(value = { "path1", "path2" }) public void test1(String path) throws Exception { File file = new File(path); } @org.junit.Test @junitparams.Parameters(value = { "path1", "path2" }) public void test2(String path) throws Exception { File file = new File(path); } } 

You can see more samples of usage here.

In addition about JUnitParams, why writting parameterized tests with it is easier and more readable:

JUnitParams project adds a new runner to JUnit and provides much easier and readable parametrised tests for JUnit >=4.6.

Main differences to standard JUnit Parametrised runner:

• more explicit - params are in test method params, not class fields
• less code - you don't need a constructor to set up parameters
• you can mix parametrised with non-parametrised methods in one class
• params can be passed as a CSV string or from a parameters provider class
• parameters provider class can have as many parameters providing methods as you want, so that you can group different cases
• you can have a test method that provides parameters (no external classes or statics anymore)
• you can see actual parameter values in your IDE (in JUnit's Parametrised it's only consecutive numbers of parameters)

If TestNG is an option, you could use Parameters with DataProviders.

Each individual file's test will have its result shown in the text-based report or Eclipse's TestNG plugin UI. The number of total tests run will count each of your files individually.

This behavior differs from JUnit Theories, in which all results are lumped under one "theory" entry and only count as 1 test. If you want separate result reporting in JUnit, you can try Parameterized Tests.

Test and inputs

public class FileTest { @DataProvider(name="files") public File[][] getFiles(){ return new File[][] { { new File("file1") }, { new File("file2") } }; // or scan a directory } @Test(dataProvider="files") public void testFile(File file){ //run tests on file } } 

Example output

PASSED: testFile(file1) PASSED: testFile(file2) =============================================== Default test Tests run: 2, Failures: 0, Skips: 0 =============================================== 

I had a similar problem and ended up writing a simple JUnit 4 runner that allows med to dynamically generate tests.

https://github.com/kimble/junit-test-factory