Advice on unit testing

I came across a test question recently that asked what % of code should be covered by unit tests, the correct answer was 100%. I also came across an article called 'Evil Unit Testing' that lists what a unit test should 'not' do:

A unit test NEVER uses:
*a database
*an app server (or server of any kind)
*file/network I/O or file system;
*another application;
*the console (System.out, System.err, etc.)
*logging
*most other classes (exceptions include DTO's, *String, Integer, mocks and maybe a few others).

The problem is I have classes in my project that do things like:
*copy files
*use other classes (eg/I have a helper class that is used by multiple other methods in other classes).
*use other applications (it's an OSGI project that uses the 'pax' application)
and so on.....

My conundrum is this:

The article and original statement seem contradictory. How can I have 100% coverage, if I am NEVER supposed to use the resources in the list (files etc..)? For example, how can I test a file copier class, if I am never supposed to use files in a unit test?

I wouldn't agree that a unit test should not ever do all those things, partly for the reasons you mention: A unit test for a method that copies a file from A to B will have a unit test that uses files. To some degree it's a matter of how you want to name your tests, and you'll note that some of the other tests in that article do indeed use those resources. And they're often run via the unit test framework as well, so the boundaries are blurred.

What a unit test should indeed not do is anything time-consuming, because you want to be able to run unit tests frequently.

Billy Sclater wrote:
The article and original statement seem contradictory. How can I have 100% coverage, if I am NEVER supposed to use the resources in the list (files etc..)? For example, how can I test a file copier class, if I am never supposed to use files in a unit test?

There are solutions for everything. An OutputStream can be mocked, a database can be simulated (DbUnit)

A Unit test will never use real other classes. All dependencies are mocked. If a unit test uses other classes, it's not a unit test, it's an integration test. You should do automated integration and functional tests, they are just not called unit tests.

What a unit test should never do, is be aimed at providing 100% coverage. You should unit test functionality, not getters and setters. If you use a tool like Cobertura, it should give you 50/70 % coverage, but it can be more or less, depending on the application. Coverage is a good way to see if your project is tested, mostly because of the difference between 0-10% coverage and +- 50% coverage. But you shouldn't consider 100% as some holy grail of testing, because too high coverage generally means your tests are not testing the right things.

Dieter just made a point that a unit test that utilises other classes is in fact not a unit test, it is an integration test. My 'file-copier method' uses a 'pause method' from a 'helper class'. To test it, I am using the junit library.

Is this appropriate? As by definition I now have an integration test - more than one class in the test, not a unit test.

JUnit doesn't care whether your tests are strictly unit tests or not so your tests will run. But if your tests are forced to use concrete instances of classes (other than the class being tested) then you will regularly find that it is very difficult to write tests because you have no control over the objects your class is interacting with. If you can't control the environment your class is running in then you can't test it easily, especially in exceptional cases such as when there is an error.

The solution to this is not to view tests as something you write at the end of development. Tests need to be at the forefront of your mind as you are designing your application, and this will result in a design that is easy to test. The Test Driven Development methodology even goes as far as writing tests [i]before[/] the implementation code.

There are a lot of things that are important when designing testable code, but a few of the most important things (in my opinion) are:

* Write Interfaces for your domain objects. This makes it easier to mock them.
* Use dependency injection. If your class creates concrete instances of objects itself then you cannot mock those objects and have no control over them. On top of this, program to Interfaces, not implementations.
* Use a good mocking framework in your tests. I use Mockito, but there are others available.
* Keep your classes focussed. If a class has a lot of incoherent jobs then it should probably be split into several classes. That will make testing easier as each class will have fewer dependencies.

Mike makes a bunch of really valid points here. Although I have to disagree with the following statement from Dieter:

Dieter Quickfend wrote:A Unit test will never use real other classes. All dependencies are mocked. If a unit test uses other classes, it's not a unit test, it's an integration test

That suggests that the hard definition of a "Unit" in a Unit Test is a single class.

I think the definition of a Unit is a bit more flexible than that. I see it as a logical unit within your application which may consist of a single class or multiple classes. For example, the product I develop at work has to publish messages on a Tibco bus. So I have a "unit" that takes a message in the format of our domain model and translates that into a Tibco format to be sent down the wire. My Unit Tests here will be to ensure that if I put in domain object A then I get out Tibco message B. As it happens there are a handful of classes and things in the middle that actually make that happen but I do not necessarily have any unit tests for those individually. My "unit" in this case is the code that translates the message from format A to format B. I don't really care at this point how many classes are involved.

The purpose of having unit tests in your code is to give you the ability to refactor safely. In my example above, I am now free to completely change how I implement building those messages but as long as my tests pass then I know I'm building those messages correctly. If I had unit tests around each individual class in there with Mocks for every interaction with other classes then I would be encasing my code in concrete. I'm now writing tests to specify the implementation. I don't want that. I want tests to specify the functionality.

The line between what we call a Unit Test and an Integration Test is often difficult to identify. Let's take the example you give where you're writing code that reads and writes to the file system:

When writing a unit test for this I don't really want it to actually touch the filesystem because it's slow and there might be permission problems on the current platform which then leads to flakey tests which fail sometimes and not others. So in this case you can do as Dieter suggests and work with an OutputStream (mockable interface) rather than File (un-mockable final class). Or I think there is some library available that gives you an in-memory representation of a file system which allows you to actually read and write quickly in your tests. (I can't remember what it's called, sorry)

You're also going to need an Integration Test to verify that your code does actually write and read the file to and from the real filesystem. For this you're going to need to make sure the environment is set up correctly with directories created with correct user permissions granted on them.

Unit tests need to be quick as those are the ones you run repeatedly all day long on your local development environment. Integration Tests are run less frequently, once a day perhaps, and are run in an environment that is as close to production as is feasibly possible.

I agree that 100% test coverage is not a good goal. You first priority is writing good useful tests. For a large application, I got between 95% and 99% coverage. Some lines are just untestable. Suppose you have a helper class with a private constructor. You *could* use reflection to test it. But what is the value in doing so.

Categorizing the things in your list:

I think an integration tests should to these (they are still in JUnit, but a different type of test)
*a database
*an app server (or server of any kind)
*network I/O
*another application;

A unit test
*the console (System.out, System.err, etc.) - if you want to test the output, mock it. if there just happens to be output, ignore it
*logging
*most other classes (exceptions include DTO's, *String, Integer, mocks and maybe a few others) - calling small classes is fine. Calling classes with significant logic is a sign that you should mock instead of calling directly.

Gray area (I have written "unit tests" to check XML files, provide "golden master" copies of data, test configuration, test for usability WCAG/Section 508, etc. As long as they are fast, I see no problem with this.).
*File I/O

Tim Cooke wrote:
I think the definition of a Unit is a bit more flexible than that. I see it as a logical unit within your application which may consist of a single class or multiple classes. For example, the product I develop at work has to publish messages on a Tibco bus. So I have a "unit" that takes a message in the format of our domain model and translates that into a Tibco format to be sent down the wire. My Unit Tests here will be to ensure that if I put in domain object A then I get out Tibco message B. As it happens there are a handful of classes and things in the middle that actually make that happen but I do not necessarily have any unit tests for those individually. My "unit" in this case is the code that translates the message from format A to format B. I don't really care at this point how many classes are involved.

The purpose of having unit tests in your code is to give you the ability to refactor safely. In my example above, I am now free to completely change how I implement building those messages but as long as my tests pass then I know I'm building those messages correctly. If I had unit tests around each individual class in there with Mocks for every interaction with other classes then I would be encasing my code in concrete. I'm now writing tests to specify the implementation. I don't want that. I want tests to specify the functionality.

I believe you are thinking of an integration test. Although I do believe integration tests are the most bang-for-buck tests I usually write, it is also worth writing unit tests to see if every step in your current implementation functions as expected. It is quite normal that, in case of a change of implementation, you would have to change or rewrite your unit tests for that change. After all, if your implementation changes, all your assumptions become invalid.

This is all a matter of interpretation, I guess, but this is how I see it:
- Unit tests are tests that test a logical unit (an exposed (non-private) operation inside a class), isolated from all other exposed logic. All dependencies are mocked out.
- Integration tests are tests that test a logical unit which depends on functionality from other units, along with its dependent units (exposed in another class and thus reusable by other units). These dependencies should also have unit tests.

Think about it. A low-level method returning one of three strings for a certain parameter is changed and will now return only two. In a business method 5 levels of abstraction above this method, you will never test all of these possibilities. The only place this is done is the unit test for the low-level method. If you don't have a unit test for the low-level method, the cases in which the third string must be returned for the business method to behave correctly will no longer function.

Dieter Quickfend wrote:I believe you are thinking of an integration test.

Not in this example no. The "Unit" in my test is a collection of classes that work together to perform a single function. It would become an Integration Test if I were to actually send that message over the wire using Tibco to some other machine or process. But in this case I am not, so I'd still call it a Unit Test.

I think of "integration" as an interaction with some system outside of the application. Things like a database, a web service, an external message system, pretty much anything that requires a separate system to be up in order to operate. I do not necessarily consider the interaction between two classes an integration point unless it represents a communication between two logically different parts of the application, although in that case I might term it more of an "end to end" test.

Call it whatever you like really. If it works for you then go with it.

I certainly agree that a unit can consist of more than one class. I have cases in my application whereby one class creates instances of another class for its own private use. I don't treat this second type as a dependency when testing the first and have no way in my unit tests of controlling the creation of instances of this second type. In fact I completely changed the API of one of the classes today and the unit tests still passed.

The important thing from a Unit Test point of view is that the created type must not have dependencies of its own other than things that are passed in through the public API of the creating class. Any use of static global variables or methods (with Singletons often being an example) can lead to code that is difficult or impossible to test.