22

I am writing a parser and as a part of that, I have an Expander class that "expands" single complex statement into multiple simple statements. For example, it would expand this:

x = 2 + 3 * a

into:

tmp1 = 3 * a x = 2 + tmp1

Now I'm thinking about how to test this class, specifically how to Arrange the tests. I could manually create the input syntax tree:

var input = new AssignStatement( new Variable("x"), new BinaryExpression( new Constant(2), BinaryOperator.Plus, new BinaryExpression(new Constant(3), BinaryOperator.Multiply, new Variable("a"))));

Or I could write it as a string and parse it:

var input = new Parser().ParseStatement("x = 2 + 3 * a");

The second option is much simpler, shorter and readable. But it also introduces a dependency on Parser, which means that a bug in Parser could fail this test. So, the test would stop being a unit test of Expander, and I guess technically becomes an integration test of Parser and Expander.

My question is: is it okay to rely mostly (or completely) on this kind of integration test to test this Expander class?

Improve this question
1
  • 3
    That a bug in Parser could fail some other test is not a problem if you habitually commit only at zero failures, on the contrary it means that you have more coverage of Parser. What I would rather worry about is that a bug in Parser could make this test succeed when it should have failed. Unit tests are there to find bugs, after all—a test is broken when it doesn't but should have. Commented Dec 4, 2014 at 19:19

4 Answers 4

Reset to default
29

You're going to find yourself writing a lot more tests, of much more complicated, interesting, and useful behavior, if you can do so simply. So the option that involves

var input = new Parser().ParseStatement("x = 2 + 3 * a");

is quite valid. It does depend on another component. But everything depends on dozens of other components. If you mock something to within an inch of its life, you're probably depending on a lot of mocking features and test fixtures.

Developers sometimes over-focus on the purity of their unit tests, or developing unit tests and unit tests only, without any module, integration, stress or other kinds of tests. All those forms are valid and useful, and they're all the proper responsibility of developers--not just Q/A or operations personnel further down the pipeline.

One approach I've used is to start with these higher level runs, then use the data produced from them to construct the long-form, lowest-common-denominator expression of the test. E.g. when you dump the data structure from the input produced above, then you can easily construct the:

var input = new AssignStatement( new Variable("x"), new BinaryExpression( new Constant(2), BinaryOperator.Plus, new BinaryExpression(new Constant(3), BinaryOperator.Multiply, new Variable("a"))));

kind of test that tests at the very lowest level. That way you get a nice mix: A handful of the very most basic, primitive tests (pure unit tests), but have not spent a week writing tests at that primitive level. That gives you the time resource needed to write many more, slightly less atomic tests using the Parser as a helper. End result: More tests, more coverage, more corner and other interesting cases, better code and higher quality assurance.

Improve this answer
5
  • 2
    This is sensible - especially regarding the fact that everything depends on many others. A good unit test should test the minimum possible. Anything that is within that minimum possible amount should be tested by a preceeding unit test. If you've completely tested Parser, you can assume that you can safely use Parser to test ParseStatement Commented Dec 4, 2014 at 16:25
  • 6
    The main purity concern (I think) is to avoid writing circular dependencies in your unit tests. If either the parser or the parser tests use the expander, and this expander test relies on the parser working, then you have a difficult-to-manage risk that all you're testing is that the parser and the expander are consistent, whereas what you wanted to do was test that the expander actually does what it's supposed to. But as long as there's no dependency back the other way, using parser in this unit test isn't really any different from using a standard library in a unit test. Commented Dec 4, 2014 at 18:15
  • @SteveJessop Good point. It's important to use independent components. Commented Dec 4, 2014 at 18:33
  • 3
    Something I've done in cases where the parser itself is an expensive operation (eg reading data out of Excel files via com interop) is to write test generation methods that run the parser and output code to the console recreate the data structure the parser return. I then copy the output from the generator into more conventional unit tests. This allows reducing the cross dependency in that the parser only needs to be working correctly when the tests were created not every time they're run. (Not wasting a few seconds/test to create/destroy Excel processes was a nice bonus.) Commented Dec 4, 2014 at 22:18
  • +1 for @DanNeely's approach. We use something similar to store several serialized versions of our data model as test data, so that we can be sure new code can still work with older data. Commented Dec 5, 2014 at 3:10
7

Of course it is OK!

You always need functional/integration test that exercise the complete code path. And complete code path in this case means including evaluation of the generated code. That is you test that parsing x = 2 + 3 * a produces code that if run with a = 5 will set x to 17 and if run with a = -2 will set x to -4.

Below this, you should do unit tests for smaller bits as long as it actually helps debug the code. The finer grained tests you'll have, the higher probability that any change to the code will need to change the test too, because the internal interface changes. Such test have little long-term value and add maintenance work. So there is a point of diminishing returns and you should stop before it.

Improve this answer
4

Unit tests allow you to pin point specific items that break and where in the code they broke. So they're good for very fine grained testing. Good unit tests will help decrease debugging time.

However, from my experience unit tests are rarely good enough to actually verify correct operation. So integration tests are also helpful to verify a chain or sequence of operations. Integration tests get you part of the way through functional testing. As you pointed out though, because of the complexity of integration tests, it is harder to find the specific spot in the code where the test breaks. It also has somewhat more brittleness in that failures anywhere in the chain will cause the test to fail. You will still have that chain in the production code however, so testing the actual chain is still helpful.

Ideally you'd have both, but at any rate, generally having an automated test is better than having no test.

Improve this answer
0

Do a lot of tests on the parser and as the parser passes the tests, save those outputs to a file to mock the parser and test the other component.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.