It sounds like you're already doing due diligence. But ...

At the most practical level, always include a good handful of both "full-loop" integration tests in your suite for your own code, and write more assertions than you think you need. In particular, you should have a handful of tests that perform a full create-read-[do_stuff]-validate cycle.

[TestMethod] public void MyFormatter_FormatsTimesCorrectly() { // this test isn't necessarily about the stream or the external interpreter. // but ... we depend on them working how we think they work: var stream = new StreamThingy(); var interpreter = new InterpreterThingy(stream); stream.Write("id-123, some description, 12345"); var formatter = new MyFormatter(interpreter); var line = formatter.getLine(); Assert.equal( "some description took 123.45 seconds to complete (id-123)", line ); } 

And it sounds like you're already doing this sort of thing. You're just dealing with a flaky and/or complicated library. And in that case, it's good to throw in a few "this is how the library works" types of tests that both verify your understanding of the library and serve as examples of how to use the library.

Suppose you need to understand and depend on how a JSON parser interprets each "type" in a JSON string. It's helpful and trivial to include something like this in your suite:

[TestMethod] public void JSONParser_InterpretsTypesAsExpected() { String datastream = "{nbr:11,str:"22",nll:null,udf:undefined}"; var o = (new JSONParser()).parse(datastream); Assert.equal(11, o.nbr); Assert.equal(Int32.getType(), o.nbr.getType()); Assert.equal("22", o.str); Assert.equal(null, o.nll); Assert.equal(Object.getType(), o.nll.getType()); Assert.isFalse(o.KeyExists(udf)); } 

But secondly, remember that automated testing of any kind, and at almost any level of rigor, will still fail to protect you against all bugs. It's perfectly common to add tests as you discover problems. Not having a QA department, this means a lot of those problems will be discovered by end-users.

And to a significant degree, that's just normal.

And thirdly, when a library changes the meaning of a return-value or field without renaming the field or method or otherwise "breaking" dependent code (maybe by changing its type), I'd be pretty damn unhappy with that publisher. And I'd argue that, even though you should probably have read the changelog if there is one, you should probably also pass some of your stress onto the publisher. I'd argue they need the hopefully-constructive criticism ...

It sounds like you're already doing due diligence. But ...

At the most practical level, always include a good handful of both "full-loop" integration tests in your suite for your own code, and write more assertions than you think you need. In particular, you should have a handful of tests that perform a full create-read-[do_stuff]-validate cycle.

[TestMethod] public void MyFormatter_FormatsTimesCorrectly() { // this test isn't necessarily about the stream or the external interpreter. // but ... we depend on them working how we think they work: var stream = new StreamThingy(); var interpreter = new InterpreterThingy(stream); stream.Write("id-123, some description, 12345"); // this is what you're actually testing. but, it'll also hiccup // if your 3rd party dependencies introduce a breaking change. var formatter = new MyFormatter(interpreter); var line = formatter.getLine(); Assert.equal( "some description took 123.45 seconds to complete (id-123)", line ); } 

And it sounds like you're already doing this sort of thing. You're just dealing with a flaky and/or complicated library. And in that case, it's good to throw in a few "this is how the library works" types of tests that both verify your understanding of the library and serve as examples of how to use the library.

Suppose you need to understand and depend on how a JSON parser interprets each "type" in a JSON string. It's helpful and trivial to include something like this in your suite:

[TestMethod] public void JSONParser_InterpretsTypesAsExpected() { String datastream = "{nbr:11,str:"22",nll:null,udf:undefined}"; var o = (new JSONParser()).parse(datastream); Assert.equal(11, o.nbr); Assert.equal(Int32.getType(), o.nbr.getType()); Assert.equal("22", o.str); Assert.equal(null, o.nll); Assert.equal(Object.getType(), o.nll.getType()); Assert.isFalse(o.KeyExists(udf)); } 

But secondly, remember that automated testing of any kind, and at almost any level of rigor, will still fail to protect you against all bugs. It's perfectly common to add tests as you discover problems. Not having a QA department, this means a lot of those problems will be discovered by end-users.

And to a significant degree, that's just normal.

And thirdly, when a library changes the meaning of a return-value or field without renaming the field or method or otherwise "breaking" dependent code (maybe by changing its type), I'd be pretty damn unhappy with that publisher. And I'd argue that, even though you should probably have read the changelog if there is one, you should probably also pass some of your stress onto the publisher. I'd argue they need the hopefully-constructive criticism ...