2

I'm writing a set of unit tests for a multi-threaded system. I want to write a test the confirms that thread B was blocked by thread A. My current solution, which I know isn't correct, is for thread A to sleep for a while, and then for the main thread to determine that thread B took at least that long to complete. I realize that this test isn't really valid because thread B could have simply been scheduled out by the system and took a long time to run that had nothing to do with being blocked.

More information, based on the comments:

I can't go into the details, but $dayjob has implemented its own task threading system not too different from e.g. Android's AsyncTask and I've been tasked (get it?) with writing the unit tests for it. I've written a dozen or so unit tests which I'm happy with, and satisfied are completely deterministic. But just a few unit tests elude me. There are a few cases where a thread needs to wait for a specific condition, and I need to confirm that it was actually blocked on the condition and didn't just happen to sleep at the wrong time.

The test I came up with won't generate any false failures, but could still generate false successes if the scheduler happens to put a thread to sleep at the wrong time.

Still more information: I'm not testing code that uses the threading library, I'm testing the library itself.

Improve this question
8
  • "Blocked" as in "waiting for a lock" or what? Commented Mar 24, 2021 at 23:49
  • Empirically, never found such a test requirement necessary or useful - it's usually pretty obvious if a thread gets stuck or looping. TBH, on multithreaded and/or distributed systems, I have never found unit tests, as a whole, to be necessary or useful:) My fear would be that any such instrumentation would introduce more problems than it could ever solve, but obviously I have not experienced every test environment. I don't envy you, and good luck...:) Commented Mar 25, 2021 at 0:57
  • In those cases where some global, or whatever, data requires protected access, most OS provide lock waits with timeouts, and so it's easy for a thread to detect an insane lock wait and moan about it, (eg. with a suitable log message). Such code can usually be left in system test, and even deliverable, builds without performance penalties, so that QA cannot moan that the build delivered was not the build tested:) Commented Mar 25, 2021 at 1:14
  • 'then for the main thread to determine'....yeah, I fear that your design is approaching an event-horizon without enough engine power to escape. Phrases like 'to complete' worry me, I hope that you have managed to avoid the super-massive black hole of 'create/terminate/join' threading designs, as encouraged by the umm....'less than optimal' pthreads etc libs:) Commented Mar 25, 2021 at 1:24
  • "Blocked" in the general sense. One test launches two tasks on the same thread queue, and I want to confirm that the second didn't run until the first finished. Another tests confirms that a task waits on a mutex. Yet another that a thread waited on a condition variable. Commented Mar 25, 2021 at 1:49

2 Answers 2

Reset to default
2

Testing of multithreaded code is a very interesting area, IMO. I'd say that you concentrate on too low level things - you are trying to detect that "thread B was blocked by thread A". You may have a ton of other problems (race conditions, data races) like a shared variable with no appropriate memory barriers used to store/load. That's why I'd prefer to concentrate on your own program's functionality/state in 2 following aspects:

  1. no unexpected state reached during the test (there are no related bugs including race conditions or data races)
  2. the test makes progress as expected (no unexpected long waits/blocks/deadlocks because of any reason, including race conditions or data races as well)

For example, in the case of Single Producer/Single Consumer Queue test, we write a test with 2 threads. One thread puts N (preferable - millions) values (numbers, for example, from 1 to N). Another thread takes N values from the queue.

To make sure we don't have race conditions or data races, the consumer might check that each value equals to the previous value + 1. This is how we check our expected state.

To make sure we make progress, the producer might increment a PutCounter after each put(). The consumer - a TakeCounter after each take(). After the test is finished, we check our final progress invariant PutCounter == TakeCounter == N. Since we may have a bug which affects the progress, we should set a reasonable Timeout on the test.

Since such tests require a significant amount of time to be run, I have disabled them from build on developer's side and only in CI enabled.

As for your particular case, we might test that the test doesn't make progress during a timeout and only for investigating/bug-fixing we make a thread dump and see that this happened because "thread B was blocked by thread A". Think about what each thread should do? What does "progress" mean for each thread or for the whole system? Then prepare a test scenario and validate with a timeout that the final progress was reached.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

...another test confirms that a task waits on a mutex.

Don't waste time testing the library. Focus on testing your own code. If you need to prove that your code called a certain method (e.g., lock) of a certain object (e.g., a given std::mutex) under certain conditions, then the way to do that is not by verifying that the calling thread actually was blocked. The way to test it is by injecting a test double (a.k.a., "mock object") for the mutex.

You create an object, the test double or "mock", that conforms to the mutex API, but which isn't actually a mutex. Your test function then sets up the specific conditions under which the unit-under-test is expected to lock the mutex, it calls your unit, and then it queries the mock mutex object to ask, "was your lock() method called?"

Setting up the specific conditions may entail passing in other "test double" objects. And likewise, depending on the complexity of the unit that you're testing, you may have to pass in other test double objects to prevent it from crashing the test harness after it has called the lock() function.

Most unit-testing frameworks for OO languages include some means for easily creating test doubles.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.