I have thus far avoided the nightmare that is testing multi-threaded code since it just seems like too much of a minefield. I'd like to ask how people have gone about testing code that relies on threads for successful execution, or just how people have gone about testing those kinds of issues that only show up when two threads interact in a given manner?
This seems like a really key problem for programmers today, it would be useful to pool our knowledge on this one imho.
Look, there's no easy way to do this. I'm working on a project that is inherently multithreaded. Events come in from the operating system and I have to process them concurrently.
The simplest way to deal with testing complex, multithreaded application code is this: If it's too complex to test, you're doing it wrong. If you have a single instance that has multiple threads acting upon it, and you can't test situations where these threads step all over each other, then your design needs to be redone. It's both as simple and as complex as this.
There are many ways to program for multithreading that avoids threads running through instances at the same time. The simplest is to make all your objects immutable. Of course, that's not usually possible. So you have to identify those places in your design where threads interact with the same instance and reduce the number of those places. By doing this, you isolate a few classes where multithreading actually occurs, reducing the overall complexity of testing your system.
But you have to realize that even by doing this, you still can't test every situation where two threads step on each other. To do that, you'd have to run two threads concurrently in the same test, then control exactly what lines they are executing at any given moment. The best you can do is simulate this situation. But this might require you to code specifically for testing, and that's at best a half step towards a true solution.
Probably the best way to test code for threading issues is through static analysis of the code. If your threaded code doesn't follow a finite set of thread safe patterns, then you might have a problem. I believe Code Analysis in VS does contain some knowledge of threading, but probably not much.
Look, as things stand currently (and probably will stand for a good time to come), the best way to test multithreaded apps is to reduce the complexity of threaded code as much as possible. Minimize areas where threads interact, test as best as possible, and use code analysis to identify danger areas.
It's been a while when this question was posted, but it's still not answered ...
kleolb02's answer is a good one. I'll try going into more details.
There is a way, which I practice for C# code. For unit tests you should be able to program reproducible tests, which is the biggest challenge in multithreaded code. So my answer aims toward forcing asynchronous code into a test harness, which works synchronously.
It's an idea from Gerard Meszaros's book "xUnit Test Patterns" and is called "Humble Object" (p. 695): You have to separate core logic code and anything which smells like asynchronous code from each other. This would result to a class for the core logic, which works synchronously.
This puts you into the position to test the core logic code in a synchronous way. You have absolute control over the timing of the calls you are doing on the core logic and thus can make reproducible tests. And this is your gain from separating core logic and asynchronous logic.
This core logic needs be wrapped around by another class, which is responsible for receiving calls to the core logic asynchronously and delegates these calls to the core logic. Production code will only access the core logic via that class. Because this class should only delegate calls, it's a very "dumb" class without much logic. So you can keep your unit tests for this asychronous working class at a minimum.
Anything above that (testing interaction between classes) are component tests. Also in this case, you should be able to have absolute control over timing, if you stick to the "Humble Object" pattern.
Tough one indeed! In my (C++) unit tests, I've broken this down into several categories along the lines of the concurrency pattern used:
Unit tests for classes that operate in a single thread and aren't thread aware -- easy, test as usual.
Unit tests for Monitor objects (those that execute synchronized methods in the callers' thread of control) that expose a synchronized public API -- instantiate multiple mock threads that exercise the API. Construct scenarios that exercise internal conditions of the passive object. Include one longer running test that basically beats the heck out of it from multiple threads for a long period of time. This is unscientific I know but it does build confidence.
Unit tests for Active objects (those that encapsulate their own thread or threads of control) -- similar to #2 above with variations depending on the class design. Public API may be blocking or non-blocking, callers may obtain futures, data may arrive at queues or need to be dequeued. There are many combinations possible here; white box away. Still requires multiple mock threads to make calls to the object under test.
As an aside:
In internal developer training that I do, I teach the Pillars of Concurrency and these two patterns as the primary framework for thinking about and decomposing concurrency problems. There's obviously more advanced concepts out there but I've found that this set of basics helps keep engineers out of the soup. It also leads to code that is more unit testable, as described above.
I have faced this issue several times in recent years when writing thread handling code for several projects. I'm providing a late answer because most of the other answers, while providing alternatives, do not actually answer the question about testing. My answer is addressed to the cases where there is no alternative to multithreaded code; I do cover code design issues for completeness, but also discuss unit testing.
Writing testable multithreaded code
The first thing to do is to separate your production thread handling code from all the code that does actual data processing. That way, the data processing can be tested as singly threaded code, and the only thing the multithreaded code does is to coordinate threads.
The second thing to remember is that bugs in multithreaded code are probabilistic; the bugs that manifest themselves least frequently are the bugs that will sneak through into production, will be difficult to reproduce even in production, and will thus cause the biggest problems. For this reason, the standard coding approach of writing the code quickly and then debugging it until it works is a bad idea for multithreaded code; it will result in code where the easy bugs are fixed and the dangerous bugs are still there.
Instead, when writing multithreaded code, you must write the code with the attitude that you are going to avoid writing the bugs in the first place. If you have properly removed the data processing code, the thread handling code should be small enough - preferably a few lines, at worst a few dozen lines - that you have a chance of writing it without writing a bug, and certainly without writing many bugs, if you understand threading, take your time, and are careful.
Writing unit tests for multithreaded code
Once the multithreaded code is written as carefully as possible, it is still worthwhile writing tests for that code. The primary purpose of the tests is not so much to test for highly timing dependent race condition bugs - it's impossible to test for such race conditions repeatably - but rather to test that your locking strategy for preventing such bugs allows for multiple threads to interact as intended.
To properly test correct locking behavior, a test must start multiple threads. To make the test repeatable, we want the interactions between the threads to happen in a predictable order. We don't want to externally synchronize the threads in the test, because that will mask bugs that could happen in production where the threads are not externally synchronized. That leaves the use of timing delays for thread synchronization, which is the technique that I have used successfully whenever I've had to write tests of multithreaded code.
If the delays are too short, then the test becomes fragile, because minor timing differences - say between different machines on which the tests may be run - may cause the timing to be off and the test to fail. What I've typically done is start with delays that cause test failures, increase the delays so that the test passes reliably on my development machine, and then double the delays beyond that so the test has a good chance of passing on other machines. This does mean that the test will take a macroscopic amount of time, though in my experience, careful test design can limit that time to no more than a dozen seconds. Since you shouldn't have very many places requiring thread coordination code in your application, that should be acceptable for your test suite.
Finally, keep track of the number of bugs caught by your test. If your test has 80% code coverage, it can be expected to catch about 80% of your bugs. If your test is well designed but finds no bugs, there's a reasonable chance that you don't have additional bugs that will only show up in production. If the test catches one or two bugs, you might still get lucky. Beyond that, and you may want to consider a careful review of or even a complete rewrite of your thread handling code, since it is likely that code still contains hidden bugs that will be very difficult to find until the code is in production, and very difficult to fix then.
I also had serious problems testing multi- threaded code. Then I found a really cool solution in "xUnit Test Patterns" by Gerard Meszaros. The pattern he describes is called Humble object.
Basically it describes how you can extract the logic into a separate, easy-to-test component that is decoupled from its environment. After you tested this logic, you can test the complicated behaviour (multi- threading, asynchronous execution, etc...)
There are a few tools around that are quite good. Here is a summary of some of the Java ones.
Some good static analysis tools include FindBugs (gives some useful hints), JLint, Java Pathfinder (JPF & JPF2), and Bogor.
MultithreadedTC is quite a good dynamic analysis tool (integrated into JUnit) where you have to set up your own test cases.
ConTest from IBM Research is interesting. It instruments your code by inserting all kinds of thread modifying behaviours (e.g. sleep & yield) to try to uncover bugs randomly.
SPIN is a really cool tool for modelling your Java (and other) components, but you need to have some useful framework. It is hard to use as is, but extremely powerful if you know how to use it. Quite a few tools use SPIN underneath the hood.
MultithreadedTC is probably the most mainstream, but some of the static analysis tools listed above are definitely worth looking at.
Awaitility can also be useful to help you write deterministic unit tests. It allows you to wait until some state somewhere in your system is updated. For example:
await().untilCall( to(myService).myMethod(), greaterThan(3) ); or
await().atMost(5,SECONDS).until(fieldIn(myObject).ofType(int.class), equalTo(1)); It also has Scala and Groovy support.
await until { something() > 4 } // Scala example Another way to (kinda) test threaded code, and very complex systems in general is through Fuzz Testing. It's not great, and it won't find everything, but its likely to be useful and its simple to do.
Quote:
Fuzz testing or fuzzing is a software testing technique that provides random data("fuzz") to the inputs of a program. If the program fails (for example, by crashing, or by failing built-in code assertions), the defects can be noted. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior.
...
Fuzz testing is often used in large software development projects that employ black box testing. These projects usually have a budget to develop test tools, and fuzz testing is one of the techniques which offers a high benefit to cost ratio.
...
However, fuzz testing is not a substitute for exhaustive testing or formal methods: it can only provide a random sample of the system's behavior, and in many cases passing a fuzz test may only demonstrate that a piece of software handles exceptions without crashing, rather than behaving correctly. Thus, fuzz testing can only be regarded as a bug-finding tool rather than an assurance of quality.
Testing MT code for correctness is, as already stated, quite a hard problem. In the end it boils down to ensuring that there are no incorrectly synchronised data races in your code. The problem with this is that there are infinitely many possibilities of thread execution (interleavings) over which you do not have much control (be sure to read this article, though). In simple scenarios it might be possible to actually prove correctness by reasoning but this is usually not the case. Especially if you want to avoid/minimize synchronization and not go for the most obvious/easiest synchronization option.
An approach that I follow is to write highly concurrent test code in order to make potentially undetected data races likely to occur. And then I run those tests for some time :) I once stumbled upon a talk where some computer scientist where showing off a tool that kind of does this (randomly devising test from specs and then running them wildly, concurrently, checking for the defined invariants to be broken).
By the way, I think this aspect of testing MT code has not been mentioned here: identify invariants of the code that you can check for randomly. Unfortunately, finding those invariants is quite a hard problem, too. Also they might not hold all the time during execution, so you have to find/enforce executions points where you can expect them to be true. Bringing the code execution to such a state is also a hard problem (and might itself incur concurrency issues. Whew, it's damn hard!
Some interesting links to read:
I've done a lot of this, and yes it sucks.
Some tips:
throwable field and check it in tearDown (see Listing 1). If you catch a bad exception in another thread, just assign it to throwable.AtomicBoolean in your tests. It is thread safe, and you'll often need a final reference type to store values from callback classes and suchlike. See example in Listing 3.@Test(timeout=60*1000)), as concurrency tests can sometimes hang forever when they're broken.Listing 1:
@After public void tearDown() { if ( throwable != null ) throw throwable; } Listing 2:
import static org.junit.Assert.fail; import java.io.File; import java.lang.reflect.InvocationHandler; import java.lang.reflect.Proxy; import java.util.Random; import org.apache.commons.collections.Closure; import org.apache.commons.collections.Predicate; import org.apache.commons.lang.time.StopWatch; import org.easymock.EasyMock; import org.easymock.classextension.internal.ClassExtensionHelper; import static org.easymock.classextension.EasyMock.*; import ca.digitalrapids.io.DRFileUtils; /** * Various utilities for testing */ public abstract class DRTestUtils { static private Random random = new Random(); /** Calls {@link #waitForCondition(Integer, Integer, Predicate, String)} with * default max wait and check period values. */ static public void waitForCondition(Predicate predicate, String errorMessage) throws Throwable { waitForCondition(null, null, predicate, errorMessage); } /** Blocks until a condition is true, throwing an {@link AssertionError} if * it does not become true during a given max time. * @param maxWait_ms max time to wait for true condition. Optional; defaults * to 30 * 1000 ms (30 seconds). * @param checkPeriod_ms period at which to try the condition. Optional; defaults * to 100 ms. * @param predicate the condition * @param errorMessage message use in the {@link AssertionError} * @throws Throwable on {@link AssertionError} or any other exception/error */ static public void waitForCondition(Integer maxWait_ms, Integer checkPeriod_ms, Predicate predicate, String errorMessage) throws Throwable { waitForCondition(maxWait_ms, checkPeriod_ms, predicate, new Closure() { public void execute(Object errorMessage) { fail((String)errorMessage); } }, errorMessage); } /** Blocks until a condition is true, running a closure if * it does not become true during a given max time. * @param maxWait_ms max time to wait for true condition. Optional; defaults * to 30 * 1000 ms (30 seconds). * @param checkPeriod_ms period at which to try the condition. Optional; defaults * to 100 ms. * @param predicate the condition * @param closure closure to run * @param argument argument for closure * @throws Throwable on {@link AssertionError} or any other exception/error */ static public void waitForCondition(Integer maxWait_ms, Integer checkPeriod_ms, Predicate predicate, Closure closure, Object argument) throws Throwable { if ( maxWait_ms == null ) maxWait_ms = 30 * 1000; if ( checkPeriod_ms == null ) checkPeriod_ms = 100; StopWatch stopWatch = new StopWatch(); stopWatch.start(); while ( !predicate.evaluate(null) ) { Thread.sleep(checkPeriod_ms); if ( stopWatch.getTime() > maxWait_ms ) { closure.execute(argument); } } } /** Calls {@link #waitForVerify(Integer, Object)} with <code>null</code> * for {@code maxWait_ms} */ static public void waitForVerify(Object easyMockProxy) throws Throwable { waitForVerify(null, easyMockProxy); } /** Repeatedly calls {@link EasyMock#verify(Object[])} until it succeeds, or a * max wait time has elapsed. * @param maxWait_ms Max wait time. <code>null</code> defaults to 30s. * @param easyMockProxy Proxy to call verify on * @throws Throwable */ static public void waitForVerify(Integer maxWait_ms, Object easyMockProxy) throws Throwable { if ( maxWait_ms == null ) maxWait_ms = 30 * 1000; StopWatch stopWatch = new StopWatch(); stopWatch.start(); for(;;) { try { verify(easyMockProxy); break; } catch (AssertionError e) { if ( stopWatch.getTime() > maxWait_ms ) throw e; Thread.sleep(100); } } } /** Returns a path to a directory in the temp dir with the name of the given * class. This is useful for temporary test files. * @param aClass test class for which to create dir * @return the path */ static public String getTestDirPathForTestClass(Object object) { String filename = object instanceof Class ? ((Class)object).getName() : object.getClass().getName(); return DRFileUtils.getTempDir() + File.separator + filename; } static public byte[] createRandomByteArray(int bytesLength) { byte[] sourceBytes = new byte[bytesLength]; random.nextBytes(sourceBytes); return sourceBytes; } /** Returns <code>true</code> if the given object is an EasyMock mock object */ static public boolean isEasyMockMock(Object object) { try { InvocationHandler invocationHandler = Proxy .getInvocationHandler(object); return invocationHandler.getClass().getName().contains("easymock"); } catch (IllegalArgumentException e) { return false; } } } Listing 3:
@Test public void testSomething() { final AtomicBoolean called = new AtomicBoolean(false); subject.setCallback(new SomeCallback() { public void callback(Object arg) { // check arg here called.set(true); } }); subject.run(); assertTrue(called.get()); } 
I handle unit tests of threaded components the same way I handle any unit test, that is, with inversion of control and isolation frameworks. I develop in the .Net-arena and, out of the box, the threading (among other things) is very hard (I'd say nearly impossible) to fully isolate.
Therefore, I've written wrappers that looks something like this (simplified):
public interface IThread { void Start(); ... } public class ThreadWrapper : IThread { private readonly Thread _thread; public ThreadWrapper(ThreadStart threadStart) { _thread = new Thread(threadStart); } public Start() { _thread.Start(); } } public interface IThreadingManager { IThread CreateThread(ThreadStart threadStart); } public class ThreadingManager : IThreadingManager { public IThread CreateThread(ThreadStart threadStart) { return new ThreadWrapper(threadStart) } } From there, I can easily inject the IThreadingManager into my components and use my isolation framework of choice to make the thread behave as I expect during the test.
That has so far worked great for me, and I use the same approach for the thread pool, things in System.Environment, Sleep etc. etc.
Pete Goodliffe has a series on the unit testing of threaded code.
It's hard. I take the easier way out and try to keep the threading code abstracted from the actual test. Pete does mention that the way I do it is wrong but I've either got the separation right or I've just been lucky.
For Java, check out chapter 12 of JCIP. There are some concrete examples of writing deterministic, multi-threaded unit tests to at least test the correctness and invariants of concurrent code.
"Proving" thread-safety with unit tests is much dicier. My belief is that this is better served by automated integration testing on a variety of platforms/configurations.
Have a look at my related answer at
Designing a Test class for a custom Barrier
It's biased towards Java but has a reasonable summary of the options.
In summary though (IMO) its not the use of some fancy framework that will ensure correctness but how you go about designing you multithreaded code. Splitting the concerns (concurrency and functionality) goes a huge way towards raising confidence. Growing Object Orientated Software Guided By Tests explains some options better than I can.
Static analysis and formal methods (see, Concurrency: State Models and Java Programs) is an option but I've found them to be of limited use in commercial development.
Don't forget that any load/soak style tests are rarely guaranteed to highlight problems.
Good luck!
tempus-fugit library here, which helps write and test concurrent code ;)I like to write two or more test methods to execute on parallel threads, and each of them make calls into the object under test. I've been using Sleep() calls to coordinate the order of the calls from the different threads, but that's not really reliable. It's also a lot slower because you have to sleep long enough that the timing usually works.
I found the Multithreaded TC Java library from the same group that wrote FindBugs. It lets you specify the order of events without using Sleep(), and it's reliable. I haven't tried it yet.
The biggest limitation to this approach is that it only lets you test the scenarios you suspect will cause trouble. As others have said, you really need to isolate your multithreaded code into a small number of simple classes to have any hope of thoroughly testing them.
Once you've carefully tested the scenarios you expect to cause trouble, an unscientific test that throws a bunch of simultaneous requests at the class for a while is a good way to look for unexpected trouble.
Update: I've played a bit with the Multithreaded TC Java library, and it works well. I've also ported some of its features to a .NET version I call TickingTest.
I just recently discovered (for Java) a tool called Threadsafe. It is a static analysis tool much like findbugs but specifically to spot multi-threading issues. It is not a replacement for testing but I can recommend it as part of writing reliable multi-threaded Java.
It even catches some very subtle potential issues around things like class subsumption, accessing unsafe objects through concurrent classes and spotting missing volatile modifiers when using the double checked locking paradigm.
If you write multithreaded Java give it a shot.
The following article suggests 2 solutions. Wrapping a semaphore (CountDownLatch) and adds functionality like externalize data from internal thread. Another way of achieving this purpose is to use Thread Pool (see Points of Interest).
I have had the unfortunate task of testing threaded code and they are definitely the hardest tests I have ever written.
When writing my tests, I used a combination of delegates and events. Basically it is all about using PropertyNotifyChanged events with a WaitCallback or some kind of ConditionalWaiter that polls.
I am not sure if this was the best approach, but it has worked out for me.
I spent most of last week at a university library studying debugging of concurrent code. The central problem is concurrent code is non-deterministic. Typically, academic debugging has fallen into one of three camps here:
Now, as above commentators have noticed, you can design your concurrent system into a more deterministic state. However, if you don't do that properly, you're just back to designing a sequential system again.
My suggestion would be to focus on having a very strict design protocol about what gets threaded and what doesn't get threaded. If you constrain your interface so that there is minimal dependancies between elements, it is much easier.
Good luck, and keep working on the problem.
Running multiple threads is not difficult; it is piece of cake. Unfortunately, threads usually need to communicate with each other; that's what's difficult.
The mechanism that was originally invented to allow communication between modules was function calls; when module A wants to communicate with module B, it just invokes a function in module B. Unfortunately, this does not work with threads, because when you call a function, that function still runs in the current thread.
To overcome this problem, people decided to fall back to an even more primitive mechanism of communication: just declare a certain variable, and let both threads have access to that variable. In other words, allow the threads to share data. Sharing data is literally the first thing that naturally comes to mind, and it appears like a good choice because it seems very simple. I mean, how hard can it be, right? What could possibly go wrong?
Race conditions. That's what can, and will, go wrong.
When people realized their software was suffering from random, non-reproducible catastrophic failures due to race conditions, they started inventing elaborate mechanisms such as locks and compare-and-swap, aiming to protect against such things happening. These mechanisms fall under the broad category of "synchronization". Unfortunately, synchronization has two problems:
The astute reader might notice that "Very prone to bugs" and "Completely untestable" is a deadly combination.
Now, the mechanisms I mentioned above were being invented and adopted by large parts of the industry before the concept of automated software testing became prevalent; So, nobody could see how deadly the problem was; they just regarded it as a difficult topic which requires guru programmers, and everyone was okay with that.
Nowadays, whatever we do, we put testing first. So, if some mechanism is untestable, then the use of that mechanism is just out of the question, period. Thus, synchronization has fallen out of grace; very few people still practice it, and they are becoming fewer and fewer every day.
Without synchronization threads cannot share data; however, the original requirement was not to share data; it was to allow threads to communicate with each other. Besides sharing data, there exist other, more elegant mechanisms for inter-thread communication.
One such mechanism is message-passing.
With message passing, there is only one place in the entire software system which utilizes synchronization: the implementation of the concurrent blocking queue collection class that we use for storing messages. (The idea is that we should be able to get at least that little part right.)
The great thing about message passing is that it does not suffer from race conditions and is fully testable.
Assuming under "multi-threaded" code was meant something that is
In other words we are talking about testing custom stateful thread-safe class/method/unit - which should be a very rare beast nowadays.
Because this beast is rare, first of all we need to make sure that there are all valid excuses to write it.
Step 1. Consider modifying state in same synchronization context.
Today it is easy to write compose-able concurrent and asynchronous code where IO or other slow operations offloaded to background but shared state is updated and queried in one synchronization context. e.g. async/await tasks and Rx in .NET etc. - they are all testable by design, "real" Tasks and schedulers can be substituted to make testing deterministic (however this is out of scope of the question).
It may sound very constrained but this approach works surprisingly well. It is possible to write whole apps in this style without need to make any state thread-safe (I do).
Step 2. If manipulating of shared state on single synchronization context is absolutely not possible.
Make sure the wheel is not being reinvented / there's definitely no standard alternative that can be adapted for the job. It should be likely that code is very cohesive and contained within one unit e.g. with a good chance it is a special case of some standard thread-safe data structure like hash map or collection or whatever.
Note: if code is large / spans across multiple classes AND needs multi-thread state manipulation then there's a very high chance that design is not good, reconsider Step 1
Step 3. If this step is reached then we need to test our own custom stateful thread-safe class/method/unit.
I'll be dead honest : I never had to write proper tests for such code. Most of the time I get away at Step 1, sometimes at Step 2. Last time I had to write custom thread-safe code was so many years ago that it was before I adopted unit testing / probably I wouldn't have to write it with the current knowledge anyway.
If I really had to test such code (finally, actual answer) then I would try couple of things below
Non-deterministic stress testing. e.g. run 100 threads simultaneously and check that end result is consistent. This is more typical for higher level / integration testing of multiple users scenarios but also can be used at the unit level.
Expose some test 'hooks' where test can inject some code to help make deterministic scenarios where one thread must perform operation before the other. As ugly as it is, I can't think of anything better.
Delay-driven testing to make threads run and perform operations in particular order. Strictly speaking such tests are non-deterministic too (there's a chance of system freeze / stop-the-world GC collection which can distort otherwise orchestrated delays), also it is ugly but allows to avoid hooks.
It seems there is a framework from Microsoft for testing in a multithreaded manner:
https://github.com/microsoft/coyote/
From the project's GitHub description (excerpt):
Coyote is a cross-platform library and tool for testing concurrent C# code and deterministically reproducing bugs.
Using Coyote, you can easily test the concurrency and other nondeterminism in your C# code, by writing what we call a concurrency unit test. These look like your regular unit tests, but can reliably test concurrent workloads (such as actors, tasks, or concurrent requests to ASP.NET controllers). In regular unit tests, you would typically avoid concurrency due to flakiness, but with Coyote you are encouraged to embrace concurrency in your tests to find bugs. Coyote is used by many teams in Azure to test their distributed systems and services, and has found hundreds of concurrency-related bugs before deploying code in production and affecting users. In the words of an Azure service architect: Coyote found several issues early in the dev process, this sort of issues that would usually bleed through into production and become very expensive to fix later. Coyote is made with ❤️ by Microsoft Research. How it works Consider the following simple test:
[Fact] public async Task TestTask() { int value = 0; Task task = Task.Run(() => { value = 1; }); Assert.Equal(0, value); await task; } This test will pass most of the time because the assertion will typically execute before the task starts, but there is one schedule where the task starts fast enough to set value to 1 causing the assertion to fail. Of course, this is a very naive example and the bug is obvious, but you could imagine much more complicated race conditions that are hidden in complex execution paths. The way Coyote works, is that you first convert the above test to a concurrency unit test using the Coyote TestingEngine API:
using Microsoft.Coyote.SystematicTesting; [Fact] public async Task CoyoteTestTask() { var configuration = Configuration.Create().WithTestingIterations(10); var engine = TestingEngine.Create(configuration, TestTask); engine.Run(); } Next, you run the coyote rewrite command from the CLI (typically as a post-build task) to automatically rewrite the IL of your test and production binaries. This allows Coyote to inject hooks that take control of the concurrent execution during testing. You can then run the concurrent unit test from your favorite unit testing framework (such as xUnit). Coyote will take over and repeatedly execute the test from beginning to the end for N iterations (in the above example N was configured to 10). Under the hood, Coyote uses intelligent search strategies to explore all kinds of execution paths that might hide a bug in each iteration. The awesome thing is that once a bug is found, Coyote gives you a trace through the engine.TestReport API that you can use to reliably reproduce the bug as many times as you want, making debugging and fixing the issue significantly easier.
I hope that helps.
For J2E code, I've used SilkPerformer, LoadRunner and JMeter for concurrency testing of threads. They all do the same thing. Basically, they give you a relatively simple interface for administrating their version of the proxy server, required, in order to analyze the TCP/IP data stream, and simulate multiple users making simultaneous requests to your app server. The proxy server can give you the ability to do things like analyze the requests made, by presenting the whole page and URL sent to the server, as well as the response from the server, after processing the request.
You can find some bugs in insecure http mode, where you can at least analyze the form data that is being sent, and systematically alter that for each user. But the true tests are when you run in https (Secured Socket Layers). Then, you also have to contend with systematically altering the session and cookie data, which can be a little more convoluted.
The best bug I ever found, while testing concurrency, was when I discovered that the developer had relied upon Java garbage collection to close the connection request that was established at login, to the LDAP server, when logging in. This resulted in users being exposed to other users' sessions and very confusing results, when trying to analyze what happened when the server was brought to it's knees, barely able to complete one transaction, every few seconds.
In the end, you or someone will probably have to buckle down and analyze the code for blunders like the one I just mentioned. And an open discussion across departments, like the one that occurred, when we unfolded the problem described above, are most useful. But these tools are the best solution to testing multi-threaded code. JMeter is open source. SilkPerformer and LoadRunner are proprietary. If you really want to know whether your app is thread safe, that's how the big boys do it. I've done this for very large companies professionally, so I'm not guessing. I'm speaking from personal experience.
A word of caution: it does take some time to understand these tools. It will not be a matter of simply installing the software and firing up the GUI, unless you've already had some exposure to multi-threaded programming. I've tried to identify the 3 critical categories of areas to understand (forms, session and cookie data), with the hope that at least starting with understanding these topics will help you focus on quick results, as opposed to having to read through the entire documentation.
Concurrency is a complex interplay between the memory model, hardware, caches and our code. In the case of Java at least such tests have been partly addressed mainly by jcstress. The creators of that library are known to be authors of many JVM, GC and Java concurrency features.
But even this library needs good knowledge of the Java Memory Model specification so that we know exactly what we are testing. But I think the focus of this effort is mircobenchmarks. Not huge business applications.
There is an article on the topic, using Rust as the language in the example code:
https://medium.com/@polyglot_factotum/rust-concurrency-five-easy-pieces-871f1c62906a
In summary, the trick is to write your concurrent logic so that it is robust to the non-determinism involved with multiple threads of execution, using tools like channels and condvars.
Then, if that is how you've structured your "components", the easiest way to test them is by using channels to send messages to them, and then block on other channels to assert that the component sends certain expected messages.
The linked-to article is fully written using unit-tests.
It's not perfect, but I wrote this helper for my tests in C#:
using System; using System.Collections.Generic; using System.Threading; using System.Threading.Tasks; namespace Proto.Promises.Tests.Threading { public class ThreadHelper { public static readonly int multiThreadCount = Environment.ProcessorCount * 100; private static readonly int[] offsets = new int[] { 0, 10, 100, 1000 }; private readonly Stack<Task> _executingTasks = new Stack<Task>(multiThreadCount); private readonly Barrier _barrier = new Barrier(1); private int _currentParticipants = 0; private readonly TimeSpan _timeout; public ThreadHelper() : this(TimeSpan.FromSeconds(10)) { } // 10 second timeout should be enough for most cases. public ThreadHelper(TimeSpan timeout) { _timeout = timeout; } /// <summary> /// Execute the action multiple times in parallel threads. /// </summary> public void ExecuteMultiActionParallel(Action action) { for (int i = 0; i < multiThreadCount; ++i) { AddParallelAction(action); } ExecutePendingParallelActions(); } /// <summary> /// Execute the action once in a separate thread. /// </summary> public void ExecuteSingleAction(Action action) { AddParallelAction(action); ExecutePendingParallelActions(); } /// <summary> /// Add an action to be run in parallel. /// </summary> public void AddParallelAction(Action action) { var taskSource = new TaskCompletionSource<bool>(); lock (_executingTasks) { ++_currentParticipants; _barrier.AddParticipant(); _executingTasks.Push(taskSource.Task); } new Thread(() => { try { _barrier.SignalAndWait(); // Try to make actions run in lock-step to increase likelihood of breaking race conditions. action.Invoke(); taskSource.SetResult(true); } catch (Exception e) { taskSource.SetException(e); } }).Start(); } /// <summary> /// Runs the pending actions in parallel, attempting to run them in lock-step. /// </summary> public void ExecutePendingParallelActions() { Task[] tasks; lock (_executingTasks) { _barrier.SignalAndWait(); _barrier.RemoveParticipants(_currentParticipants); _currentParticipants = 0; tasks = _executingTasks.ToArray(); _executingTasks.Clear(); } try { if (!Task.WaitAll(tasks, _timeout)) { throw new TimeoutException($"Action(s) timed out after {_timeout}, there may be a deadlock."); } } catch (AggregateException e) { // Only throw one exception instead of aggregate to try to avoid overloading the test error output. throw e.Flatten().InnerException; } } /// <summary> /// Run each action in parallel multiple times with differing offsets for each run. /// <para/>The number of runs is 4^actions.Length, so be careful if you don't want the test to run too long. /// </summary> /// <param name="expandToProcessorCount">If true, copies each action on additional threads up to the processor count. This can help test more without increasing the time it takes to complete. /// <para/>Example: 2 actions with 6 processors, runs each action 3 times in parallel.</param> /// <param name="setup">The action to run before each parallel run.</param> /// <param name="teardown">The action to run after each parallel run.</param> /// <param name="actions">The actions to run in parallel.</param> public void ExecuteParallelActionsWithOffsets(bool expandToProcessorCount, Action setup, Action teardown, params Action[] actions) { setup += () => { }; teardown += () => { }; int actionCount = actions.Length; int expandCount = expandToProcessorCount ? Math.Max(Environment.ProcessorCount / actionCount, 1) : 1; foreach (var combo in GenerateCombinations(offsets, actionCount)) { setup.Invoke(); for (int k = 0; k < expandCount; ++k) { for (int i = 0; i < actionCount; ++i) { int offset = combo[i]; Action action = actions[i]; AddParallelAction(() => { for (int j = offset; j > 0; --j) { } // Just spin in a loop for the offset. action.Invoke(); }); } } ExecutePendingParallelActions(); teardown.Invoke(); } } // Input: [1, 2, 3], 3 // Ouput: [ // [1, 1, 1], // [2, 1, 1], // [3, 1, 1], // [1, 2, 1], // [2, 2, 1], // [3, 2, 1], // [1, 3, 1], // [2, 3, 1], // [3, 3, 1], // [1, 1, 2], // [2, 1, 2], // [3, 1, 2], // [1, 2, 2], // [2, 2, 2], // [3, 2, 2], // [1, 3, 2], // [2, 3, 2], // [3, 3, 2], // [1, 1, 3], // [2, 1, 3], // [3, 1, 3], // [1, 2, 3], // [2, 2, 3], // [3, 2, 3], // [1, 3, 3], // [2, 3, 3], // [3, 3, 3] // ] private static IEnumerable<int[]> GenerateCombinations(int[] options, int count) { int[] indexTracker = new int[count]; int[] combo = new int[count]; for (int i = 0; i < count; ++i) { combo[i] = options[0]; } // Same algorithm as picking a combination lock. int rollovers = 0; while (rollovers < count) { yield return combo; // No need to duplicate the array since we're just reading it. for (int i = 0; i < count; ++i) { int index = ++indexTracker[i]; if (index == options.Length) { indexTracker[i] = 0; combo[i] = options[0]; if (i == rollovers) { ++rollovers; } } else { combo[i] = options[index]; break; } } } } } } Example usage:
[Test] public void DeferredMayBeBeResolvedAndPromiseAwaitedConcurrently_void0() { Promise.Deferred deferred = default(Promise.Deferred); Promise promise = default(Promise); int invokedCount = 0; var threadHelper = new ThreadHelper(); threadHelper.ExecuteParallelActionsWithOffsets(false, // Setup () => { invokedCount = 0; deferred = Promise.NewDeferred(); promise = deferred.Promise; }, // Teardown () => Assert.AreEqual(1, invokedCount), // Parallel Actions () => deferred.Resolve(), () => promise.Then(() => { Interlocked.Increment(ref invokedCount); }).Forget() ); } One simple test pattern that can work for some (not all!) cases is to repeat the same test many times. For example, suppose you have a method:
def process(input): # Spawns several threads to do the job # ... return output Create a bunch of tests:
process(input1) -> expect to return output1 process(input2) -> expect to return output2 ... Now run each of those tests many times.
If the implementation of process contains a subtle bug (e.g. deadlock, race condition, etc.) that has 0.1% chance to emerge, running the test 1000 times gives ~63% probability for the bug to emerge at least once. Running the test 10000 times gives >99% probability.
0.1% chance ...?
p to fail, that means it has a probability 1-p to succeed. What's the probability that all N runs will succeed? That's (1-p)ˆN. What's the probability that at least one of N runs will fail? That's 1 - (1-p)ˆN. Now plug in p = 0.001 and N=10000 into this last equation.If you are testing simple new Thread(runnable).run() You can mock Thread to run the runnable sequentially
For instance, if the code of the tested object invokes a new thread like this
Class TestedClass { public void doAsychOp() { new Thread(new myRunnable()).start(); } } Then mocking new Threads and run the runnable argument sequentially can help
@Mock private Thread threadMock; @Test public void myTest() throws Exception { PowerMockito.mockStatic(Thread.class); //when new thread is created execute runnable immediately PowerMockito.whenNew(Thread.class).withAnyArguments().then(new Answer<Thread>() { @Override public Thread answer(InvocationOnMock invocation) throws Throwable { // immediately run the runnable Runnable runnable = invocation.getArgumentAt(0, Runnable.class); if(runnable != null) { runnable.run(); } return threadMock;//return a mock so Thread.start() will do nothing } }); TestedClass testcls = new TestedClass() testcls.doAsychOp(); //will invoke myRunnable.run in current thread //.... check expected } You may use EasyMock.makeThreadSafe to make testing instance threadsafe
unsafeblocks, the compiler can actually guarantee race free thread safety. For me, this is even more important feature of Rust than the memory safety that's usually mentioned.