How to unit-test asynchronous operations?

My solution for multithreaded testing in Java. Been using this code for a couple of years. Can easily be generalized for any asynchronous task. The main concept is using a shared barrier construct.

  /**
   * Executes the task the given number of times in the given number of threads
   * @return All the unchecked exceptions thrown by the task during execution, or
   * an empty collection; never null.
   */
  public Collection<Throwable> run(Runnable task, int nThreads, final int iterationsPerThread) throws BrokenBarrierException, InterruptedException {
    // each thread will await upon the start barrier, then run the task, then await at the finish barrier (where the main thread will be waiting)
    final CyclicBarrier startBarrier = new CyclicBarrier(nThreads);
    final CyclicBarrier finishBarrier = new CyclicBarrier(nThreads + 1);  // +1 for the main thread
    // exceptions raised by threads will be logged and returned
    final Collection<Throwable> exceptions = new ConcurrentLinkedQueue<Throwable>();

    for (int i = 0; i < nThreads; i++) {
      new Thread("Thread " + i) {
        public void run() {
          awaitOnBarrier(startBarrier, 5);
          try {
            for (int j = 0; j < iterationsPerThread; j++) {
              task.run();
            }
          }
          catch (Throwable e) {
            e.printStackTrace();
            exceptions.add(e);
          }
          finally {
            awaitOnBarrier(finishBarrier, 60);
          }
        }
      }.start();
    }
    finishBarrier.await();
    return exceptions;
  }


  /** Calls barrier.await and supresses all its checked exceptions */
  private void awaitOnBarrier(CyclicBarrier barrier, int timeoutSeconds) {
    try {
      barrier.await(timeoutSeconds, TimeUnit.SECONDS);
    }
    catch (InterruptedException e) {
      e.printStackTrace();
      throw new RuntimeException(e);
    }
    catch (BrokenBarrierException e) {
      e.printStackTrace();
      throw new RuntimeException(e);
    }
    catch (TimeoutException e) {
      e.printStackTrace();
      throw new RuntimeException(e);
    }
  }

The general answer for event-based systems is: reentrantly spin the event loop from the test and block waiting on the result.

The general answer for actor-based systems is: wait on a blocking message receive from the actor task.

The general answer for simple shared mutable state thread-systems is: Write a complex condition-lock and thread combination (which will probably be subtly wrong the first time) to block your thread and wait on the result triggering the condition.

The most exciting defect that I ever had to debug was a dormant async bug that had been in the code for many years. I had checked in what I thought was going to be a fairly innocuous change in a component in our team's subsystem, but it ended up breaking the full build at the end of the day and was going to be backed out the next day if I didn't prove I wasn't responsible for it.

It was about 6AM after a night of constantly building debug ROMs with different instrumentation to try and get one that I could get both meaningful debug information out of but still suffered from the defect that I finally tracked down the problem. It turned out that a circular pseudo-dependency involving 3 different subsystems and five different components was deadlocking, and all I'd done is change the timing a fraction of a second to trigger the deadlock.

Asynchronicity is hard to test, and you'll always, always get it wrong. Don't sweat it too much. Unless you're building flight control software, just make sure the common use cases work 100% of the time and if broken edge cases turn up, fix them as and when.

A worthwhile read might be the unit-testing framework from Twisted (Python asynchronous networking library), called "trial". See here: http://twistedmatrix.com/trac/wiki/TwistedTrial or poke around the code for more in-depth details.

To unit test an asynchronous operation, you need an asynchronous language or library. In Erlang you hardly even notice because the language natively supports it. In Perl I've hacked some stuff up with POE but it's way more painful. But once you have the asynchronous capability, it's not much different than writing a normal unit test.

In JBlub, I ordinarily just create an inner class within my test class that implements the event listener interface I care about, and assert that the listener got called. This works fine for me when I need to test complicated UI stuff.