Distributed Systems Testing: The Lost World

"..trace logs, a relatively old technique that glorifies printf as First-Class Citizen Of The Testing Guild.."

I don't understand the stigma attached to logs and debug-by-print. When is a tool that uses x to provide y 'glorifying' x?

I've built and regularly use two tools that rely on logs:

a) I write white-box tests that allow me to check not just what a function returns but specific events in the course of the function running: http://akkartik/name/post/tracing-tests. This allows me to check, for example, that searching in a sorted array doesn't require double the lookups when the size of the array doubles. It also allows me to test subcomponents without calling them directly, leaving their interfaces underspecified and flexible to change later without needing to modify a bunch of tests.

b) I often debug my programs by dumping a trace and running it through a 'trace browser' which starts out showing just the coarsest level and allows me to drill down in specific places as I want. This 'zoomable' google-maps-like UI gives me all the benefits of time-travel debugging at a fraction of the system engineering it usually takes. (I still can't get the feature to work in gdb..) Try it out:

  $ git clone https://github.com/akkartik/mu
  $ cd mu
  $ ./mu browse-trace .traces/factorial-test

It'll take ~30s to compile the first time you run it (C compiler required on Linux/OS X). Once it compiles, you'll see an ncurses program. Use 'q' to quit, hit enter to zoom into lines (you can see how many lines are collapsed in parens), backspace to zoom out on a line.

These are both things nobody can do on a conventional toolchain, and super easy to build. Maybe print seems trivial because nobody's building tools around it?

The best way to test a distributed system is to break it intentionally after you think you have sufficient monitoring to find all the problems. Then you'll know if you have sufficient monitoring.

And if you have an outage where you didn't have sufficient monitoring, then you add it afterwards. But at least by breaking it intentionally you can at least be watching.

Distributed testing is very important, but based on article doesn't get as much papers on it. Some plausible explanation:

1. Making things that actual work is industry, which writes very few papers comparing to academia. Academia cares more about design, less about working large scale systems.

2. Testing is more about well-oiled machinery, less about some super clever algorithm. Testing is more of a craft than art.

Having seen both sides, I would argue that academia doesn't really have a subfield that fits distributed systems well (most industry systems span multiple academic fields, from networking to fault tolerance, databases, OS, etc). Not to say that people don't research it, but more people research subcomponents.

Also, I would argue that academia is about 7 years behind industry in this area, and so papers don't match up to reality well.

On the other side, industry testing is usually very specific to the internal product created, and so there is an incentive not to publish anything.

There is some nice academic research outside of CS, notably mechanical engineering, on systems testing. It is pretty applicable. Contracts, sub-unit testing, etc are all anaolgous.

My two cents.

A friend and colleague of mine developed a simple distributed testing framework in JavaScript - https://github.com/PsychoLlama/panic-server (It works in conjunction with Mocha, Jasmine, and possibly other test frameworks - https://github.com/PsychoLlama/panic-demo)

(We work on an open source distributed data sync engine and needed a way to test distributed aspects of that engine - https://github.com/amark/gun/)

IMO if testing this stuff is difficult, rather than simply understudied, one shouldn't be dismissing formal verification sans evidence.

>> There’s too many things packed into the concept of testing distributed systems, and that is pretty clear in what came into my Twitter feed

I think this gets closest to the truth. That first slide regarding "testing a microservice architecture" with its complicated block diagram hints at why the level of abstraction is too high to be useful. In the end a distributed system consists of components that have interfaces and which interact with each other through those interfaces. You test the component modules in unit testing, the components themselves at the interface level, and the whole system in end-to-end request and response flow. That latter is assisted by centralized event collection and storage tools like logstash + elasticsearch, but it could just as easily be dumped log files. Whatever works. That's the closest I've personally come to "distributed systems testing."

Distributed systems testing is well studied and applied in industry, but there is significant IP and competitive advantage attached to it. For this reason, papers are rarely published. So, this world is not lost to everyone ;). You will see such expert frameworks defined and applied in aeronautics, military and transportation systems, for example. The key challenge is reproducibility of the testing. Central logging of events is not enough, you need some way to enforce or guarantee the global causal ordering of events. Leslie Lamport is recommended reading.

What worked for me best was to have as thorough monitoring as possible (IO, mem, CPU, per process, etc.) on infrastructure side and as complete event sourcing on the app domain side. It's easier said than done of course, and in a brownfield situation you will never achieve perfection in terms of coverage.

Once that's in place, write tests with selenium or unit test framework with parallel extensions (depending if cases are frontend or backend based) and analyze using home made tools.