Testing Distributed Systems
asatarin.github.io
asatarin.github.io
Good automation around preparing/provisioning the testing environment is a necessary companion to the testing tools/frameworks themselves.
Most commonly, fully-capable testing environments aren't available during the inner loop of development (where the dev setup can usually only run unit tests or integration tests for 1-2 services + a database).
Because of this, people tend to rely solely on their CI pipelines to run integ/e2e tests, which can slow things down a lot when one of those tests fails (since the write/run/debug loop has to go through the CI pipeline).
As an industry, I think we should start taking automation and developer productivity more seriously—not least when it comes to writing and debugging tests for complex distributed systems. The more we can lower the marginal cost of writing and running tests, the more effective our test suites will become over time.
Shameless plug: My company (https://garden.io/) is developing a framework and toolchain to bring the full capabilities of a CI pipeline to the inner loop of development, so that developers can efficiently run all/any test suites (including integ/e2e tests) in their personal dev environments.
We do this by capturing the full dependency graph (builds, deploys, tests, DB seeding etc.) of the system in a way that can power CI, preview environments and inner-loop development.
While the problem isn't inherently trivial, the issues the tooling can solve is, which is the order of startup and is usually solved with a waitforit startup script, as these services generally talk over network with each other.
The real challenges, like determining seed data etc is too project specific to be abstracted away.
Not to take away from your problem domain, it would be nice to have framework to use so the developers don't have to wire the plumbing for these automations manually anymore - it's just not going to solve the issue that developers will have to wait for the ci pipeline to get their results
Hardware resources are definitely an issue. That's why we generally recommend using remote development environments, which aren't as resource-constrained as the local dev machine. Making that comparably smooth to the local dev experience (e.g. for live reloading of services without rebuilding containers) needs some clever tooling (which is partly the reason we're building our product).
With production-like remote dev environments, you get the same capabilities as your CI environment, but can run test suites ad-hoc (and without having to spin them up and tearing them down for every test run).
There's no fundamental reason why CI environments should have capabilities that individual dev environments can't have—it's all a matter of automation in the end.
> The real challenges, like determining seed data etc is too project specific to be abstracted away.
Very much agree with that! The generic stuff (dependencies, parallel processing, waiting for things to spin up etc.) should be taken care of by the tooling, but without constraining the project-specific stuff (which is highly individual).
On the contrary, I almost always advocate for having a rack full of machines in a/the office, running some kind of workload management (kubernetes, vmware/proxmox or a combination of the two).
Hardware is dirty cheap, and plenty fast these days.
If you have an office, chances are you already have a server room anyways (enterprise grade network switches require cooling anyway) you might as well throw a bunch of physical machines in there.
The only real issue I see is that most developers have literally no idea of the runtime resources needed by their code, for a number of reasons (like runtimes hiding that kind of information and the general mindset that pushing out new releases/features is more important than tuning existing ones) so in the cloud developers will just provision bigger and bigger environments. It's all fun and games until two things happens: (A) environment provisionin in the cloud will take long times just like on developer machines andb (B) company earnings will be eroded by cloud infrastructure bills (on-prem infrastructure OTOH provide tax shielding)
I was surprised to see that while there is so much tooling in opentelemetry (protocols, agents, SaaS) there doesn't seem to be any option for generating static reports right now. Is my plan just wrong?
This is the ethos of what we're building at https://www.kurtosistech.com/ : expecting devs to learn the nuances of Docker and Kubernetes is like expecting them to code in assembly. You can do it, but there are higher levels of abstraction that are more effective (e.g. "I want this binary talking to this binary on this port with these args; you figure out the rest").
The hypothesis is that a unified platform for packaging a microservice, wiring it together with other services, setting up dev & testing environment environments, hooking it into CI, and furnishing a toolkit for debugging them (with some of the tools on this list, like a built-in Chaos Monkey) can reduce the cost to develop by distributed systems by an order of magnitude. If we can achieve that, does managing a distributed system become a single-developer job rather than a whole-team job?
The summary papers from cloud providers are also especially useful.
Distributed systems, it honestly is just a toy for the tech monopolies.
Nobody I've ever heard of in a company with less than a billion dollars of revenue is bothering.
Who ever thought "let's build a software program which has major issues dealing with clock errors!"
My day job doing distributed compute at a "main street" company is/was completely eaten by "why the hell would you even, just throw Postgres on a 64 core 2TB RAM machine with replication."
Why indeed.
That's kind of why I brought up the idea of clock errors.
Single leader distributed systems are honestly an entirely different paradigm to multi-leader. You have a main node, the main node controls routing and which data goes where.
There can only be one main node at one time, hence no latency/clock errors/multi leader fail-over/PAXOS/voting/stale keys/exotic application specific conflict resolution logic.
Done, simple.
(Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.)
Really, what I am calling "distributed systems problems" are the subset with multiple simultaneous, geographically separate leaders.
This is not accurate.
You can certainly have clock errors in "single leader distributed systems", simply by virtue of the fact that the primary rotates throughout the cluster.
And, perhaps even more surprisingly, you can certainly have clock errors in "single node non-distributed systems like Postgres", i.e. clock errors are completely orthogonal to distributed consensus protocols (unless you're doing something dangerous like leader leases, which I don't recommend).
The problem is simple: what if the system time strobes or gets stuck in the future? How does this affect things like recording financial transactions? What if NTP stops working? Distributed systems aside, it's pretty tricky.
I've worked on this problem of clock errors specifically for TigerBeetle [1], a financial accounting database that can process a million transactions a second. In our domain, we can't afford to timestamp transactions with the wrong timestamp, because this could mean that financial transactions (e.g. credit card payments) don't rollback if the other party to the transaction times out, so liquidity could get stuck and tied up, for the duration of the clock error.
[1] https://www.tigerbeetle.com/post/three-clocks-are-better-tha...
This is how most distributed system solve the consensus problem: the nodes automatically run leader election algorithm so that eventually only one leader remains.
>> (Secretly not so simple, all of the above problems are actually hugely important to resolve in any monolithic database. However, Postgres sorts it out for you.)
Except that in case of network partition of the primary node, Postgres can't promote a new primary node from the replicas automatically, unlike consensus systems such as ETCD/Zookeeper.
In your opinion what would qualify as a "distributed system"?
But I was very much thinking of multiple leader distributed systems.
Anecdotal. But dealing with clock errors and multi-leader issues are of far greater complexity than caching data.
Replicas, load balancers, and all the vagaries of single leader systems are not absurdly difficult technical problems.
They're problems that can be solved by a team with a single senior engineer who knows what they're doing.
I believe that when most people talk about "distributed systems", they're talking about some form of microservices architecture, where components of a single system are arbitrarily separated by network requests for nebulous reasons.
Not so, decentralized systems are also inherently distributed. The whole point of both the old "Web 3.0" and the newer "Web3" is to make "distributed" thinking a part of the web, beyond the now trivial "linking to a document-like resource on a different host" scenario that defines hypertext/the "Web of documents".
(Of course one may then retort, as Jack Dorsey apparently has, that "Web3" itself is merely "a VCs' plaything". One has to be aware that these developments are somewhat speculative and pretty far from a real consensus.)
Also it’s a complete fallacy that “if we’re successful or grow large enough, we’ll just rewrite it”.
And then there’s the logical problems of managing complexity, distributing work to development teams and even things as determining change impact that become terrible difficult.
I say we should design with distribution in mind but deploy without it. Assuming you can scale up forever is an expensive and stupid mistake.
Incidentally it turns out that even adding any deferred processing (in our case SQS+workers) on simple LOB systems can have nasty problems like clock skew and QoS to consider which are very distributed pains :)
Pretty much. I don't want to generalise for legacy context however.
> I say we should design with distribution in mind but deploy without it.
Oh yes, absolutely. You can pretty reasonably predict when your compute needs will scale up such that you genuinely need to make the shift, and plan for it with a modular architecture.
> Assuming you can scale up forever is an expensive and stupid mistake.
See here's the thing. I think it has been, in the past.
I'm definitely sure that there will be applications in the future which will scale past single leader, multiple nodes.
What I've seen in the last few years was we just didn't ever have those demands (except for deep learning).
I wasn't working for small business, at a genuine mega-corp.
The total production data inflows and egresses of my mega corp, never peaked past I think 200MB/s in any year (for business critical systems. There was some user facing video that was jettison-able). The daily peak was far lower than that.
All production compute needs, bandwidth and compute, were DWARFED by employees on Zoom and Youtube.
The sum total of all proprietary OLTP data, across the company? And we had roughly 14 different legacy proprietary business systems from acquisitions. We had COBAL, we had DB2, we had it all.
The architect of our consolidation had it at less than 5TB (excluding photographs, backups and duplication etc).
40+ years of business critical OLTP data. And all the analytics was done on trailing views of the replicas.
Given the direction of Moore's law, I can safely say that for my former multi-billion dollar employer, Moore's law is going to outpace any of our business requirements.
(Except for photographs. But all our deep learning was being done by a spin out).
Distributed systems are exciting and interesting, but currently not worth their investment for 99% of the cases. Things might change though in some decades.
The issue becomes when you reach the scale of compute needs such that you cannot "jam" all your requirements into a single leader multiple node cluster.
The amount of data you'll need for THAT, is becoming unimaginable.
64 core 128 thread leader nodes have outrageous potential to scale.
That said, that doesn't quite apply to my Postgres example (barring some nifty DBA tricks and third party extensions).
But you can appreciate how much data you're going to need to blow past what 128 threads can do for you.
It does however shift 10k shitty queries per second which is impressive.
It's called Filibuster:
https://filibuster.cloud/index.html
Example of testing a small Java application with Filibuster:
Also, industry is now transitioning where even small business might need geo distribution , not just for compute but also for data. That is going to make distributed systems even more complex even for the small guys.
Cloud services have made a lot of this stuff easier for average Joe but you can’t really avoid dealing with at least some fundamental concepts of distributed systems