Lessons learned writing highly available code
medium.com
medium.com
It is, no lie, far harder to make a system that can stay available and does the right thing during the partition. Even though the research has been around for decades, we're still in baby steps in actual functioning systems. Even Riak, the poster child for this sort of thing, was straight up last-write-wins for years while they were making fun of everyone else. Hard problems!
Sometimes a machine will go down and come back up. Looks like the one I'm on has an uptime of about 90 days, which seems low, but doesn't really matter. If too many of them go down at the same time, I'm notified. Haven't ever been notified.
You don't have to do 'failure detection a hundred different ways' (what does that even mean?) if you just crash, and build your systems to go with the flow, independently, like ants.
I agree the (distant) second best alternative is to do as you suggest: to try to rely on a paxos implementation that keeps a stable leader and tries to keep track of liveness. That's pretty hard too! Besides all of aphyr's jepsen test series, check this shit out: https://github.com/coreos/fleet/issues/1289 . Tough! Tough even when you think you know what you're doing. And of course no help against freak occurrences where you can't elect a stable leader any more, or you flap leaders, or the election gets stuck.
Currently the only known-good implementation of raft is in Idris. Do you know Idris? I sure as hell don't. Fortunately I know ants. Ants can die in great numbers and still move the leaf. In practice, in production, thousands of terabytes, even on AWS, my ants move and never bother me. Let it crash.
and what's ants?
So the problem with distributed consensus algorithms is that they are hard to understand. It didn't help that Lamport wrote his original paper playfully, using a complex and unfamiliar metaphor. But as a result, many implementations of the relevant algorithms tend to miss complex edge cases. Even famous ones that many large companies rely on have either had meaningful serious bugs or have been misunderstood and misused by downstream applications.
There are a couple of ways to try to fix this. The common way is to try to write a bunch of unit tests. This doesn't work. Unit tests test only those things that your tests manage to cover, and you will probably not think of all of the edge cases.
The next most common way is to use something like QuickCheck, which automatically generates millions of cases and spends as long as you want (days, hours, weeks) hammering your code. This is much better, but still nondeterministic.
The better way is to go fully deterministic, and prove out that your algorithm works, either by exhaustively checking all possible interleavings (code paths) with a model checker, or by mathematical proof.
Historically, the model checkers (e.g. Spin or Groove) have used a pseudolanguage that you describe your algorithm in, which is then exhaustively run to completion. This approach can prove that you are on the right track, but since you cannot run those pseudolanguages in production, they are not the complete solution, because you must then translate the pseudolanguage into source code of your chosen language. This is nontrivial and very frequently there are subtle transcription errors.
An alternative approach is to use a model checker that uses your native language directly; e.g., Erlang has the brilliant Concuerror program, which instruments your actual code. This is great because if it can verify that your code works properly, then you are done; no transcription is needed. Nevertheless I don't believe there are yet any Concuerror-assisted public distributed consensus algorithms, even in Erlang. I would love to be mistaken on this point.
The last approach is to take a proof assistant language like Coq or Idris, and formally prove out the properties of the algorithm using mathematical proof techniques. This is probably optimal, because the exhaustive model checkers can, with complex/slow enough algorithms, run forever or run out of memory trying to test all the cases. However, Coq and Idris are not exactly popular languages and at this time it's not easy to implement line of business applications with them. So although there is a proven, correct implementation of Raft in Coq that guarantees linearizability, good luck accessing it. If you don't use Coq, then you're forced to transcribe it to your chosen language, which, as before, is error prone and does not result in a proven implementation.
It would be possible to mechanically transcribe Coq/Idris code into a more common language while maintaining its provability, but to my knowledge that hasn't happened yet. More likely is that Idris and its successors inherit mainstream language features and start making inroads.
Note also that maybe you don't care about being correct. For example, in the spirit of moving fast and breaking things quickly or whatever, at least one major notable VC-funded project in the news has taken the approach of just increasing timeouts in order to mask bugs in their implementation. And 99.999% of the time that will probably work fine, just as centralizing your database into one king-hell instance and just dealing with downtime every blue moon will probably be fine too. Own your own reliability alpha.
Disclaimer: I've only used Riak for toy projects, never on a real production app, but...
I think that's being overly simplistic. They've had vector clocks from basically the beginning (though I gather they've been superseded now) and have favored the strategy of offloading conflict resolution to the code that's reading the value. Without implementing that conflict resolution, then it might default to last-write-wins, but that's not the same thing as "straight up last-write-wins" since they give you the tools to implement something more intelligent.
I tend to find more than 1 retry with a clear timeout is better. [e.g. I know after Y seconds that X is dead to the world and it can be safely re-ran from the top]
https://pragprog.com/book/mnee/release-it
Another clever tip I read (not from this book) was to use prime numbers for recurring events and retry wait times. The idea was that prime numbers would help avoid unwanted system stampedes or synchronization.
Well, that won't go over well with the HN crowd, I bet! It's also the most important point in my opinion. Just because a new tool solves an old problem doesn't mean it won't have all new problems of its own. Can probably go even further:
For any popular tool or framework less than 1 year old, it has critical problems that aren't known yet. If it's not a popular tool, it will always have critical problems that aren't known.
I remember a slew of small tech companies having issues when they decided to build out there app stack with nginx fronting a nosql hotness which inevitably led to overload scenarios because of the lack of constraints/scalability concerns.
Please note : this was years ago, and I know that nginx is a great web server and mongo is a great nosql db, but without a focus on scalability, you can end up with a large headache.
I suspect if the client and server worked together in this fashion you could get much closer to the linear completion time seen with no backoff and also closer to linear work (since each client would try exactly twice in an ideal world where the server accurately predicted when it should try again).
Truncated exponential (where you cap the upper limit of the retry delay to some maximum) is also often a good idea, to prevent a short service outage from spiking the retry timers to crazy numbers.
I bet the best approach might be to have the statement_timeout be the largest of all of your per-user timeouts (in case your watchdog script fails, can't connect, etc. for whatever reason).
> The ALTER ROLE command allows both global and per-database settings to be overridden with user-specific values. (...)
> The SET command allows modification of the current value of those parameters that can be set locally to a session; it has no effect on other sessions. The corresponding function is set_config(setting_name, new_value, is_local). (...)
> Setting statement_timeout in postgresql.conf is not recommended because it would affect all sessions.
My practice with long-running query killer scripts is to have them ignore queries that are known to be long by running those queries on a specific user/machine(s) and then hoarding and protecting those credentials.
I would like to add a few points.
1. Design the program as if it has crashed the last time. Always start up to recover from the last crash state. A normal shutdown is just simply the degenerate case. This would make restart and retry of program so simple.
2. Do retry with RANDOM exponential backoff to spread out the load. You don't want all the retries end up at the same intervals.
The trick is to keep your state in the right place, the answer usually being to punt state management to a database/message queue/etc. - you definitely want to avoid, say, stateful frontends.
(Among other problems, automated testing won't catch a hardware failure, or someone tripping over the power cord, or a cosmic ray flipping a bit and corrupting memory.)
While it's never 100%, you can still test quite a lot of those conditions.
We developed a TCP proxy[0] that allows us to test such cases: Datastore X being down / timing out / being excessively slow.
Tests are not that hard.
Debug the issues are harder.
To the original post author's comment about gdb/go: You can't use gdb to debug this type of problems.The interesting failures are caused by a cascade of errors - someone writes an innocent bug, which causes a single-node fault, which exercises some pathway in the fault recovery code that has unintended side-effects, which results in an unexpected condition elsewhere in the system.
Another commenter mentioned testing. That's a good idea. Here's two articles on that:
https://queue.acm.org/detail.cfm?id=2800697
http://www.drdobbs.com/testing/testing-complex-systems/24016...
If a low-level dependency starts to flake out (bad code/config/data push, unreliable host or network, load spike causing your storage to melt), retrying at intermediate stages can amplify the load and turn a small/medium-sized problem into a cascading failure. You can partially mitigate by retrying 0 or 1 times and then just give up, especially if you control all your callers or can afford that flakiness in your error budget.
Otherwise, depending on your needs you could maintain a retry budget (don't unless you really know you need it): instead of a flat K retries before giving up, enforce a process-wide per-operation cap on your retry rate. When the system is healthy, overall retry rate will be low so you have "room" to issue all K retries if needed. But if your load spikes, or your downstream dependencies start flaking out for some reason (your "fault" or otherwise), you'll avoid a positive feedback loop of death by quickly hitting your retry rate cap and shedding load which only would have made things worse and was destined to fail anyway.
EDIT: A few other remarks:
* OP's general point about putting limits on everything is really important, and closely related to the general wisdom in distributed systems development: everything can and will fail, so make the failure modes obvious and explicit. You get general robustness by avoiding runaway queries, but you also force discussions at design/development time like "what happens when this queue fills up" or "what happens if you're never able to take that lock" (though you should try to avoid needing to take locks ;)
* Instrument the hell out of the paths through your system and add lots of metrics. Logging is great, do it wherever it makes sense and you can, but a lot of the time you'll be able to get more insight (or the same amount of insight with less time spent doing ad-hoc log analytics) with something simple like counters whose value you can track over time and plot on a graph somewhere. Examples: classify downstream dependency latency into buckets, count of times you hit a code path (maybe some users require a more complex DB query than others or you're coding defensively, never expect a case to happen, but want to know if it does), cache hit rate, top sources of traffic in your system. Eventually you'll want to optimize bottlenecks and without data to identify them and prove they're fixed, you're flying blind.