Circuit Breaker
martinfowler.com
martinfowler.com
If you're going to hit this kind of problem often, I'd suggest using a language with better concurrency primitives, either actors (Erlang/Akka in Scala) or a reactor (node.js).
Granted, it's often not up to a late-stage maintainer which language was used, but the state of the art in language design is giving us much more power to deal with the kind of problems this "design pattern" solves.
You might implement it differently, for example the circuit breaker might just be a wrapper around your original function `Request => Future[Response]`. But I don't see how using actors saves you from thinking about the consequences of throwing a normal workload at a failing service.
If you're using an actor-model language then you probably implemented that pattern without thinking about it because it's just part of handling each state of an actor. Especially in erlang where you have a timeout keyworkd and where the OTP framework exposes more complex error recovery patterns.
In fact, unnatural as they are, I often had to use Java-style patterns to break through Akka patterns. Like it or not, the "Java-style patterns" are a lot closer to the metal than futures or actors are.
Some hypothetical examples:
Java/Ruby: 8 threads, mean service time: 50ms (20 requests/s). A system like this has upper-bound throughput of 160 requests/s. If one request is allowed to timeout, say 5s, this reduces the effective throughput of the system to 7 threads as one is "locked" servicing the timed-out request. It doesn't take many requests like this to significantly degrade performance, and the only remedy (absent circuit breakers) is to throw a ton more hardware/workers at the problem = $$$.
Consider the problem in, say, node.js. If a single request times out, that request will cause problems, but it won't have _nearly_ the same capacity-starving effects as the thread-bound example above; the request will simply timeout, but other requests won't be starved because the number of in-flight concurrent requests isn't limited by the thread/process count of the system.
I'm realizing as I write this that I'm straying a little off-topic here, but the point is, using process/thread-based concurrency in a high-performance system where failure is likely is a bad idea. It's just too easy to get the kind of failures Fowler describes: "What's worse if you have many callers on a unresponsive supplier, then you can run out of critical resources leading to cascading failures across multiple systems."
This works well for many workloads. It allows a straightforward blocking-IO model, but you don't typically worry about a few slow requests bogging everything down (which you do if you only have 4-8 unicorn processes). I'd say in many apps, the database becomes a bottleneck before the pool runs out of threads.
The recovery method will be specific to the remote resource. Eg, if it's often congestion related then a retry with an exponential delay might work. A circuit breaker is just one of many possible fault detection and recovery modes. What language has built-in support for all of them?
Also, I've used this idiom in a different context. Some of my code parses a file containing multiple records, one per line. Some of the records are unparseable, for rare but valid reasons I won't get into. The proper action in that case is to report the problem, skip the record, and go on to the next.
But sometimes people use the wrong format entirely. If (say) 95 of the first 100 records are invalid, then it's very likely that the wrong file was passed as input. In that case, the "circuit breaker" should trip, rather than incorrectly trying to parse the rest of the file, which might have occasional "valid" records. My implementation has a low-level reader, plus an intermediate circuit breaker to add the specific failure policy.
Since your suggested solution doesn't seem to match the problem, and because the referenced idiom applies to more than concurrency, I respectfully disagree with your assessment.
Even if you are using actors or the reactor pattern at some point they need the equivalent of a circuit breaker in them. It just pushes it to a lower layer.
Unless you're just advocating avoiding distributed calls which require a timely response whenever you can, which I agree with as much as is feasible, but obviously can't always be avoided.
I agree that at least in Erlang/OTP timeouts are first class citizens (not sure about Akka) and so that kind of pattern is implemented more naturally.
Am I correct that blocked calls can still accumulate before $threshold calls timeout ? e.g. if 1000 calls are initiated before 5 of them timeout, there are still 1000 blocked calls.
It seems that this problem can also be resolved by limiting the number of in-flight calls: If too many calls are already waiting for a response, reject further calls.