Fail Fast Is Failing Fast
pathelland.substack.com
pathelland.substack.com
GRAY FAILURES.
In a gray failure, a server or part of the network doesn’t fail fast but instead starts running slow. Running slow is WAY worse than failing-fast. The slow component, sometimes running at less than 1% of its normal speed, may be healthy enough to say “I’m still here!” but slow enough to clog up all the work. This makes fail-fast schemes vulnerable."
[...]
>"In electronics, there is a phenomenon called
METASTABILITY.
It means the system may be in an unstable state for an unbounded time. To have a valid signal as an output to the circuit, it must reside within a certain voltage or current level. If the output lands in the middle gray area, wonky things happen to the next circuit. It, too, can do weird things."
[...]
>"Consider a team a team of jugglers handling dozens of balls on stage. If some of them are sent into slow motion in a non-correlated way, it can be a problem. Each juggler has his own timeframe and sees balls arrive when they arrive. For a short while this may make sense. It becomes virtually impossible to juggle time-based interactions across the team as each juggler’s time speeds up and slows down unbeknownst to them."