I'm unfamiliar with "circuit breaking" and "bulkheading" in this context. Could you provide a bit more description so i can research them and make sure i know what i should know? :)
“Bulkheading” is making sure that failures in one area don’t cascade into causing failures elsewhere analogous to how bulkheads on a ship prevent a single breach from sinking the whole ship.
While 508s were being sent, there was also a significant number of 503s (IIRC). A 503 is an absolute hallmark of something reaching max utilization. Sometimes it's a bad code push that results in memory bloat and then swap or significantly increased request handling time on a poorly threaded server (think a code push that makes a blocking linear request in a loop), but the vast majority of time an upstream dependency (specifically a data store) has been overloaded.
So for whatever reason a data store gets slower. What happens upstream when this happens? The number of incoming requests is constant, but the time each individual thread spends attempting to talk to the data store is constant (or worsening). This means each request blocks longer (resulting in potential thread starvation) or there are more concurrent requests (load) to the data store.
This creates a feedback loop of doom: as a data store slows down, its load (the number of requests it's handling at once) increases until complete failure.
The only way to stop this behavior is by "failing fast." This is how a circuit breaker works. When the data store starts responding slowly, it’s important not to hammer it with even more load, so your client watches the number of load related failures (or response times) and automatically fails requests immediately without sending it to the data store (circuit breaker). This allows the data store to become unloaded and get out of the doomed feedback loop.
"Bulkheading" is about one part not taking down a whole, which is more thoroughly described here: https://docs.microsoft.com/en-us/azure/architecture/patterns...
Even with a monolith, often one can spin up multiple copies, and assign different routes in a front end http proxy to different copies of the monolith.
It makes it extremely clear which endpoints are consuming how much resources. And if one copy goes down, the others are ok.
Absolutely one of the best things you can do, not just for protecting against faults, buy also for understanding the cost of different endpoints.
https://cloud.google.com/appengine/docs/standard/python/how-...
The idea is that at the time a request is made, you have a pretty good idea of the maximum amount of time it can take, so you attach that as metadata with the request.
If any service anywhere in infrastructure sees that a request has taken more than its budgeted time, it has permission to immediately stop handling that request and return an error.