Designing Resilient Systems: Circuit Breakers or Retries?

I’m struggling to see why this should be an or. Use circuit breakers to prevent overloading or abusing individual instances and use retries, or backup requests to prevent customer impacts.

I've found that the best practice tends to be retry with exponential back off then ultimately log a failure (raising a warning with airbrake, rollbar, etc) and fail hard.

The post uses the term “bulwark” - this is the first time I’ve heard of that. Is it any different than a mutex or semaphore?

Good discussion. My only critique is that a discussion of what is the best kind of failure should always start by discussing how different situations mean different preferences on the kind of failure you get.

A good basic intro. A "circuit breaker" is discussed as independent of the load balancer. Often those functions are combined. How does the load balancer detect when one of its resources has gone down? Or is overloaded and just slow. And how do you keep the load balancer from being a single point of failure?

Not much mention of whether requests are idempotent. Some things you don't want to retry, like payments.

Are you retrying to use different resources? Or in hopes the same resource will come back up? Those imply different strategies. Older telephony systems tried one retry with different resources, then gave up. If a call had failed twice, it probably wasn't a transient problem.

Title should just be Circuit Breakers. Alternatively put back the (part 1).

s/or/and

Deadline based pipeline seems to work best for our cast.