> 92% of catastrophic failures in tested distributed systems were triggered by incorrect handling of nonfatal errors
This. If you take nothing else away from the article (which has a lot) take this: fail well, don’t fail poorly.
This. If you take nothing else away from the article (which has a lot) take this: fail well, don’t fail poorly.
What is the system state when it does error?
What is the best possible recovery from each error state?
What can the user/caller expect for an error?
E.g. if you're in python and raise a value error when an API is rate limited, someone down stream from you is going to have a bad time.
I have a lot of Go testing shims that look like:
type UserGetter interface {
GetUser(userID string) (User, error)
}
type TestUsers struct {
Users map[string]User
Error error
}
func (t TestUsers) GetUser(userID string) (User, error) {
if t.Error != nil {
return User{}, t.Error
}
user, have := t.Users[userID]
if !have {
return User{}, ErrUserNotFound
}
return user, nil
}
It's more complicated when you have something like a byte stream where you want to simulate a failure at arbitrary points in the stream, but similar techniques can get you as close as you like, depending on how close that is.
From there, in terms of "how do you handle non-fatal errors", there really isn't a snap rule to give. Quite often you just propagate because there isn't anything else to do. Sometimes you retry some bounded number of times, maybe with backoff. Sometimes you log things and move on. Sometimes you have a fallback you may try. It just depends on your needs. I write a lot of network code, and I find that once my systems mature it's actually the case that rather a lot of the errors in the system get some sort of handling beyond "just propagate it up", but it's hard for me to ever guess in advance what they will be. It's a lot easy to mentally design all the happy paths than it is to figure out all the ways the perverse external universe will screw them up and how we can try to mitigate the issues.
> In 2014, Yuan et al. found that 92% of catastrophic failures in tested distributed systems were triggered by incorrect handling of nonfatal errors.
To the point I've seen it cause real bugs in production.
The problem in Node is you can either throw to make a 4xx or you can return a 4xx so downstream there are 2 things to check.
Typical example would be processing an event that you can't handle from a message queue. You don't want to crashloop, so you'd probably have to throw it away on a dead letter queue and continue processing. But then, is your system still correct? What happens if you later receive another event relating to the same entity, which depends on the first event? Or sometimes you can't even tell which entity the malformed or bug-triggering event relates to, and then it's a real problem.