Comment by ttymck | Hacker News Reader

Comment by ttymck | Hacker News Reader

Parent Full thread

> a failure of one service does not cascade and degrade the functionality of all other services using it.

This always struck me as a blatantly disingenuous generalization.

What are the two services being described here? It matters very much.

If my product is an email platform, and the "API service" accepted the "send email" request by enqueuing a "send email" message for the "email sender service", if the "email sender service" is down, it absolutely does degrade the experience of API service users: "your email is sent" vs "your email will be sent as soon as we're able" is two very different promises.

I think I'd like to work for the company who doesn't care about the difference in those two promises. It'd be very easy to just implement the worst possible solution and provide the flakiest possible experience. Of course, I haven't seemed to find those employers yet in my career. Everyone wants reliability and performance and everything else that sounds nice, all without specifying how the product should really behave. Where are the companies where I can implement these brilliant solutions without the downsides?

I think it's clearly better to have the email eventually sent than the case where a completely synchronous service is down resulting in no email being sent

Under certain circumstances you might want to put backpressure on the email emitter. Under certain circumstances you might want to hard error.

If I'm trying to reset my password and your UI says to check my email but I don't receive an email I'm gonna be annoyed. My first guess will be some kind of intermittent delivery issue, so I'll probably send a new request to your UI. If this is important to me then after an hour or so I might open a support ticket. I'll have wasted a bunch of time checking to see if I've finally got the email. I'd much rather have gotten an error and the retried when your status page indicates the issue is resolved.

MessagingService includes its queue depth, or age, to LoginService in the enqueue response. If over $threshold, LoginService tells UI to add a parenthetical "(our email is backed up but you'll get it soon)" notice. Add alerting around high queue depth and I think you've buttoned this up.

Agree that's another way to solve it. Treating password reset emails as transient, erroring, and not enqueuing would also work and might be simpler.

For sure, guess it depends on the volume of the service and balancing customer support vs. engineering effort.

Thanks for your feedback! Delayed e-mail can have of course have many other reasons than the service supposed to send it being down. But let's go with your example. Let's say you have a LoginService, and a MessagingService, and if you request a password reset, the LoginService sends a ResetRequested event. Now if the MessagingService is down, you'd still get told to check your inbox, although the e-mail hasn't even been sent. That's a bad user experience. Agreed. Your suggested solution is to make the communication between LoginService and MessagingService not event driven. So if the MessagingService is down, you can "hard error". But how much better is the experience then, really? As a user, you request the reset, and after a few seconds (you're already annoyed and checking if your internet connection is down), you get an error message that you can't reset your password right now. I'd argue that option is equally unacceptable. What you need to do, is to make the MessagingService better. In an event driven system, each participating service makes a promise. The MessagingService must give service level guarantees, it must be built and scaled to be able to process all the events it's listening to. If it has an error, not you should get the error message - you are the last person able to do something about it - but someone who fixes it. If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software. In the end, that's what event-driven architecture is about.

> Delayed e-mail can have of course have many other reasons than the service supposed to send it being down.

Of course. But I was presuming that your email service makes an API call to an email service provider rather than speaking SMTP itself. That API call has three possible cases:

- Ok: You're done, your ESP is responsible for SMTP retries. Maybe there's a webhook to re-enqueue something eventually after failures.

- Transient error: You retry. But you probably don't want to wait that long and there's probably no point in interleaving attempts to send other emails through the same API. So you may as well "block" (as in yield to the async scheduler) while waiting to retry.

- Persistent error: Here's when I suggest hard failing up to the user.

> If it has an error, not you [the end user] should get the error message - you are the last person able to do something about it - but someone who fixes it.

In the post you replied to I walked through in detail how my experience would be noticably better if I got an error message.

> If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software.

I agree you should attempt to eliminate all failability. I disagree that if you think your software is perfect you don't need to write code that provides the least unpleasant user experience possible if an error happens. Email is an exceptional example: it is theoretically impossible to write a service that infallibly delivers email to {gmail,yahoo,outlook} within a short period of time.

Reply on news.ycombinator.com