What Kind of Asynchronous Is Right for You?
reactivesystems.eu
reactivesystems.eu
This always struck me as a blatantly disingenuous generalization.
What are the two services being described here? It matters very much.
If my product is an email platform, and the "API service" accepted the "send email" request by enqueuing a "send email" message for the "email sender service", if the "email sender service" is down, it absolutely does degrade the experience of API service users: "your email is sent" vs "your email will be sent as soon as we're able" is two very different promises.
I think I'd like to work for the company who doesn't care about the difference in those two promises. It'd be very easy to just implement the worst possible solution and provide the flakiest possible experience. Of course, I haven't seemed to find those employers yet in my career. Everyone wants reliability and performance and everything else that sounds nice, all without specifying how the product should really behave. Where are the companies where I can implement these brilliant solutions without the downsides?
If I'm trying to reset my password and your UI says to check my email but I don't receive an email I'm gonna be annoyed. My first guess will be some kind of intermittent delivery issue, so I'll probably send a new request to your UI. If this is important to me then after an hour or so I might open a support ticket. I'll have wasted a bunch of time checking to see if I've finally got the email. I'd much rather have gotten an error and the retried when your status page indicates the issue is resolved.
Of course. But I was presuming that your email service makes an API call to an email service provider rather than speaking SMTP itself. That API call has three possible cases:
- Ok: You're done, your ESP is responsible for SMTP retries. Maybe there's a webhook to re-enqueue something eventually after failures.
- Transient error: You retry. But you probably don't want to wait that long and there's probably no point in interleaving attempts to send other emails through the same API. So you may as well "block" (as in yield to the async scheduler) while waiting to retry.
- Persistent error: Here's when I suggest hard failing up to the user.
> If it has an error, not you [the end user] should get the error message - you are the last person able to do something about it - but someone who fixes it.
In the post you replied to I walked through in detail how my experience would be noticably better if I got an error message.
> If you want a better user experience, don't build unreliable services and then apologize for the inconvenience. Built better (more reliable, more scalable) software.
I agree you should attempt to eliminate all failability. I disagree that if you think your software is perfect you don't need to write code that provides the least unpleasant user experience possible if an error happens. Email is an exceptional example: it is theoretically impossible to write a service that infallibly delivers email to {gmail,yahoo,outlook} within a short period of time.
This article advocates for event-driven systems, but elides their subtle but major downside over request-response: they make your control flow "non-linear" and very difficult to reason about and analyze statically. That is, when an event fires, what happens in response? Well, any bit of code in the entire system could be triggered by it.
Events are powerful, but very much a double-edged sword, similar to GOTO.
On another note, as you describe GOTO as a double sided sword, now I really wonder what the positive side of GOTO is..
"Firing an event" is roughly equivalent to "running an undefined set of procedures in an undefined order".
That's useful, because consumer behavior is usually undefined, or at least far outside the scope of a single software system.
Your system firing the events should assume the size of that set is 0. That is, it should not rely on its own events for its internal control flow, why introduce what is essentially UB? That would be like a webserver querying its own HTTP endpoints instead of its local database for the same data. Abusing external interfaces for internal purposes. Wasteful and difficult to trace.
Event loops have happened in every event driven system I’ve ever encountered. The original proponents having sworn that nothing like that would happen, and yet it did.
Code is a collaboration. Everyone is putting their ideas into it. And some days our best ideas aren’t very good. Systems that require constant vigilance always unravel in the end, and event based systems require constant vigilance.
With an event system, if the loop happens infrequently enough, it will only result in a slight decrease in throughput of the system, and so the system will continue to work for hours, days, even weeks. And once a bug has survived for more than a few weeks it becomes difficult to triage.
Someone’s notion that they didn’t want errors to go unhandled. So I had to rewrite the logic so the conditional for reporting the error was inside the listener instead of around it.
Either way, it's processing some inbound messages before others for a process that is getting messages for requests as well as responses.
Personally, I have seen ocassional attempts to prioritize inbound queues, but it gets unwieldy quickly: you end up either reading nearly the whole mailbox on each iteration, or having to queue non-priority messages yourself, or do terrible things in erts code to add the ability to send to the front of a mailbox.
Since you perform this pattern matching, this means that messages may not be processed in the order they were received.
* In some systems I want to reply ASAP because a message is on the hot path, so the response is acknowledgement of persisting data and an fsync to assure the request was saved and maybe an identifier so the progress of the request can be checked. I might want to send some work to background processes.
* I feel systems lack end-to-end responsiveness. If you're polling with REST to see if something is finished then there's a problem with your design chosen by the kind of async you chose.
* I feel async/sync nonblocking/blocking push/pull is orthogonal to everything in your system design and everything falls out from that decision. If only there was a way to map between the three. I am working in this space: I want async/sync to be something that can be seamlessly switched between and handled by the infrastructural plumbing and you can focus on the business logic.
* I think systems design is actually a tree for performance reasons similar to Rust communicating between nodes is slow as it introduces round trip times.
What does their definition of event-driven really look like in practice.
Nobody has a clue.
Here is the ideal event driven system, it's async-to-async: https://github.com/tinspin/rupy/wiki/Fuse
The example is not working because I had to shut down the services for multiple reasons, but the high level of it is that you use 4 (potentially different) threads to do one request/response middle man transaction.
That way you have _zero_ io-wait or thread idling. I'm surprised nobody has copied this approach since I invented it 10 years ago. I understand why though you need your entire chain to be async and that means rewriting everything and that is a big risk when it's hard to debug.
But if you succeed you can build something that is 10x perf/watt than all other implementations. Which is going to be important when interest rates go higher and crash our entire industry.
Sometimes such people succeed anyway, but typically when they find themselves surrounded with people suffering from Impostor Syndrome. Which is to say, people who think, “this is to complex for me, I must be too dumb” and not, “this is to complex for me, it must be too dumb”
The problem with most really exceptional ideas is that, once said out loud, they sound completely obvious to most everyone else. If they come out early in a design process, people think you haven’t contributed much at all. They only see the value if the suggestions come after a plan has already started to wind itself around the axles. At which point the people suffering from Sunk Cost may push back because we worked so hard to solve this problem the hard way we’d feel foolish going back now.
If you want people to understand what you’re saying, do the work. Maybe try reading some Feynman. Stop using contractions in your examples. Use bigger charts, and use charting types people in industry actually use, instead of inventing your own. Pull out a thesaurus and find more accurate words (but still at an 5-8th grade reading level). Software made of complicated ideas doesn’t scale on the human side. Because eventually you’re staring at fifty “simple” ideas that interact in very complex patterns. If the ideas start out more complex than necessary, you run out of runway (similarly if you try to dumb everything down with “All Xs are Ys” absolutism you end up with a hundred ideas instead of fifty, and did by the same sword).
As simple as possible, but no simpler. And if you can’t explain your ideas to college freshman, you don’t understand the domain as well as you think you do.
I agree and already imagined an animation that would make it clear why async. on both ends of transition servers connections is useful.
The sorry excuses of visualizations I have now are truly WTF.
Also all links being out-of-order does not help.
I just need to pick tools. Leaning towards animated .gif now.
Edit: I fixed the example link atleast. It fetches data from the distributed db.