Hacker News Reader

Comment by jimbokun | Hacker News Reader

Since these insert events are being buffered, what happens if the process dies? Are all of those inserts lost?

I feel like Erlang/Elixir is designed to handle cases like this robustly, but it's not clear to me how this code avoids losing data when a process crashes, potentially up to 5s worth of updates!

View on HN

Short Answer - yes, data will be lost. THAT SAID - this is true of any buffering solution, it's why even the 'sync' style of request can still time out.

Long Answer - What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir. There's a tradeoff that persistent storage is slower than memory, but it's persistent. So do you want to be fast, or do you want to be durable?

However, there is some flexibility to address this at a system level. Namely, you wait for confirmation. A client, be that an actual user, another process, etc, wants to know if the write has been persisted. Whereas many other languages make this sort of caching layer completely transparent to the client (i.e., they return success immediately, and so failures mean invisible data loss as the cache is dropped), Erlang/Elixir's model makes it so you HAVE to think about this. I sent a message to the downstream process; do I care about a response? If I don't get a response for any given interval, I don't know what happened to that message. I can still do useful work in the meantime, but I don't know that my message was fully handled (persisted). I can retry or I can report failure or whatever.

This is true in any distributed system, and in Erlang/Elixir it's expressed very evidently in the language constructs (rather than being hidden from you).

You can build local disk caches if you want (in fact, there are included tools to make this super easy for you; ETS, DETS, and Mnesia allow you to shove Erlang terms into memory storage, disk based storage, and a hybrid of the two with some nice DB-like behaviors, respectively), but you need to choose to slow your message ingestion to the speed of local disk writes in that case (as well as handle synchronization of deletes in the event of multiple writers to your downstream). An depending what your upstream is, that doesn't provide a guarantee (i.e., a write to local disk != a persisted write from a user perspective, because the disk could crash before it ever makes it off the local one).

> What happens in any language where you batch stuff in memory? If the system breaks down, you lose that data. This isn't unique to Erlang/Elixir.

A little over a decade ago I did a project using Erlang. My next project was using Ruby, and I was suddenly _horrified_ by the fact that I had no idea what would happen if the application crashed (and Ruby apps, at least in those days, crashed fairly often). Erlang/Elixir both forces you to think about these things, and gives you tools to address them, where many other languages (or their libraries) simply assume that we will stay on the happy path.

Of course, you can be very productive using languages that just ignore the possibility of very rare failures. Many successful systems work on the basis that sometimes shit just happens and maybe some data does get lost. But, after programming with Erlang or Elixir for a while, that situation starts to feel less acceptable!

100%

I consider the couple years I built systems in Erlang to be fundamental for me. It's affected, for better and worse, my entire approach to system design, at every level. It's meant the stuff I or (now that I'm in management) my teams tend to write is incredibly resilient (compared with the other teams in the department), but also meant that I have a really hard time with any Silicon Valley interview.

I 100% agree with this, but would add that actually in my case I feel like it has helped with Silicon Valley style interviews. Architecting a single BEAM application involves a lot of the same thought processes that go into designing any resilient distributed system so I have felt well prepared for any design questions. Additionally, being forced to write algorithms with immutable data structures, recursion and explicit state prepares you ridiculously well for solving the graph questions that FAANG like to give in interviews.

Specific design questions, yes. The "45 minute to design a system" ones, not so much. The hard part of those is not getting too bogged down in details, and this just means I have one more, extremely important, likely unappreciated by the interviewer, detail I can get bogged down in: what happens when things fail.

What happens when things fail is an absolutely critical part of design / architecture. It's disappointing that interviewers works consider that an unimportant detail.

Again, 45 minutes for system design interviews. I could go on a long diatribe of why the FAANG interview strategies are flawed, but everyone knows that, FAANG interviewers included. You play the game if you want the role; if you're just, you know, good, and don't want to learn to play the game, then you're an acceptable false negative.

The article touched on it briefly, but perhaps it wasn't clear.

If the process dies, the Supervisor invokes the `terminate/2` callback so you're still able to process events.

Here are the relevant lines: https://github.com/plausible/analytics/blob/b724def948d51a0f...

Keep in mind you need to trap exit signal to tell the supervisor to invoke the callback, as done so here: https://github.com/plausible/analytics/blob/b724def948d51a0f...

The erlang docs also mentions this: https://erlang.org/doc/design_principles/gen_server_concepts...

I usually utilize this with an ETS cache to save and recover GenServer state in the event of a crash.

I use this callback to capture the 'data' that caused the crash, so that it can be used to recover later or better understand why the crash happened.

If I have as significantly long transaction, I can restart the transaction from transaction_id + 1 forward and then handle the actual transaction out of band by hand if its something very odd.

I usually insert a logger in these code paths too, but the crash handler is usually after the logger, which makes it easier to reason what has happened if you are dealing with mutable state.

I think you might solve this by writing a "reliable" supervisor process which actually receives the messages, and preserves message or state history to replay for its supervised process in case it crashes. Of course this really just shoves the problem up the stack, but at least you can stick to "let it crash" when writing the supervised process, and take more care when writing the supervisor which has a smaller scope of responsibility.

Yea, they're just in memory, so they're lost. Erlang/Elixir doesn't solve this. There's no getting around the fact that you need an ack that your event was processed and saved to some durable store. A supervisor won't save you from lack of acknowledgment at the app level. Any restart or retry logic within beam itself is useless if a beaver chews through a power cord and the machine halts. And what is an ack, anyway? Is an ack from mongo as good as an ack from postgres? This is all stuff the language can't know. These are all business questions, all the language can do is give you tools handle these cases, which Erlang/Elixir does.

You can add retry logic in your genServer, but for anything but the most simple use cases, you would want to add a Supervisor and define a retry strategy.

https://elixir-lang.org/getting-started/mix-otp/supervisor-a...

Furthermore I have heard it is actually possible for casts to be dropped under load. I am having a hard time confirming this, and had a hard time confirming it back then too. And even worse, I realize most of the time we code as if the casts will be reliably delivered.

Can someone chime in on this?

I think that casts are guaranteed delivery if the process exists. Problem is, the caster can't know that with certainty (there could be a race between checking, process death and sending). Or the recipient could just ignore the cast altogether. Out if the box it's impossible to know which happened.

Calls by contrast check out a monitor on the counterparty and crash with a timeout so you have guaranteed delivery and acknowledgement, or crash the calling process.

A lot of times people from other plarforms rush to use casts when they "don't need a response" but the actual meanings have more to do with failure domains and rate limiting back pressure; my personal feeling is you should default to call and only use cast when you need failure isolation.

I don't think that's quite true, at least not of a gen_X cast.

If you use erlang:send/3 with options or erlang:send_nosuspend/2, you can have some cases where messages would be dropped without trying. I think that may be the source of the drop under load you're thinking of?

Without nosuspend, if you send to a node that's connected, dist will queue it to be sent, but there's no guarantee it's received because networking, and it may not even be sent if the other side has gone away and the send queue is already too large; There is a hard to express constraint that if your message doesn't get received in this case, dist will disconnect eventually, but it's important to note that the tick time-out doesn't guarantee timely delivery either; as long as some data is flowing, you can get quite a backlog; I think I've gotten net_adm:ping times above 30 minutes in some cases. Also, if the dist connection is dropped, but both nodes are online, it will likely reconnect shortly, and some messages will have been lost.

If you send to a node that's not connected, and didn't specify no_connect, dist will queue the message while attempting to connect, but if that attempt fails, the messages will be dropped.

It's also possible for code running as a gen_server (or GenServer, I suppose) to check how many messages are queued for it, and run different logic. I've written gen_servers that would drop optional requests if the queue was large. Also, if client timeouts are known, there are ways to approximate the time spent waiting in queue, and drop requests if they are received after the client already timed out; it's a little tricky to do this though.

Erlang casts are always delivered, or the sender crashes before that point.

To local pids. Remote, no guarantee is made. Ergo, generally better to assume no guarantees. Same as with everything else; you have at most once, or at least once; there is no exactly once (except in marketing speak that uses an identifier to allow at least once to dedupe within a window).

https://medium.com/learn-elixir/message-order-and-delivery-g... is what you want.

I'll chime in explicitly to your question though - messages don't just disappear without a reason, but the reason may not be visible to you. It's possible a receiving process got a message, crashed partway through processing it (possibly after you even sent other messages, so dropping those, too), restarted, and now is handling the next incoming messages, making it look like a message dropped. It's possible a receiving process is on another node (in distributed Erlang) and the packet dropped. Etc etc.

These all look the same. The way to handle them is always the same; accept you have at most once delivery, or include an 'ack or retry' mechanism and write your system to handle at least once delivery. As that article mentions, Erlang makes you start thinking of your system as a distributed system from the get go. This initially feels incredibly inconvenient, but as you go down the distributed system, fault tolerant, CAP-bound rabbit hole, it becomes incredibly helpful. The system doesn't hide the things it can't guarantee from you; it forces you to feel uneasy about them.

(Whereas, for example, in Java, I don't know that that worker thread has terminated. So some other thread's fiddling with some bit of shared memory to communicate something to that thread does nothing. Which I've had happen in production and led to a single instance out of the fleet have stale data, which caused us no end of grief. Erlang forces you to ask, as you've just done, "what happens if this message doesn't get received because something happened to the receiver"?)

Actually in Erlang/elixir, your own crash is something you can plan for and handle. if you check the code you will see in init() that the author captures exits. then there is a function terminate () which is called when a process exits for some reason (including crashes)

So for example one possible implementation of state is that you catch your own crash, save some state and then restore it when you restart

You aren't supposed to ever crash, however the whole language it's based around designing a framework to handle what happens if you do crash

So you design a whole framework of supervisors, watching processes, that all have a defined startup and shutdown order. You build structures with rules such as: if my cache module fails, them restart this whole chunk of application over here.

it's very cool

It doesn't. We read data from the db in the init function when the GenServer starts and make callers persist it before sending it to the GenServer. Our GenServers keep data in memory as the one in the post but we could read it from the db each time it ticks to process a record.

In this case, that's correct—you could lose up to 5s worth of state, plus any buffered messages. That's by design here. The business impact of that is going to depend on use-case, but I personally would find it difficult to believe that there would be any real impact in that event, in this case, under high load.

Obviously, different tolerances to that are going to necessitate different designs. You can configure the size of the message buffer when a GenServer starts, and if you have very low tolerance for lost data, you'd want to use a synchronous message (using call instead of cast, which blocks the sender until it returns) and appropriate error handling.

BEAM has a plethora of features for reliable applications, you just have to apply them appropriately.

I'd like a solid answer to this too; it's my biggest concern with any sort of "batch online requests" functionality in any language; unsafe shutdowns can/will always happen at some point, and it feels like this is always a data loss risk.

To some degree there's always data loss risk... A backhoe could cut a line, you could lose power to a PSU, rack, data center, a meteor could destroy the willamette valley, etc. What you want your system to provide you with is the framework to understand the risks and what the recovery looks like and when recovery is a lost cause.

Isn't it true of any, in-memory buffer implementation? It is very easy to add some form of recovery/persistence to a GenServer. You can have it write to a DB, ETS or disk. You lose some throughput obviously, it's all about balance.

Reply on news.ycombinator.com