When "letting it crash" is not enough
flawless.dev
flawless.dev
- If there's a flaw in your application code that causes a crash (as is the motivating example in the essay), then restoring the entire program into the state it was in just before the crash happened would just cause it to crash again ad infinitum. Sure, this model helps against "my VM instance got preempted", but that's a pretty different category of "crash" (and also notably unrelated to supervision trees).
- "External" state (like an API endpoint being down/returning gibberish) can be part of the reason why your program got into a bad state. In fact, that's disproportionately likely, since external "weirdness" is comparatively hard to cover exhaustively in tests. In such a situation, the suggested computation model would never be able to recover even when restarted, because it would forever retain the (bad) API response. Effectively, this is just caching all the non-pure effects of your computation, and we all know about cache invalidation being a hard problem...
The hardest part was of course managing external state and journaling exactly where you had got to with external transaction APIs. Further backend reconciliation was available to flag this (and avoid Post Office scenarios).
Note that French NF525 almost mandates this design, at least for point-of-sale systems: every financial transaction has to be durably written for tax auditing purposes.
Durable execution is meant to complement your application. You will never want to model everything with it. It solves the problem of needing to decide how often to manually make snapshots of some important state, this becomes implicit. Workflows in flawless can still fail, you could call the `panic` function, or divide by zero. In the end it's arbitrary compute.
"External" state is one of the text book examples for using durable execution. If you are interacting with 5 different services and calling 5 different API endpoints, you sometimes want to have transactional behaviour. Leave all 5 systems in a consistent state after your interaction. You can't only call 2 and stop. Durable execution and patterns like saga [1] are one of the most straight forward ways (for me) to solve this.
In flawless specifically, I try to give enough context to the user why things failed. It's very easy to reconstruct the whole computation from the log. And let the user decide if they want to re-run the workflow. If you charge someone's credit card, but the call to extend their subscription fails (service down), you can't safely just re-run this. You have two choices, either you continue progressing and roll back the charge, or you fail and have someone manually look at it. In general, you want to use flawless in scenarios where the "called exactly once" guarantee is important. If you can just throw away the state and it's safe to re-run from the start, then you don't need flawless for this part of the app. The less state you have to care about, the better.
EDIT: The alternative would be to manually construct a state machine with a database. "Check if the credit card was charged. Call Stripe. I finished charging the credit card, save this information. Call the subscription service, it failed, restart everything. Check if the credit card was charged ...". And depending on your workflow, this can be a very complicated process where 90% of your code is just dealing with possible failures. Especially if failures happen on the edge of some calls it can become very tricky.
[1]: https://medium.com/cloud-native-daily/microservices-patterns...
- Application requests a JWT token. It then crashes and gets restarted. It gets past the problematic point, but later when trying to make a request, it crashes due to the cached token being expired.
- Application interacts with the current time in a meaningful manner. Due to the log replay, it will always live in the past and when switching from the cache-sourced time to the current time, some issues might occur, like deltas being larger than expected
- Application goes through a webshop checkout flow. After restart, some of the items in its cart have been already sold, but the app doesn't check this, since it already went through a (cached) check and got the result
Unfortunately, cloud provider APIs are usually eventually consistent, and getting a full snapshot at scale is nigh impossible.
So, in order to work around this, I effectively built a write ahead log style atop Postgres. Something like Sagas would have been great, but as far as I can tell, there was no real pattern for multiple Sagas operating on global state doing coordination. This is where Postgres SSI came in handy, where I could read the assumed state of the system, and if another worker came in and manipulated it, the write ahead entry wouldn’t get written as the txn would fail to commit. The write ahead entry would then get asynchronously processed by another worker, in case the first worker failed.
In that world, what you're generally looking at is
local state + incoming message -> new state + outgoing messages
Outgoing messages are sent only after persisting, and will be retried until success. Unique message IDs are used for idempotency (also for incoming messages).
Important: the actor runs with no side effects -- that's what makes rerunning things safe. In that kind of world, side effects are often achieved via e.g. sending a message that updates a materialized view somewhere (see e.g. Kafka).
With this setup, the source of badness is often isolated to the incoming message, and after a few failures an incoming message can be moved into a "dead letter queue" for ops to look at. In many scenarios, this actually works remarkably well.
https://pmatseykanets.github.io/beanstalkd-docs/protocol/#bu...
Hopefully!
But, sadly, this is not always the case. It's sad because it's incredibly hard to debug crashes that don't happen deterministically.
For those unaware: "There are only two hard things in computer science: cache invalidation and naming things." Plus variants like, "Oh, and off-by-one errors."
It's called checkpoint/restart, and was a feature of some early operating systems. Mostly for programs whose run time exceeded the mean time before failure of the system.
Tandem's whole system concept was built around that.
Amusingly, Second Life, of all things, has durable execution of the little LSL programs that make in-world objects go. They're checkpointed every minute or two, and if a region crashes, they are restarted, stack, heap, and all. They survive machine crashes and ports to new hardware. Even migration from a dedicated data center to AWS. Some have been running for well over a decade. Internally, they are Mono programs.
It's moderately amusing to see "save state somewhere" described as so new people haven't heard of it.
I am hard-pressed to think of anything in the software world today that is actually, truly, new. We all stand on the shoulders of giants.
The 1st level more-work-but-easier way to do this is to build or use a heavily constrained VM/language you run from within your main application that doesn't allow for most of the hard problems to even exist.
I can't find any ready-made tools to do this that I wouldn't consider an endeavor. Emacs has to be the most famous application to utilize dump/restore state but it's not exactly turnkey.
And before that, there were TeX and emacs, which do a memory dump and restore to save time loading their initial state. One of those, I think emacs, would tweak a core dump file to be a valid executable again.
Dumping the whole memory was a lot more viable when that was only a few kilobytes.
Checkpoint/restart has also long been a high-end capability found in HPC (High Performance Computing) systems. I recall it even it made it onto some commercial UNIXes, although I don't remember whether it was bundled in or took the form of some additional layered software such as LSF (Load Sharing Facility) (example: https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=admini... ).
There are some people who have tried to bring some checkpoint/restart to Linux. For example, see the CRIU project: https://criu.org/Checkpoint/Restore
I acknowledge that the original poster is talking about all this being done at the application level, not outside-the-app at the OS level...
For more references on the topic, see the wikipedia entry for Application checkpointing: https://en.wikipedia.org/wiki/Application_checkpointing
The entire point of crashing early is to reset the state of the program completely, hoping that the path that crashes it will not be taken too often. (and it only works well in the situations where crashing means negligible interruption of the service) By preserving the state between crashes, you diminish the pool of possible non-crashing states.
https://learn.microsoft.com/windows-hardware/drivers/debugge...
Some Googling found Checkpoint/Restore In Userspace, or CRIU. It’s like Flawless, but for Linux processes:
I bet that Flawless can make better guarantees about reliability due to constraints of the WebAssembly sandbox.
But the world is different for time travel debug, in that you can replay the whole history of the recorded execution (my understanding is that this is not required for Durable Execution) but you cannot resume the real process.
In fact, since it's used for capturing faults, you wouldn't generally want to resume execution - it's going to fail the same way (whereas a higher level restart framework might allow you to throw away some bad state and continue the computation - at the cost of not being able to precisely duplicate the bug).
[0] https://www.linux-kvm.org/images/d/d0/02x06b-DeterministicRe...
[0]: https://docs.docker.com/engine/reference/commandline/checkpo...
Does anyone here have experience with it? Maybe know the sort of problems RR excels at helping find?
EDIT: And is there some way to still use a better visual overlay a-la Visual Studio or QTCreator?
Like the article says, we would record a log of all the interactions with the outside world. The lower level parts of the system had the capability of transferring an in-progress call from one server to another. SIP and RTP traffic would be moved to another server, and finally the workflow log was sent. The server that took over the call would then replay the script and arrive at the same execution state that the first server had before the call was moved. Workflow authors didn't have to care about any of this. To them it looked like the call started and ended on the same server.
Then somewhere else you see this gem.
> Workflows in flawless are written in Rust, in fact they are just regular Rust functions. This means that they can contain arbitrary logic. But instead of native code, the functions are compiled to WebAssembly and executed in a completely deterministic environment.
As much as I love Rust, this sounds like, here is a problem, let me throw fancy Rust and WebAssembly and that should fix it.
It's interesting, but I can't imagine how it is going to work at scale, both in terms of managing state across a very large application and in terms of running thousands of instances concurrently.
If you give up consistency, you end up corrupting data every once in a while.
If you don’t, then you can have exactly once, but it might take a long time if there’s a network failure.
NFSv3 solved this back in the 1980’s. (V2 and V1 may have, but I don’t know.)
It did it without requiring deterministic execution or other programming language innovations, so I share your skepticism about rust and web assembly solving these problems.
(I really like rust, and recommend it for pretty much all new systems code. However, it is not a panacea.)
1. Exactly Once is absolutely not a solved problem. CAP says nothing about 'exactly once', it's about design choices for your data. But you can do all sorts of side-effectful things when it comes setting the data.
2. Giving up consistency does not mean you "corrupt your data every once in awhile". I don't know why you would say that. Choosing availability means you have to make decisions about how your data becomes consistent, but nothing about that means it is corrupted.
3. Choosing consistency says nothing about "exactly once". Consider this basic workflow backed by a consistent database: request comes into service, service sends email, services goes to store that email was sent in consistent database and crashes, user gets crash back, reruns request, email sent again. Oh so send the email AFTER it's stored in the database, well service crashes after saving in database, same problem.
As for your other points, with eventual consistency you get weird problems like “I wrote X, then Y, but then read X and the system converged to X, except a week later I read Y, but just from half my fleet, and only for a 30 minute window”.
Unless you layer consistency on top of that (which is not always possible), then, tautologically, you don’t get consistency. In particular, most intuitive application-level invariants are going to be violated in all sorts of bizarre ways that take many pages to explain.
In practice, eventual consistency is closer to corrupting than not corrupting, because most application developers aren’t going to follow this conversation, and will use the database wrong.
I’m sure you or I could model our program and the storage in TLA+ and confirm we’re correctly maintaining application state, but that doesn’t help most developers.
Also, it’s not economically efficient.
We’re talking about “only” getting 6 nines instead of 7 because we chose CP instead of AP, but as a side benefit the system is easier to maintain, and it took less than 1/10th as much to implement, and has orders of magnitude fewer implementation bugs.
If you are getting duplicate notifications, then you don't have an "exactly once" system, but definition. Exactly once systems are not possible, bu your own explanation. You can store data in a durable way such that applying it multiple times is safe, but that is not "exactly once". Your options are at-least once or at-most once. Again, by your own statement: if you have duplicate notifications in your system, you do not have "exactly once".
> As for your other points, with eventual consistency you get weird problems like “I wrote X, then Y, but then read X and the system converged to X, except a week later I read Y, but just from half my fleet, and only for a 30 minute window”.
This is not data corruption, this is the semantics of an eventually consistent system. If you don't want those semantics, don't have an eventually consistent system, but that is entirely different than data corruption. Developers not understanding the system is different than corruption.
I agree that eventual consistent is probably not a good idea for most problems.
They don’t compose well with systems that don’t provide exactly once semantics, but that doesn’t somehow make them stop existing or stop working.
Also, you don’t need to arrange for data structures that can be modified multiple times safety.
You just need the exactly once system to support transactions, compare and swap, atomic rename, etc. Such things are usually pretty easy to retrofit.
For instance, Google and Microsoft S3 support conditional writes that check the etag for equality. That’s enough to let you layer consistency and exactly once on top. AWS S3 doesn’t support this, and people have been asking for it for a long time.
These things are extremely well understood and done at scale all the time.
Edit: I think they use etags for this. They might expose it as atomic rename or create if not exists instead. The expressive power of the two should be equivalent.
Some feedback: this paragraph lands like a Reddit post from a teenager, who just took a big drag off a joint, and thinks they've invented a new field of science. (And also hasn't completed it so isn't ready to share it yet!)
Of course, everything's already been thought of. What you've described so far as durable computing sounds indistinguishable from event sourcing.
The code snippet is intriguing, though. We're abstracting event sourcing away from the developer---it happens automatically for every non-purely-functional operation---so the programmer can just write normal code.
Though, you'd have a lot of events, and I'd wonder about performance and scalability for larger apps.
Also, assuming events are at the finest level of granularity, I also imagine this starts to look like the equivalent of some kind of intermittent heap dump.
I'm wondering if it has the same scalability concerns - sticking everything in Postgres is fine at small-ish scale, but what happens when you outgrow Postgres, either because you have higher availability requirements (can't handle primary DB restarts) or because of the sheer volume of the workload?
https://temporal.io/blog/time-travel-debugging-production-co...
Anyways I wish all the best for flawless as this problem is worth solving/popularizing.
(I work at Temporal)
Temporal may have properties that make it not a good fit for a particular use case, but scalability is really not one.
But I don't exactly understand: If you exactly recover the same state, don't you end up with the same faulty undesirable state? (That's what you also get with RR, by intention, to debug the problem.)
Clearly, here this is not the intention, i.e. you don't want to recover exactly the same state. So, it means, some parts will not be recovered. How would it be decided what parts to recover and what parts not? This seems like an impossible problem to solve in general.
And then, the state will not be exactly the same. It might be now in a sane state, but is it the right state that the user wants?
But this would cover only some specific set up failures, namely where something goes wrong in the host, which you can easily recover by resetting its state and retrying again. This is only a very limited amount of failures. I guess most failures come from bugs in the user code, which would all be state within the VM.
Also, maybe your app depends on some resources from network, e.g. some NFS, or maybe some other remote server, or whatever, which you anyway cannot control. This is not easily recoverable then. (Tools like https://criu.org/Main_Page, which try to serialize an app state, to be able to recover it later, have the same problem.)
We were dealing with such code in a TCB and the only way is to audit your code and ensure that any action that can only be done once at most saves that action before doing the action.
I think you have the same concepts in queues no? Some queues will repeat at most once, some at least once, something like that. I can’t remember.
Exactly once execution is one of the guarantees I want to provide. Flawless should always give you peace of mind, and in cases it can't guarantee exactly once execution it will sacrifice progress. When interacting with the real world, hardware and software can fail in creative ways and it's not always possible to automatically recover without manual intervention. Sometimes it's just not possible to solve issues without looping in a human. But flawless will not repeat a call if it can't guarantee it was not performed or if it's not idempotent.
For many features, like HTTP requests, flawless uses a dual commit system. In cases where data was sent to an external server, but no response is received (timed out), we can't know what the external system observed, and will not allow progress to continue (fail the workflow). You can relax this requirement by marking the HTTP call as idempotent.
I discovered an interesting simple case of this on my utility's websites. Autopay for some reason didn't trigger. It showed I had autopay enabled, but no payment ever made.
Tried to pay manually, got "error: autopay enabled, no one-time payment allowed". The only way to pay my bill was to delete my autopay, and manually pay.
I now cannot re-enable autopay getting the error "autopay disabled for this account" (possibly because it had the past-due balance I had just paid?).
Crossing my fingers if I log back in next week all the states will reconcile themselves. Software!
It’s not a real error handling solution. I only use it for my quick and dirty projects.
Many errors are recoverable but the ones that are not fuck up this scheme because a human needed to have written a proper error recovery scheme that isn’t just “retry over and over from a previous point.”
At the end of the day, you gotta put in manual work for error handling. You can’t rely on your language or any library to create a generic error handler.
1. A tech stack that isolates crashes such that they do not affect the rest of the system. Example: you receive a malformed API response. The code responsible for parsing it crashes, but the rest of your application does not.
2. You use this to code in a declarative style.
Coding declaratively means your code has the shape of the data you expect to have.
Example (in elixir):
Let's say you are calling an external API where you expect to get back a list with a single element.
If you code non decleratively, your code might look like this:
thing_i_want = List.first(my_api_response)
Coding declaratively you would write:
[thing_i_want] = my_api_response
The alternative is to code defensively: check the lenght of the list before extracting the first/only element. That works, but it tends to be very verbose and more importantly it means the entire call stack needs to be aware that you might get exceptions bubbled up from anywhwere.
Of course, for this to work you need software that has lightweight processes and a supervisor architecture..not usually worth it unless you're getting it for free! (as with elixir/erlang).
On my laptop, RabbitMQ would get into a state where it was unavailable and taking 100% CPU. I think it happened when I switched networks (moved laptop home) and the spinning was from repeatedly crashing. I never sorted the root cause, just wrote a script to kill and restart rabbitMQ. (The script to kill rabbit was named 'fudd'.)
As it is, the animation crashes at the most opportune moment possible: after a side-effect statement has been persisted to the log, and even before the subsequent statement executes.
Where I want to see that animation crash is in the middle of that "HTTP request" at the bottom, where that request is an HTTP POST, and where the crash is "the server successfully received the request, and processed it, but we crashed before we could get the response."
The resuming execution's log is either "empty" … or "we sent it, but never got the result" — the former has you repeating the side-effect, and the in latter, you can at least sort of tell you're doomed.
[0]: https://en.wikipedia.org/wiki/Three-address_code
[1]: https://temporal.io/
[0]: https://learn.microsoft.com/en-us/azure/azure-functions/dura...
Now the problem is that every time they fail, I need to restart it. But not from the beginning, but where it failed. So I comment out everything before the failure and rerun.
It would be nice to use durable execution for this.
For instance if you have a system with a user interface and some various components, like say, a database, and the database becomes unavailable, you don't want the entire system to crash. You want it to display an error message to the user and maybe go into some kind of diagnostic mode or other "things are not normal" state.
Something like https://github.com/jlouis/fuse is one approach. I had a small stab at creating something similar: https://github.com/davidw/hardcore
In a distributed system you always need to handle the case where you sent a request and didn't get a response and the request may or may not have been processed. Once you accept that, it's totally reasonable for the request processor to crash during a request, because the requester always needs to be know what to do in case they get no response or a non-specific error.
Maybe you'd want a wrapper around certain sections to say: only only log the output of this group of statements.
I thought we're all supposed to program for "the browser" (in reality Chrome) to reduce development costs.
I stopped reading somewhere in the presentation of flawless. Ran out of care for the moment. Might revisit it though. Sounds interesting.