Hacker News Reader

Comment by omeid2 | Hacker News Reader

I thought the idea of "exactly once" was a very questionable claim given what we know about computing, specially distributed.

Then somewhere else you see this gem.

> Workflows in flawless are written in Rust, in fact they are just regular Rust functions. This means that they can contain arbitrary logic. But instead of native code, the functions are compiled to WebAssembly and executed in a completely deterministic environment.

As much as I love Rust, this sounds like, here is a problem, let me throw fancy Rust and WebAssembly and that should fix it.

View on HN

The main page (https://flawless.dev/) has a diagram/video that shows how it would work and it basically writes any 'side effects' to a log which is used to track their existence. If they exist in the log, you read them; if they don't you run the code that generates them and then log it.

It's interesting, but I can't imagine how it is going to work at scale, both in terms of managing state across a very large application and in terms of running thousands of instances concurrently.

As the saying goes, "it is turtles all the way down". How do you ensure the log is "written" and "synced" _exactly once_?

Just write the action of writing a log to a different log file. Duh. /s

That can absolutely work... if you are confined into a single machine. (In fact, that's roughly how cloud instances work.) I see no way to generalize that into a distributed system. In my previous job I worked on a game server engine which solves a very limited version of this problem via virtual actors and it was still way too hard.

Exactly Once is basically a solved problem. The CAP theorem says you can either get consistency or availability in a system that has network partitions.

If you give up consistency, you end up corrupting data every once in a while.

If you don’t, then you can have exactly once, but it might take a long time if there’s a network failure.

NFSv3 solved this back in the 1980’s. (V2 and V1 may have, but I don’t know.)

It did it without requiring deterministic execution or other programming language innovations, so I share your skepticism about rust and web assembly solving these problems.

(I really like rust, and recommend it for pretty much all new systems code. However, it is not a panacea.)

This comment has some errors in it:

1. Exactly Once is absolutely not a solved problem. CAP says nothing about 'exactly once', it's about design choices for your data. But you can do all sorts of side-effectful things when it comes setting the data.

2. Giving up consistency does not mean you "corrupt your data every once in awhile". I don't know why you would say that. Choosing availability means you have to make decisions about how your data becomes consistent, but nothing about that means it is corrupted.

3. Choosing consistency says nothing about "exactly once". Consider this basic workflow backed by a consistent database: request comes into service, service sends email, services goes to store that email was sent in consistent database and crashes, user gets crash back, reruns request, email sent again. Oh so send the email AFTER it's stored in the database, well service crashes after saving in database, same problem.

If your system provides strong consistency, it is possible to build exactly once processing over it. NFSv3 is an existence proof, and there are plenty of theoretical results showing it is possible. Roughly speaking, you run the job and install the result in the consistent store iff your output register is null. If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notifications then you’re screwed, but the system in the article precludes that, as do many existing systems.

As for your other points, with eventual consistency you get weird problems like “I wrote X, then Y, but then read X and the system converged to X, except a week later I read Y, but just from half my fleet, and only for a 30 minute window”.

Unless you layer consistency on top of that (which is not always possible), then, tautologically, you don’t get consistency. In particular, most intuitive application-level invariants are going to be violated in all sorts of bizarre ways that take many pages to explain.

In practice, eventual consistency is closer to corrupting than not corrupting, because most application developers aren’t going to follow this conversation, and will use the database wrong.

I’m sure you or I could model our program and the storage in TLA+ and confirm we’re correctly maintaining application state, but that doesn’t help most developers.

Also, it’s not economically efficient.

We’re talking about “only” getting 6 nines instead of 7 because we chose CP instead of AP, but as a side benefit the system is easier to maintain, and it took less than 1/10th as much to implement, and has orders of magnitude fewer implementation bugs.

> If you side effect outside of the exactly once system, and the receiver can’t suppress duplicate notifications then you’re screwed

If you are getting duplicate notifications, then you don't have an "exactly once" system, but definition. Exactly once systems are not possible, bu your own explanation. You can store data in a durable way such that applying it multiple times is safe, but that is not "exactly once". Your options are at-least once or at-most once. Again, by your own statement: if you have duplicate notifications in your system, you do not have "exactly once".

> As for your other points, with eventual consistency you get weird problems like “I wrote X, then Y, but then read X and the system converged to X, except a week later I read Y, but just from half my fleet, and only for a 30 minute window”.

This is not data corruption, this is the semantics of an eventually consistent system. If you don't want those semantics, don't have an eventually consistent system, but that is entirely different than data corruption. Developers not understanding the system is different than corruption.

I agree that eventual consistent is probably not a good idea for most problems.

Many, many distributed systems provide exactly once semantics by giving up availability.

They don’t compose well with systems that don’t provide exactly once semantics, but that doesn’t somehow make them stop existing or stop working.

Also, you don’t need to arrange for data structures that can be modified multiple times safety.

You just need the exactly once system to support transactions, compare and swap, atomic rename, etc. Such things are usually pretty easy to retrofit.

For instance, Google and Microsoft S3 support conditional writes that check the etag for equality. That’s enough to let you layer consistency and exactly once on top. AWS S3 doesn’t support this, and people have been asking for it for a long time.

These things are extremely well understood and done at scale all the time.

Edit: I think they use etags for this. They might expose it as atomic rename or create if not exists instead. The expressive power of the two should be equivalent.

Exactly once is a mathematical impossibility. It has been proven. See the two generals problem. You can get similar end results if you have idempotent calls and allow duplicate calls to be made. This gives the illusion or impression of exactly once.

Reply on news.ycombinator.com