yazaddaruvala

I do care about being intellectually honest. I haven’t once said that you or anyone else in this thread is wrong. If you really read everything Ive written, you’ll see I’ve always offered multiple perspectives.

In my words, your perspective has always been accurately shared. I care about that.

Meanwhile, our differences are philosophical. I’m not interested in your limited perspective. Just like you’re not interested in my broader perspective.

I’m not asking you to believe or say anything. You don’t like the additional perspective I’ve shared, ok, move on.

I can also appreciate academics need a time and place to be academic about their definitions (e.g. papers, and in-depth answers).

> What are some real world apps using CRDTs that have really good experiences?

I also feel, academics could do more to appreciate questions like this one are non-academic. The question above could just as easily be interpreted by many as "I'm building a multi-player application. I want it to be a really awesome experience but I don't know if I'll get the concurrency right. I've heard concurrency is hard. I've heard good things about CRDTs tho, should I be using them?".

To which we should absolutely be mentioning truly pure CRDT based applications/frameworks, but not mentioning similar but nuancedly different applications/frameworks, hurts 1. any less educated readers' ability to instinctively understand what a CRDT is; and 2. how it might help them create the next great multi-player application. It also hurts someone that has spent the last N nights trying desperately to make their pure CRDT application/framework fun and valuable but its just not happening. Allowing them to loosen up and be a bit more creative, helping to empower their application/framework because the universe of "asymptotically approaching a CRDT; but will definitely not have concurrency bugs/issues" (which doesn't have a nice academic name + explicit ruleset yet) is so much larger "than strictly a CRDT".

Meanwhile, I could if this was a paper or lecture have said "... are examples asymptotically approaching a CRDT; So don't really worry just because something is not strictly a CRDT doesn't mean you're gonna have bugs in your concurrency implementation". But "in passing" on a comment thread that should have been 30 seconds of all our time, who wants to specifically write all of that nuance every comment? In fairness I even tried to differentiate academic vs non-academic.

If you're still reading this thread (especially anyone that has been downvoting me) I hope you can appreciate this context.

Yes I do. In general, the convenience is too nice.

If you're specifically worried about health issues:

- The Pharmacy features are pretty locked down (but nothing is perfect). I wouldn't buy pills from a rando 3p seller, or even "Sold by Amazon" tho.

- For other health related products (like creams, kitchen stuff, clothes or other things that are on me), I just check that the Brand is the listed seller and no other sellers exist on the product.

- For non-health related products (shower curtains, garden hose, etc) I don't really care if it is "not perfect" in terms of quality, etc.

Meanwhile, if you're buying products where the Brand=ZFDSC or any other character salad - you should expect whatever you get. I still have purchased them (e.g. a sticker that makes my windows look like stained glass - I don't care that its cheap garbage from China - it looks good and unlikely to be bad enough to kill me at that distance), but certainly would not buy those Brands' expecting lasting quality.

I would say Not enough for how many engineers they had. It had all of the promise, and was frankly mismanaged.

They kept promoting engineers who wanted it to become an AWS service (agnostic of Amazon) rather than solving the Amazon problems at hand.

For example they built a new product identifier… Amazon did not need yet another database of products. This stuff was constantly slowing them down to less than a crawl.

Last I heard all of that was canceled, and their focus brought back to Amazon Retail only problems. But tech debt and culture issues like that are really hard to unwind.

FWIW, while "counterfeit", technically Amazon doesn't consider commingling to be Brand related abuse which is what I worked on. Abuse of commingling would be handled by other systems related to Seller fraud.

"The only case Automerge cannot handle automatically, because there is no well-defined resolution, is when users concurrently update the same property in the same object (or, similarly, the same index in the same list). In this case, Automerge picks one of the concurrently written values as the "winner", and it ensures that this winner is the same on all nodes"

"The object returned by getConflicts contains the conflicting values, both the "winner" and any "losers". You might use the information in the conflicts object to show the conflict in the user interface."

To me, this seems identical to how git works. Specifically git fetch (does automatic resolution storing all of the changes), vs git merge (showing the conflicts in the user interface).

Disclaimer: I haven't worked at Amazon for 3 years - my info is likely stale.

So then Git uses a CRDT internally? Merge conflicts are user experience "semantic conflicts" within the Git application.

The internal state in the Git commit tree contain all commits consistent across the synced replicas. Without further inputs, all the repos would render the exact same user experience conflict.

"The data type i.e. Git's internal state is conflict free - but there are times the user needs to correct the semantic state."

"The Git application wants to guarantee semantic state, so unless the semantic conflicts are resolved, the Git internal state i.e. 'data type' will continue to be conflict-free using the 'rollback technique' to reconcile conflicts with the local copy's 'data type'."

Roughly: git fetch would be the conflict-free update of the internal data type. While git merge would be the application level, semantic conflict resolution.

> A transaction in a blockchain isn't generally considered to have happened until it’s been included in a block & witnessed throughout the network.

For the sake of argument:

- It’s not meaningful for Apple Notes’ local edits to only be local. Sure they will render, but if a tree falls and no one hears is, did it make a sound? So let’s ignore local edits.

- A sentence in Apple Notes isn’t generally considered to have happened until it’s been rendered in every client and users can see it if they look.

Is that any less fair of a statement?? Is this game of nuanced wording selection at all meaningful tho to answer the question “Is Apple Notes a popular application of CRDTs?”

Meanwhile, keep in mind in a blockchain entire sets of nodes could be firewalled from each other for days, each subsection performing consensus and creating blocks. Days later the firewall could collapse and the nodes will need to automatically reconcile. If I abstract away the internal replicas and say the two “offline parts” of the CRDT need to keep “local” state and reconcile once both/all parts come back online. Would this abstraction help meet the strict definition?

I know you probably still strongly disagree - but coming from a non-academic background these word games around CRDTs hurt rather than help people’s understanding and use of them.

Instead we should be asking, how can we improve Git as a CRDT (where perfect "conflict-free" is the enemy of good enough) to better automerge more conflicts? How can we improve the general theory of CRDTs based on what we learned from implementations in “video game A”, “blockchain Z”, and “web application framework J”?

Meanwhile, a lot of people on these threads would say Automerge is not a CRDT. By definition Automerge.getConflicts means Automerge has conflicts and is not conflict-free.

I on the other hand prefer and applaud Automerge for breaking the rules! I hope it continues to see more success.

Any and all networked video games with some form of rollback or correction. Best effort with a fallback to Rollback might actually be the “best” ie ergonomic experience for CRDTs that are most widely used.

Again, not academically a CRDT because technically game state is not perfectly replicated to every client. Each client only gets partial game state.

Additionally, game clients require low latency syncing, which could academically be considered “coordinating”. Even tho the client actually accepts and renders the input’s results locally probabilistically before any conflict resolution / rollback is returned to the client for correction.

Again people are likely going to be pedantic but with three post now, I’d like to hope, people might see the common theme:

The most popular, highly ergonomic, best implementations of CRDTs actually break the academic rules of CRDTs.

This is a relatively typical trap of an overly academic mental model. Most real world algorithms and data types are actually more creative than their academic “rulesets”. eg Timsort.

Especially if you’re building a product for actual use (as apposed to for review in a paper), then don’t fall into the over engineered/academic trap. Be creative, learn the academic rules, then intentionally break those rules, build what actually adds value and make it ergonomic rather than try to perfectly implement a concept that academics defined so stringently it’s only useful for other academics.

Since people loved my other hot take :)

I suppose "good experience" can be subjective, but Blockchains are also in practice a distributed trust-less CRDT that is used by a LOT of people.

- With redundant validation, across more than 2 nodes, used to solve for conflict resolution.

- and proof of work/stake used to solve for distributed trust.

Again, academically, maybe not a CRDT because they will not "always converge". Hard forks can happen. However, pragmatically, Blockchains are a CRDT.

Git could be "conflict-free" with a simple `rand(ours, theirs)`.

It would be useless, but technically "conflict-free". Is the addition or removal of that rand function really, pragmatically the difference in the answer to "what is a CRDT?"

> 2. An algorithm (itself part of the data type) automatically resolves any inconsistencies that might occur.

> 3. Although replicas may have different state at any particular point in time, they are guaranteed to eventually converge.

[0]

Again, in theory it fails 2 and 3. However, in practice 3 is a normal part of working with git in a team. Barring a hard fork in git - which is equivalent to a deep copy of a CRDT. Like any deep copy of a data type, a CRDT's deep copy can be used in non-conformant manners (forks are VCS specific jargon for a CRDT deep copy; or shallow copy sometimes).

> If I commit A and then B but someone’s clone only has B applied, you can’t synchronize even automatically; different git histories don’t resolve in any way automatically

Maybe I don't understand your point specifically, but this example seems entirely solved by --rebase. In practice --rebase is typical, and best described as "do your best to automatically resolve histories; I'll handle any of the complex conflicts".

All that said, I already agreed: "academically Git is not a CRDT". However, and I'm happy to disagree with you, in practice Git is the most popular CRDT.

[0] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

Some CRDT purists would say "its not perfectly conflict free so its not a CRDT".

Sure[0], but for the rest of us that are pragmatic about best effort conflict resolution Git is likely the most successful CRDT application.

[0] https://en.wikipedia.org/wiki/No_true_Scotsman

I engraved my entire phone number, including the +1, and I’ve had my AirPod returned to me twice now.

Just add it to my Instagram timeline and I can skip the trip and the cost. Everyone else (including me in 30 years) thinks I went.

Silly and sad, but real :(

I find especially all their slight variations to be quite poetic.

“___ is a pedo” ?

Like, the guy has done good shit (better than most), but he has certainly set his reputation on fire with many of his (at best “inaccurate”) tweets. At that point you gotta caveat praise lol.

You can also look into “realtime data warehousing”.

Meanwhile, it’s so common, AWS built it into their product.

https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-dy...

It seems they need strong consistency for certain CUJs and then a lot of data warehousing for historical transactions.

It’s strange to me that they didn’t first convert their 2 table DynamoDB architecture into DynamoDB and Redshift architecture or similar. This is a pretty common pattern.

In reality, there might be a 3-5x reduction to brake pedal usage. That still means on any given trip, every 5th stop sign/red light will require break pedal usage.

If anything the single pedal driving causes me to let off of the accelerator earlier than I normally would in an ICE car, and therefore I have more time to react with the break pedal if needed.

I doubt I'm unique in these things.

The "Fortnite creative engine" is roughly a Virtual Worlds editor / generator along side the Fortnite Virtual Worlds marketplace.

It was (and sometimes is) really bad!!

It took 4+ months of therapy to get back to "manageable", and over 2 years later I felt back to normal and in control. I've now stopped any intoxicants (alcohol included) almost entirely. I sometimes will participate with a good group of friends, but I'm far more cautious of any intoxicant.

Everything else about “why can’t I mix these functions” is a direct result of that broken consistency exposed by the OS. As such I’m not sure there is a zero-cost abstraction (stackful coroutines is close) that any programming language can build to improve the situation.

Meanwhile, the trend is clear - every OS has adopted (and is adopting more) non-blocking syscalls because they are direly needed. The only benefit blocking syscalls offer is sugar to improve syscall ergonomics.

I think if more people talked about it this way it would become clear that adding a “blocking” tag to the syscalls and bubble that tag up the call stack is the right next step to depreciating those legacy OS APIs. I don’t mean to say we should accept poor ergonomics but adding the blocking tag is a great first step to 1. Reminding people of the problem. 2. Identifying areas where research is needed to improve ergonomics and replace the existing “blocking” tag with minimal downsides.

On multiple teams of FAANG engineers, Ive seen the GC be a big part of some meltdown and none of us knew a good way around it. Tuning only goes so far. Otherwise, you just have to give it more memory and CPUs (horizontally or vertically).

Almost always the answer was, this might need to be reworked in C++ entirely or JNI. In same cases it even got funded and was a tremendous success (like 10x fewer resources). However, then you’re dealing with C++ and all of its issues.

I’m looking forward to more value types in Java but managing the heap will always be a bottleneck eventually.