aphyr

Thank you!

To build on this, Datomic includes a pre-commit conflict check that would prevent this particular example from committing at all: it detects that there are two incompatible assertions for the same entity/attribute pair, and rejects the transaction. We think this conflict check likely prevents many users from actually hitting this issue in production.

The issue we discuss in the report only occurs when the transaction expands to non-conflicting datoms--for instance:

[Stu favorite-number 41]

[Stu hates-all-numbers-and-has-no-favorite true]

These entity/attribute pairs are disjoint, so the conflict checker allows the transaction to commit, producing a record which is in a logically inconsistent state!

On the documentation front--Datomic users could be forgiven for thinking of the elements of transactions as "operations", since Datomic's docs called them both "operations" and "statements". ;-)

Yeah, this basically boils down to "a potential pitfall, but consistent with documentation, and working as designed". Whether this actually matters depends on whether users are writing transaction functions which are intended to preserve some invariant, but would only do so if executed sequentially, rather than concurrently.

Datomic's position (and Datomic, please chime in here!) is that users simply do not write transaction functions like this very often. This is defensible: the docs did explicitly state that transaction functions observe the start-of-transaction state, not one another! On the other hand, there was also language in the docs that suggested transaction functions could be used to preserve invariants: "[txn fns] can atomically analyze and transform database values. You can use them to ensure atomic read-modify-update processing, and integrity constraints...". That language, combined with the fact that basically every other Serializable DB uses sequential intra-transaction semantics, is why I devoted so much attention to this issue in the report.

It's a complex question and I don't have a clear-cut answer! I'd love to hear what the general DB community and Datomic users in particular make of these semantics.

Yeah, of course :-)

You'll find many of those additional variants (e.g. strong session SI) in the linked papers, and they're cited extensively in the Jepsen reports as well. I just haven't had time to write up every single model--tried to stick to the major ones. :-)

Same. Hi, Apple! :D

I'd love to do more work with predicates in general. That's an open research problem I've been noodling on for years. Pretty much any SQL DB would be a good candidate for that work!

I'm gonna be a weirdo and say I actually loved Fauna's FQL. A little Lisp-ish functional language for queries is a great way to interact with document-structured data. SQL is fantastic for sheer breadth, though its specification is a nightmare and actually writing portable SQL is real challenging. One of those places where a stronger spec and conformance tests would have really helped.

It's not a huge market. I usually have a queue of clients, but both deal flow and actual scheduling are wildly variable--sometimes I'll go without income for six months or more. I've considered hiring one or two people, but I couldn't offer the kind of stability people need from an employer. I do subcontract though! Editing, legal, finance, occasional code, that sort of thing.

There's a lot of folks out there who can do basic testing work with Jepsen. I've taught... I dunno, maybe a few hundred people directly in Jepsen workshops. A couple people have worked alongside me, and I'm sure lots more have learned from the docs online. Writing a report is a more involved problem--certainly not intractable, but for me it involves testing, experiment design, lots of reading, doc review, writing, editing, finding reviewers, and of course all the business stuff.

Even an unpaid report like this involves weeks of work: reading docs, designing tests, executing and refining them, filing issues, writing the report, and editing it. Each report costs thousands of dollars in hardware and and editing. I'd love to do an FDB test! It's just that between contracting, volunteer work, and research, I can only do so much.

The point of isolation levels is that the database manages those locks (or other concurrency control mechanisms) for you.

This depends on what you mean by "atomic". Prior to 5.0, MongoDB's defaults were to use a sub-majority write concern. This allowed all kinds of interesting atomicity violations, even on single documents. For instance, you could write a value, some clients might read it, and then your write would be silently lost as if it never happened. Clients could disagree on whether the write happened or not. A single client could observe, then un-observe the write. It gets weird. :-)

The FUSE stuff actually isn't too bad--Jepsen goes to a lot of trouble to make all this stuff automatic. The test harness pulls dependencies, compiles LazyFS, and mounts the filesystem for you. Just pass `--lazyfs` at the CLI. :-)

Basically all of it. MariaDB exhibits every class of anomaly we found in MySQL, all in single-node deployments.

I, uh, do want to point out that the alternative here is not "everything is OK". If you don't abort when, say, two users update the same row concurrently, then you might cause (e.g.) silent data loss for one of them. Or you might end up with a record in an illegal state--say, one with two different fields that should never be in their particular states together. You have to look at your transaction structure, intended application invariants, and measured frequency of concurrency to figure out if using a relaxed isolation level is actually safe or not.

Aw shucks, thanks :-)

That depends! As the article discusses, snapshot allows anomalies--like write skew--which might violate application invariants. Depends on your workload.

Yup! See https://jepsen.io/analyses/mysql-8.0.34#list-append, which also has a link to the code: https://github.com/jepsen-io/mysql/blob/4c239cb5c66a7f1a55fa...

This isn't CONCAT-specific, BTW--we just use CONCAT because it allows us to infer anomalies in linear, rather than exponential time. Same kinds of behaviors manifest with plain old read/write registers.

Necessary depends on your use case! I spend a lot of time waiting on analyses, and the more operations I can test, the more bugs I can find. I probably invest more time in performance optimization than most people.

Regarding your specific example, uh, I don't know how you measured, but that feels... off. Here:

``` package scratch;

public class Sum { public static long sum(long[] longs) { long sum = 0; for (int i = 0; i < longs.length; i++) { sum += longs[i]; } return sum; } } ```

``` (require '[criterium.core :refer [quick-bench bench]]) (def longs (long-array (range 100000000))) (defn sum-clojure [size] (reduce + 0 (range 0 size))) (import 'scratch.Sum) ```

Summing an array of 10^8 longs like you suggested takes ~120 ms on my machine.

``` user=> (quick-bench (Sum/sum longs)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 119.804166 ms Execution time std-deviation : 9.124709 ms Execution time lower quantile : 115.154905 ms ( 2.5%) Execution time upper quantile : 135.597497 ms (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 15.4197 % Variance is moderately inflated by outliers ```

Your Clojure function, which sums a lazy range, takes ~4.7 seconds--about 40x slower.

``` user=> (quick-bench (sum-clojure 100000000)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 4.739575 sec Execution time std-deviation : 346.895187 ms Execution time lower quantile : 4.557732 sec ( 2.5%) Execution time upper quantile : 5.324739 sec (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 15.3163 % Variance is moderately inflated by outliers ```

An idiomatic Clojure reduction over the same array of longs, just so we're measuring apples to apples, takes about 12 seconds--about 100x slower.

``` user=> (reduce + longs) 4999999950000000 user=> (quick-bench (reduce + longs)) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 12.037821 sec Execution time std-deviation : 433.661044 ms Execution time lower quantile : 11.790637 sec ( 2.5%) Execution time upper quantile : 12.779262 sec (97.5%) Overhead used : 20.384654 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 13.8889 % Variance is moderately inflated by outliers ```

Incidentally, this is one of the reasons I wrote `loopr` (https://aphyr.com/posts/360-loopr-a-loop-reduction-macro-for...). One of the iteration tactics it can compile to is iteration over arrays. Nowhere near Java--I think it's probably still boxing a fair bit, and I'd need to disassemble it to see--but it's still 10x faster than the standard reduce here. ~10x slower than Java.

``` user=> (quick-bench (loopr [sum 0] [x ^"[J" longs :via :array] (recur (+ sum x)))) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 1.259095 sec Execution time std-deviation : 3.876650 ms Execution time lower quantile : 1.256692 sec ( 2.5%) Execution time upper quantile : 1.265747 sec (97.5%) Overhead used : 20.321072 ns

Found 1 outliers in 6 samples (16.6667 %) low-severe 1 (16.6667 %) Variance from outliers : 13.8889 % Variance is moderately inflated by outliers ```

I maintain dependencies for a lot of people--nowhere near operating systems or stdlibs, but my OSS gets fairly broad use--and I try very hard not to change their APIs too much! It's a real concern.

Nope! For a few reasons Jepsen sticks very closely to Traditional JVM Threads.

Part of that is because Jepsen isn't hyper-concurrent. Most concurrency errors manifest in just a handful of client threads, and the JVM will run ~10K threads without too many issues.

Another part is that I spend a lot of time debugging and profiling. Core.async turns stacktraces into Swiss cheese--it has to, to perform its implicit continuation-passing magic. But that can make it really hard to figure out Who Called What When, and Why Is This Process Slow. Standard blocking IO calls are also something that my profiler (YourKit) can analyze well.

Finally, I invest a lot of time into tuning Jepsen for speed, and manage the handoff of data between threads carefully. In a system where actors were handing off state constantly core.async would be a bigger help, but there's really only a couple shapes for concurrent handoff in Jepsen, and they're addressed by either j.u.c. queues, forkjoin-style structured concurrency, or Weird Transactional Threadpool stuff.

For a little more on this, take a look at https://github.com/aphyr/dom-top or https://github.com/jepsen-io/history/blob/main/src/jepsen/hi...

Tbh I think they've actually gotten worse in recent years. This thing where the compiler spits out a stacktrace to /tmp and makes you go and cat it? Absolutely awful.

And the move to spec, ugh, UGH. I have wasted so much of my life looking at spec errors for (e.g.) namespace declarations and being completely unable to tell what was wrong. I don't know how to fix this, unfortunately.

Speaking very loosely, primitives on the JVM are values which are represented directly in memory, instead of as pointers to objects on the heap. Clojure (again, very loosely) generally treats everything as a pointer to a heap object. There is no specialized equivalent for, say, a vector of shorts, or a map where values are floats. The compiler can emit specialized function signatures for... IIRC longs and doubles, but other types (e.g. byte, float) aren't directly accessible--they go through widening conversion. It's also easy for the compiler to quietly fail to recognize it can preserve primitives in some kinds of loops, so you wind up with what Java calls "autoboxing": wrapping a primitive in a corresponding Object type.

Here's a recent example of some code in a hot path inside Elle, one of Jepsen's safety checkers. It does a lot in primitives, using packed structures and bitmasks to avoid pointer chasing.

https://github.com/jepsen-io/elle/blob/main/src/elle/BFSPath...

There was actually a Clojure version of this earlier that got pretty close perf-wise, but I wound up dropping to Java for it instead:

https://github.com/jepsen-io/elle/blob/913cbff5ebb19ba850c0a...

There's a happy medium, I think, and I also imagine that depends on scope of change and the kind of applications involved. Speaking personally, I'm OK rewriting code, say, every decade. In the Ruby world, things moved so fast that my code would rot every six months. Now that I've been maintaining the same software for 20 years I'm really starting to care about that. ;-)

Clojure's Java interop story is generally really good. Method calls are predictable and uniform. The collection types generally work both ways, rather than being a completely independent type hierarchy (as a former Scala programmer, what a breath of fresh air). Functions work as Runnable, Callable, Comparator, etc. Reify, deftype, and proxy are sufficient for maybe 98% of APIs I've worked with.

There are, however, those rare exceptions! For instance, some Java APIs literally won't work without annotations. There's just no way (last I checked) to express those in Clojure.

I don't remember exactly the nature of this problem, but I've had occasional issues with deeply overloaded argument dispatch before. Maybe it was something like "The Java API expects an object x which responds to x.foo(long l) and also x.foo(int i) differently", and there was no way to convince deftype to emit the right kind of class.

AOT is... it's weird. You want ahead-of-time compilation when, say, you want to write a Java class that uses datatypes defined in Clojure. But AOT, at least in Leiningen, tends to break stuff in truly inscrutable ways. My professional opinion is that it's haunted.

Clojure has a gradual type system--core.typed--that I've used extensively. It's... it's very cool, but for my uses it seems to be more trouble than it's worth. I keep coming back to it every few years, though.

DB vendors generally don't write Haskell clients, which put a serious damper on my "test lots of databases for a living" project. Love Haskell as a language, but it's just not a popular target, and I wasn't brave enough to go for C interop. As I touched on in the post, Haskell is also rigorous about side effects--I've found it difficult to, say, inject side-effectual performance monitoring into an otherwise pure piece of code so I can profile it. Maybe Haskell experts are better at this, or the story is better now.

There was a Ruby version of Jepsen too. That enjoyed better library support and language flexibility, but it suffered from terrible performance, a standard library built for mutability, and a janky thread model. There was, at the time, essentially no way to reliably time out a Ruby thread. That was a dealbreaker all in itself.

I wrote some of Jepsen in Erlang too, but abandoned the project quickly. It's got a really good concurrency story. A VM designed for concurrency and failure with bounded reduction queues--honestly this is something I miss in the JVM. I love Erlang as a language (and I think some of my Erlang might still be in prod!) but it also suffered from a lack of client support, not being well-suited for high-speed mutable state, and generally terrible string manipulation. There's a lot of strings in Jepsen. ;-)

I've been a Java programmer for ~18 years! A nontrivial portion of Jepsen is written in Java too. :-)

Many of the points I touched on in the post are direct critiques of Java, but maybe I should be explicit. An obvious factor is size--when I've written the same project in both languages, Clojure usually winds up 5-10 times smaller. That's definitely improving as Java adds streams, lambda expressions, and so on, but it's still a good deal more verbose. That gets in the way of the kind of exploratory programming I do, especially at the REPL, and it's harder for me to read and maintain a giant codebase. I appreciate having fewer nouns, in general: Clojure emphasizes a uniform way of traversing and transforming data structures, and in Java every single class has its own way of representing its data, and many of those ways are bad.

The standard collections library is full of mutability and thread safety pitfalls. Printed representations of datatypes are verbose. It's not particularly good at dealing with highly polymorphic data structure transformation, and it's not a particularly good static type system either--a good part of my Java Brain (TM) is devoted to knowing the mechanisms of erasure, memory layout, and when unsafe casts are actually the right choice. Functions are sort of becoming first-class with method refs and functional interfaces, but it's nowhere near the convenience and flexibility of a Lisp-1. Accessing the compiler at runtime is a pain in the ass. No hygenic macro system makes things like custom iteration or compositional error handling a pain. There's no analogue to Clojure's protocols, which are a fantastic tool for "Hey, compiler, I want a monomorphic-cached type-dispatch polymorphic call site here for a type hierarchy I don't control". Etc etc.

There are things I love about Java. Automated refactoring is easier, and I generally like the level of IDE support. Nearly anything involving primitives, I drop to Java. Ditto, APIs that require annotations. Interface specification is more rigorous. Etc etc. That's why I write in both languages. But most of Jepsen is in Clojure for good reasons. :-)

Aphyr.com is built with Luminus. :-)

Maelstrom is a frontend to Jepsen, a Clojure library for testing concurrent systems--mostly databases. Jepsen is also written in Clojure and Java--both run on the JVM.

Because it tests concurrent systems, the language itself needs good support for concurrency. Clojure's persistent data structures make it easier to write correct concurrent programs, and the language and runtime have excellent concurrency support. I also considered languages (like Haskell) with more rigorous control over side effects, but decided that Clojure's less-dogmatic approach was preferable.

Because Jepsen tests databases, it needs broad client support. Almost every database has a JVM client, and Clojure has decent Java interop.

Because testing is experimental work, I needed a language which was concise, adaptable, and well-suited to prototyping. Clojure's concision and syntactic flexibility work well for that.

Tests involve representing, transforming, and inspecting complex, nested data structures. Clojure's data structures and standard library functions are possibly the best I've ever seen. I also print a lot of structures to the console and files: Clojure's data syntax (EDN) is fantastic for this.

Because tests involve manipulating a decent, but not huge, chunk of data, I needed a language with "good enough" performance. Clojure's certainly not the fastest language out there, but idiomatic Clojure is usually within an order of magnitude or two of Java, and I can shave off the difference where critical. The JVM has excellent profiling tools, and these work well with Clojure.

Jepsen's (gosh) about a decade old now: I wanted a language with a mature core and emphasis on stability. Clojure is remarkably stable, both in terms of JVM target and the language itself. Libraries don't "rot" anywhere near as quickly as in Scala or Ruby.

Clojure does have significant drawbacks. It has a small engineering community and no (broadly-accepted, successful) static typing system. Both of these would constrain a large team, but Jepsen's maintained and used by only 1-3 people at a time. Working with JVM primitives can be frustrating without dropping to Java; I do this on occasion. Some aspects of the polymorphism system are lacking, but these can be worked around with libraries. The error messages are terrible. I have no apologetics for this. ;-)

I prototyped Jepsen in a few different languages before settling on Clojure. A decade in, I think it was a pretty good tradeoff.

I agree--rystsov has covered this well here and in other parts of the thread. I just want to add that some of the Kafka documentation did claim writes were isolated (where other Kafka documentation contradicts that claim!) so it's possible that depending on which parts of the docs users read, they might expect that G0 was prohibited. That's why this report discusses it in such detail. :-)

> Offsets are committed in the next poll() invocation.

I'm a little surprised by this--not that you're necessarily wrong, but our tests consumed messages synchronously, and IIRC (pardon, it's been 3 months since I was working on Redpanda full time and my time to go get a repro case is a bit limited) did see lost messages with the default autocommit behavior. At some point I'll have to go dig into this again.