Jepsen: MySQL 8.0.34
jepsen.io
jepsen.io
I think two isolation levels that make sense are either:
* read committed
* serializable
You either go all the way to have a serializable setup, where there are no surprises. OR, you go in read committed direction where it is obvious that if you want have a consistent view of the data within a transaction, you have to lock the rows before you start reading them.
Read committed is very similar to just regular multi-threaded code and its memory management, so most engineers can get a decent intuitive sense for it.
Serializable is so strict that it is pretty hard to make very unexpected mistakes.
Anything in-between is a no man's land. And anything less consistent than Read Committed is no longer really a database.
So I really only see serializable to be the only sane isolation model (for r/w transactions), and snapshot isolation is a good model for readonly transactions (basically you get a frozen in time snapshot of the database to work with). This also happens to be the only modes in which Spanner gives you: https://cloud.google.com/spanner/docs/transactions
So assuming you are looking at reading "consistent snapshot" in the context of a real time transaction. If the data that you want to read as a "consistent snapshot" is small, locking + reading is good enough in most cases.
If the data to read is too large (i.e. query takes long time to execute, and pulls a lot of data), you are going to have ton of scaling issues if you are depending on something like "repeatable read". Long running transactions, long running queries, etc are bane of all the database scaling and performance.
So you really want to avoid that anyways, you would almost always be much better of changing your application logic to make sure you can have much shorter, time bounded transactions and queries and setup better application level consistency scheme. Otherwise you will at some point hit scaling/performance problems and they will be an absolute nightmare to fix.
Repeatable read type of setups also make it much easier to accidentally create much longer running transactions, and long running transactions/too many concurrent open transactions/etc can create really unexpected, very hard to resolve performance issues in the long run for any database.
I've used both "read committed" and "repeatable read" with MySQL and learned to deal with each in their own way.
The problem I've seen is with large/long-lived transactions that impact performance, where the solution is to divide writes into smaller transactions in the design--"read committed" does tend to encourage smaller transactions.
Isolation Levels and MVCC in SQL Databases: A Technical Comparative Study
// Oracle, MySQL, SQL Server, PostgreSQL, and YugabyteDB.
https://fosdem.org/2024/schedule/event/fosdem-2024-3600-isol...
Also… I’ve been issues in MySQL repeatable read mode where a single SELECT, selecting a single row, returned impossible results. I think it was:
SELECT min(value), max(value) FROM table WHERE id = 1;
This isn't CONCAT-specific, BTW--we just use CONCAT because it allows us to infer anomalies in linear, rather than exponential time. Same kinds of behaviors manifest with plain old read/write registers.
My guess would be that it exhibits some or all of these same issues [edit to add: see footnote 2]. With a single-node cluster, I don't ever recall reading anything about Aurora offering different MVCC or isolation level semantics than upstream InnoDB.
AWS documentation says "These isolation levels work the same in Aurora MySQL as in RDS for MySQL" [1] and keep in mind standard non-Aurora RDS is much closer to unmodified upstream MySQL.
That said, there are some unique wrinkles in Aurora's cluster behavior due to the shared storage. For example, if you keep the default isolation level of repeatable read, long-running queries on Aurora Replicas will inherently block purge of old-row versions on the whole cluster. In contrast, a traditional MySQL replica set (using async binlog replication) does not behave that way, because each replica has its own storage and own purge threads.
[1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...
[2] Re-reading the Jepsen results, it appears all of these anomalies come from the exact same documented InnoDB behavior: "If you update some rows in a table, a SELECT sees the latest version of the updated rows, but it might also see older versions of any rows" as per https://dev.mysql.com/doc/refman/8.0/en/innodb-consistent-re... -- and presumably Aurora maintains the same behavior, since otherwise it would break compatibility with MySQL in extremely subtle and confusing ways.
The engineers at Plaid wrote a really useful article on the differences: https://plaid.com/blog/exploring-performance-differences-bet...
The biggest, for me, is that Aurora clusters use shared storage and therefore the isolation model is slightly different(plus other ramifications), read committed is only possible by setting a cluster wide parameter and read uncommitted is not possible, as far as I can tell.
That "shared responsibility model," they lean on it heavily
AWS/Rackspace support just say: "It's your problem as we don't manage what is inside the AWS service".
I love this; what a great example of pushing the state of the art forward. Kudos!
aphyr, thank you. call me maybe and later jepsen.io have been consistently some of the best content I've ever read on the internet.
I’ve been reading your stuff for almost 10 years and doing work at this level of rigor makes the world a better place.
If you want to update a record based upon data in another record, you should do a locking read on that something else and maybe the record you're updating. If you run an sql query to update a record based upon some other record using a single query, MySQL will lock both for you anyways.
If you need to update something based upon multiple something elses, in my experience that's very deadlock prone. Instead you should lock some kinda locking record, then do a repeatable read on the data you want, then do an update.
The point in time of the repeatable read isn't established until you perform a consistent read. Select... for update isn't a consistent read. So it works perfectly fine in the face of concurrency while not locking dozens or hundreds of rows using a normal SQL update.
How to do this tho? Do you mean like this?
BEGIN TRANSACTION
// all concerned rows in the select statement
SELECT * FROM A, B... FOR SHARE;
// update relevant rows
UPDATE A SET x = true ... FOR UPDATE;
COMMIT
END
UPDATE A, B SET A.x = B.x where A.b_id = B.id and A.id = 1;
If it's more complex, it might have to be more like this:
SELECT * FROM A ... FOR UPDATE;
SELECT * FROM B... FOR SHARE;
UPDATE A...
If B is only ever updated after locking A, you can safely do this:
SELECT * FROM A ... FOR UPDATE;
SELECT * FROM B...;
UPDATE A...
Generally speaking, I wouldn't recommend locking "for share" then updating it later. This can result in deadlocks because you're upgrading it to an exclusive lock.
Postgres (presumably other databases) can propagate these locks to table locks [1] and cause contention for the whole infra.
[1] https://blog.heroku.com/curious-case-table-locking-update-qu...
Similar to how Rust forces you to reason about allocators.
https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml...
https://opensource.docs.scylladb.com/stable/cql/consistency....
> We designed a small test suite for MySQL using the Jepsen testing library at version 0.3.4. We used the mysql-connector-j JDBC adapter as our client. We tested MySQL 8.0.34, and MariaDB 10.11.3 on Debian Bookworm. Our tests ran against a single MySQL node as well as binlog-replicated clusters with one or two read-only followers, without failover. We also ran our test suite against a hosted MySQL service: AWS’s RDS Cluster, using the “Multi-AZ DB Cluster” profile. This is the recommended default for production workloads, and offers a binlog-replicated deployment of MySQL 8.0.34 where secondary nodes support read queries.
You wouldn't assume that Plex and XMBC shared much compatibility despite forking around the same time.
---
[1] https://jepsen.io/consistency
[2] https://jepsen.io/consistency/models/serializable
[3] https://www.postgresql.org/docs/16/sql-set-transaction.html
I just found out yesterday[1] that in Postgres “serializable” transactions can still have anomalies if other non-serializable transactions are running in parallel! So check your DBMS very carefully before trying this, I guess.
IIRC Jim Gray said that Repeatable Read is 99% of Serialisable anyway, all serialisable does is hide phantoms.
[1] speaking for MS SQL, which is mainly locking based.
Partly because you don't really want arbitrary long transactions that span however long the user wants to be editing for.
Partly because it's rather rude to roll back all the users edits with a "deadlock detected, please reload the form and fill it out again".
The DB transactions would need to be kept open for user edits only if one were using a pessimistic model.
Am I thinking about this correctly?
"Serializable" is a system property that describes how two or more transactions will take effect. In this context, I would define "transaction" as a business activity with a clear beginning, middle & end and exhibiting specific, predictable data dependencies. Without any knowledge of the transaction type(s) and their semantics per the business domain, it would be impossible to make assumptions about logical ordering of anything.
SQLite is the closest thing to what you are asking for. All writes are serialized by default, but this is probably not what you really want. We can ensure multiple concurrent connections don't corrupt the data files, but we aren't achieving anything in business terms with this.
Even in the context of "transaction" the business activity, they are an extremely useful tool for building up exactly the kind of sequencing and dependency guarantees you refer to.
I was expecting you'd argue for a weaker isolation level than serializable, but then you said:
> Without any knowledge of the transaction type(s) and their semantics per the business domain, it would be impossible to make assumptions about logical ordering of anything.
Serializable isolation level only guarantees some total order of transactions, and yes, it doesn't guarantee that the order will be exactly what you want (e.g. first come, first serve). So, are you now suggesting strict serializability [1], then?
[1] https://jepsen.io/consistency/models/strict-serializable
For example, I imagine that it depends on the workload. If the workload isn't contentious SERIALIZABLE might not make a big difference? Then again if the workload isn't contentious maybe it doesn't matter?
Either way, I'd love to see numbers. Not because I don't believe anyone but I'm just curious what ballpark we're talking about.
Edit: Also, SQLite and Cockroach only allow SERIALIZABLE transactions so the unviability of SERIALIZABLE seems questionable.
SQLite is single writer, so transaction isolation is easy, writes are linear by their very nature.
Cockroach does some really funky stuff, but its serialization guarantees are only within certain conditions. Traditionally it has also had low write throughput compared to other systems, mainly due to its distributed nature. Jepsen touches on that here https://jepsen.io/analyses/cockroachdb-beta-20160829 though things have vastly improved since then.
To your earlier point, it may not even matter depending on the workload, or if you're aware of your database limitations. In cases where it does matter then being aware of the limitations of something like Repeatable Read makes the trade-off worth it.
We've actually been hard at work on adding Read Committed and Repeatable Read isolation into CockroachDB. The risks of weak isolation levels are real, but they do have a role in SQL databases. We did our best to avoid the pitfalls and inconsistencies of MySQL and even PostgreSQL by defining clear read snapshot scopes (statement vs. transaction).
The preview release for both will be dropping in Jan. Some links if you're interested: - RFC: https://github.com/cockroachdb/cockroach/blob/master/docs/RF... - Hermitage test: https://github.com/ept/hermitage/blob/master/cockroachdb.md
SQLite is unviable in a lot of use cases. Also transactions are mostly a joke anyway, they were completely broken in MySQL for years and no-one cared, real systems don't actually use them much.
If you don't, you sooner or later get presented with unexpected 'transaction aborted due to deadlock' errors in prod. Better have someone who's already been through that then, at the very least.
Transaction deadlocks are another common issue that is triggered by concurrent transactions even at lower levels and should be retried also.
We handle this by passing our transaction a function to run - it will retry a few times if it gets a deadlock. But I don't consider this to be very low level.
Oh neat, I was just thinking about something like this the other day.
> 4.2 Recommendations
> The core problem is that MySQL claims to implement Repeatable Read but actually provides something much weaker. We see two avenues to resolve this problem.
> The first is to keep MySQL’s behavior as it is, and to clearly document the consistency model “Repeatable Read” actually provides. There is precedent in other databases: PostgreSQL’s Repeatable Read is actually Snapshot Isolation, and exhibits behaviors which violate PL-2.99 Repeatable Read. However, PostgreSQL’s documentation eventually mentions that their Repeatable Read implementation is actually Snapshot Isolation. MySQL could similarly document that their “Repeatable Read” means “Read Committed, plus some sort of guarantees that hold until the transaction writes something, at which point mysteries occur.” A precise characterization of those mysteries would be most welcome.
Calling what MySQL's "Repeatable Read" as "Read Committed plus..." would be more confusing as even the simplest repeated read without mutations wouldn't work as expected. The documentation should be more upfront about how MySQL "Repeatable Read" doesn't mean what might be expected. In the meantime keep the "MySQL consistent read documentation"[0] close by.
> The second option is to treat these behaviors as bugs and fix them. Jepsen would be delighted if MySQL and other vendors were to commit to providing PL-2.99 Repeatable Read. However, even satisfying the incomplete, ambiguous ANSI definition of Repeatable Read would be an improvement over current affairs.
I doubt this would be feasible with the extent of deployment. At scale bug-fixes are bugs in themselves. The best that could be done is to create distinct isolation levels for the existing MySQL-RR and the compliant RR isolation levels (somewhat akin to the utf8/utf8mb4 evolution).
[0] https://dev.mysql.com/doc/refman/8.0/en/innodb-consistent-re...
Why would anyone start a new project with MySQL? Is it really superior in anything? I'm in industry for 20+ years and as far as I remember MySQL was always the worst and most popular RDBMS at any given moment.
MySQL also tends to be faster for read-heavy workloads and simple queries.
Also replication is easier to setup with MySQL in my (outdated) experience, even though it's gotten better with Postgres recently and I haven't really been able to compare them myself since I'm just using Amazon RDS Postgres these days and haven't had the need to setup master-master replication (which is the pain point in postgres, and was pretty straightfoward with mysql the last time I worked with it). Setting up read-replicas with postgres is still ezpz.
Postgres specific features tend to be much better than MySQL ones, Postgresql JSON(b) support blows MySQL out of the water. And as far as I can remember MySQL still doesn't support partial/expression indexes, which is a deal breaker for me. Especially in my json heavy workloads where being able to index specific json paths is critical for performance. If you don't need that kind of stuff, you might be fine - but I would hate to hit a wall in my application where I want to reach for it and it's not there.
MySQL used to be the only game in town, so it was the "default" choice - but IMO postgres has surpassed it.
Do generated column indexes meet this need?
CREATE TABLE json_with_id_index (
json_data JSON,
id INT GENERATED ALWAYS AS (json_data->"$.id"),
INDEX id (id)
)
create index on my_table(document ->> 'some_key') where (document ? 'some_key' AND document ->> 'some_key' IS NOT NULL);
MySQL supports "multi-valued indexes" over JSON data, which offer a non-obvious solution for partial indexes, since "index records are not added for empty arrays": https://dev.mysql.com/doc/refman/8.0/en/create-index.html#cr...
MariaDB doesn't support any of this directly yet though: https://www.skeema.io/blog/2023/05/10/mysql-vs-mariadb-schem...
Postgres has a MVCC implementation that is recognized as inferior[0] to what MySQL and Oracle do, and requires dealing with vacuuming and all of its related problems.
Postgres has a process-based connection model that is recognized as less optimal than the thread-based one that MySQL has. There are ongoing efforts[1] to move Postgres to a thread-based model but it's recognized as a large and uncertain undertaking.
Other commenters have also explained the still very noticeable difference in replication support, the lack of query planner hints, the less intuitive local tooling.
One thing to keep in mind is that both databases keep evolving, and old prejudices won't take us far. Postgres is improving its performance and replication support with each release. MySQL 8.0 added atomic DDL and a new query planner (MariaDB did their own query planner rework in 11.0, widening their differences). Both are improving their observability. So the race is far from over. But I definitely wouldn't count MySQL out.
[0] https://ottertune.com/blog/the-part-of-postgresql-we-hate-th... [1] https://www.postgresql.org/message-id/flat/31cc6df9-53fe-3cd...
It also works for others, Github for example.
The only thing I am missing at the moment is a native UUID type so I don't have to write functions that convert 16bit binary to textual representation and back when examining the data manually on the server.
I know facebook uses mysql, but I also know that it is a bastardised custom version that has known constraints and has limited use (no foreign keys for example).
I spoke to the DBA who first deployed MySQL at Github and the vibe I got from him immediately was that he had doubled down on his prejudice: which is fine, but its not ok to ignore that it can be a lot of effort to work around issues with any given technology.
For a great example of what I mean: most people wouldn’t choose PHP for a new project (despite it having improved majorly) - the appeal to authority there is to say “it works for Facebook” without mentioning “Hack” or the myriad of internal processes to avoid the warts of PHP.
That a large headcount company can use something does not make it immune from criticism.
Is this really true?
I used to be a full-time PHP developer but I personally don't touch that language anymore. But it's still very popular around the world, I've seen multiple projects start this year use PHP, because that's the language the founders/most developers in the company are familiar with. Probably depends a lot on where in the world you're located.
Last Stack Overflow survey had ~20% of the people answering the survey saying that they still use PHP in some capacity.
Personally, I like using Typescript/Javascript on both front end and backend, but I don’t look down at PHP backends at all. And it’s come a long way as a language.
I’ve been a fan of rolling your own stdlib as the semantics there are old and weird, but vscode tells you so who cares anymore.
Most people on HN, or most developers in the world?
PHP is still very popular, and plenty of people start new projects in it all the time.
> does not make it immune from criticism
Show me a technology without critics and I'll show you a technology zero people use.
Pulls, Pushes, Issues and GitHub stars are terrible ways to gauge the popularity of a language.
There is no better measure I'm aware of, and I'll take any measure you supply.
I would definitely also argue that Ruby is in pretty significant decline, the majority of Ruby projects were sysadminy projects from the 2010 era and most sysadminy types learned it as an alternative to perl. Web developers who learned it were mostly using Rails which has fallen somewhat out of favour. YMMV obviously, but I can understand it's decline as Python has concretely taken over the working space and devops tools like Chef/Puppet are not en-vogue any longer as Go and Kubernetes/CNCF stuff took the lions share.
Equally: javascript (node, really) is less favourable to many JS devs than Typescript. If you aggregate TS and JS then you'll see that the ecosystem is growing but many people who are JS folks have switched to TS.
I'm taken aback by what you seem to suggest though; Would you seriously claim that most new projects ARE using PHP?
I would happily argue that point with any data you supply, it's completely contrary to my experience and understanding of things and I have a pretty wide and disparate social circle in tech companies.
No. I didn't say that, and we need to clarify what you meant originally to make sense here.
When you say "most people wouldn't start a project in php", there are two ways to interpret "most" in that sentence: "the majority of" (ie 50%+) or "nearly all of" (ie a much higher percentage). Both are accepted definitions for "most".
I assumed you meant the latter: ie "nearly everyone would not start a project in php", which is what I disagree with, because the former makes little sense in context.
If you did in fact mean "a majority of people would not start a project in php" then of course I agree because that sentence can be substituted to mention any programming language in existence and still be true, because none are ever so dominant over all others in terms of popularity, that more than half of all new projects are written in said language.
What I tried to convey is that PHP is not enjoying the development heyday it once had, and the numbers of people choosing PHP for a new project today (even among people who learned development with PHP) is decreasing. It's not popular.
let's try to leave it as: "I believe PHP to be in decline for new projects as a share of total new projects divided by the total number of developers who are starting new projects".
Also, note that TypeScript is tracked separately from Javascript, which is likely part of its decline. I wouldn't be surprised if JS backends are ultimately declining as well (perhaps Go and Python are taking its place?)
People keep saying that. Ruby has had a "huge decline" if you look at the percentage of commits on Github over the last decade [1], a decrease of more than a factor of 3. However, in that same decade Github has grown (much) more than a factor of 3. So the total number of Ruby commits on GitHub has grown substantially. That's not really what I would call a huge decline.
[1] https://madnight.github.io/githut/#/pull_requests/2023/3
Isolation level consistency is not a problem I heard anyone talk about, but that's probably because most devs interact with the database via Active Record which is not exactly known for its transactionality guarantees (and is, of course, a source of yet another set of problems).
MySQL 8 adds the `BIN_TO_UUID()` function (and the inverse, UUID_TO_BIN), and supports the quasi-standard bit swapping trick to handle time-based UUID's in indexed columns.
I have a few reasons for this view, but they mostly revolve around operational complexity. From a developer's point of view postgres is fantastic. Far saner SQL dialect, tons of great features. When it comes to operations though, that's where mysql has the edge, and ops is half of using a database - it's an important facet for a business to consider.
As other commenters have mentioned, postgres requires careful tuning of the autovacuum process, otherwise it can't keep up as the workload grows.
Postgres has a far more advanced query planner, but it comes at the cost of potentially blowing up your app at 3am, and it gives you no tools to patch in a quick fix while you address the root cause. This frankly ignores the reality of operating a business. Sometimes you need a quick fix, even if that might lead to users developing bad habits. Yes there is the pg_hint_plan extension, but that still only helps you later after the problem had happened. You can't pin a query plan. To me the ideal situation would be for postgres to continue to use the old query plan, but emit some structured log to tell you it thinks it's now suboptimal. But I digress.
Thirdly, postgres has no way to have an index clustered table. This lets you trade a small cost on write for greater page locality when reading related rows. Postgres let's you do this as a one time operation that takes the table offline for the duration, which isn't sufficient if you need it.
Fourthly, mysql is still easier to upgrade. You will need to upgrade your database at some point. Mysql has great support for upgrade in place, as well as using replication to build a new db. Mysql replication has always been logical replication, which has tradeoffs of course, but what it buys you is the ability to replicate across different versions. Pg's logical replication still has a bunch of sharp edges.
Ok this rant is long enough already, but I do want to emphasise that this isn't hating on postgres. I know it's controversial to be recommending mysql over postgres, but I do think the ops concerns win out.
Ps the orioledb project is fantastic and I hope it one day becomes the default for postgres.
Are you using OrioleDB in production to have those good experiences with it?
It's tackling what I see as one of the foundational weaknesses of postgres, which is the storage engine. A good number of it's downsides stem from the fundamental design of the storage layer, and if orioledb succeeds in becoming stable then there is a class of issues that would simply go away.
- It scales up fine for 99+% of companies, and for the ones that need to scale beyond that there are battle tested solutions like Vitess.
- It is what people know already so they don't have to learn anything new.
Same reason why people still make new websites in PHP I guess. It's not fancy but it works fine and won't bring any unwelcome surprises.
This doesn't explain why it was so popular for starting new small projects, but people were also choosing mongodb at that time too, so ymmv :)
Nowadays postgres has grown a lot of features but I believe it is still behind on built-in compression?
Added: this old blog post of mine is still getting traffic 10 years later; probably still valid https://williame.github.io/post/25080396258.html
- its case insensitive by default, which can make filtering simpler, without having to deal with a duplicate column where all values are lower/upper cased. - MySQL implements loose index scan and index skip scan, which improves performance of a number of join aggregation operations (https://wiki.postgresql.org/wiki/Loose_indexscan)
This is obviously up for debate, but subjectively I find this to be an absolutely terrible design decision.
With that said, in all my years and thousands of tables across multiple jobs, I have yet to see a single case where I had to change a table to be case sensitive. So I guess for me it is a sensible default.
InnoDB is comparatively slow, but you get much better transactionality (IE; something that is much closer to ACID compliance). Row level locking is faster for inserts than table level locking, but table level locking is faster for reads than row level locking.
Regardless: Both storage engines do not scale with core count as effectively as postgres due to some deadlocking on update that I have witnessed with MySQL. (not that Postgresql is the only alternative btw).
> Both storage engines do not scale with core count as effectively as postgres due to some deadlocking on update that I have witnessed with MySQL.
Strange take since a deadlock is rather an exceptional event you want never to occur so deadlocking, in algorithm design, wouldn't be considered a reason one would say that the implementation does not "scale with the core count". Whether or not the algorithm scales with the core count is for many other different reasons but not deadlocks.
Considering the "scale with the core count" design problem, Postgres process-per-connection architecture makes it a much less viable option than, say, MySQL so this is wrong as well.
deadlock was the wrong terminology to use, apologies, I keep writing from my phone as I am travelling at the moment: I meant lock contention, specifically in memory. A deadlock would be a hard stop but what I observed was a bottleneck on memory bandwidth past a certain number of cores (24) with update heavy workloads.
So, appreciate your points but I don't think I am wrong in the core thesis of my statements. x :)
The read performance advantages of MyISAM are solved better by using SSDs instead of HDDs.
MyISAM will cost you dearly when performing actions like "trying to create a new replica of an existing database".
Also, Mysql has had native replication for a very long time, including Galera which does two-step commit in a multimaster cluster. Although Postgres is making some headway in this regard, it is my impression that this is only quite recent and not yet fully up to par with Mysql yet.
You still do. The auto Vacuum daemon was added in 2008ish, so it isn't too bad. Just more complexity to manage.
> it's also using some kind of threading model
It does a process per connection just like web servers did back in the day when C10k was a thing. A lot of the buffers are configured per connection so you can get bigger buffers if you keep the number of connections small.
It's the most developer-friendly thing out there. Particularly for a datastore CLI, which is inherently something you use rarely, MySQL's is just a lot nicer, more discoverable.
I think it has the least bad HA story among (free) traditional SQL-RDBMSes too (not that I understand why anyone would start a new project on a traditional SQL-RDBMS at all).
Edit: I would just love a comment from the person who thinks 'missing feature in the past' is wrong, unfair or irrelevant as a reply to a 'missing feature in the past' comment.
That said, personally I wouldn't describe either of these as "not too long ago". Technology rapidly changes and many things from either 2001 or 2010 are considered rather old.
What do you recommend these days?
Yes, its replication support out of the box is decades ahead of Postgres.
(Mongodb has an even better replication story than Myqsl, but Mongo isn't a real database.)