Beyond the SQLite single-writer limitation with concurrent writes
turso.tech
turso.tech
I agree that "the single-writer limitation isn't just a theoretical concern", but it's also solvable without forking SQLite. ulimit's the limit! If your goal is resource maximization of a given computer, though, Postgres is likely a better fit.
Joins: https://simonwillison.net/2021/Feb/21/cross-database-queries...
Transactions: https://www.sqlite.org/atomiccommit.html#_multi_file_commit
You mean using ATTACH statement, right? If you use WAL mode, then you cannot get transaction safety / ACID with ATTACH [0]
> If the main database is ":memory:" or if the journal_mode is WAL, then transactions continue to be atomic within each individual database file. But if the host computer crashes in the middle of a COMMIT where two or more database files are updated, some of those files might get the changes where others might not.
Moreover, ATTACH do not support more than 125 databases, so that limits the shards to 125. [1]
ATTACH does not solve the concurrency problems. That's why SQLite also has a BEGIN CONCURRENT experimental branch
Though this stuff moves slowly (that announcement was almost 3 years ago!), so I'm glad to see Turso giving us options today.
I've been advocating with several projects over recent years to get SQLite3 as an archive/export/interchange format for data. Need to archive 2019 data from the database, dump it into a SQLite db with roughly the same schema... Need to pass multiple CSVs worth of data dumps, use a SQLite file instead.
As a secondary, I wonder if it's possible to actively use a SQLite interface against a database file on S3, assuming a single server/instance is the actual active connection.
I also got sqlite-s3vfs working from Python a few months ago: https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/
Both of these are very much read-only mechanisms though.
You could achieve this today using one of the many adapters that turn S3 into a file system, without needing to wait for any SQLite buy in.
For example, from Go, you could use my driver, and point it to a database file stored in S3 using this: https://pkg.go.dev/github.com/ncruces/go-sqlite3/vfs/readerv...
For read-write it's a terrible idea. Object storage assumes objects are immutable. There may be some support for appends, but modifying the middle of an object in place involves copying the entire thing.
What is on the verge of becoming viable is to use Litestream to do asynchronous replication to S3, and have read replicas that stream the data directly from S3. But what's stored in S3 isn't a database file, but a format created for the purpose called LTX.
- Support for HTTP range queries - "Fast" read times - No disk required
I was wrong.
It turns out that for specific SQL queries, it might be fine, but not fast. For queries that do aggregations, like `COUNT`, sqlite loads the whole database anyway.
I've been thinking Turso in the browser would be some eventual addition, is something the Rust rewrite would eventually unlock. And that would require considerable extra backing to make go. But here it is now!
How does this compare with hctree then? It sounds similar, both using an MVCC model.
Really cool though, love the technical details and would be interested to see further. I feel like there is certainly a space for sqlite+fancier features. Though it would break backwards compatibility, I've always wondered what kind of performance gains could be had by using a different varint implementation. And enforcing or always using "strict" for smarter column layouts.
Also would be really cool to have indexed views.
And columnstore seems like a logical direction as well.
The only think turso has in common with postgres is mvcc, which is a rather standard concurrency control model across modern databases. Idk if I answered your question :)
I'm imagining some insane replication behind the scenes, where every write is happening concurrently on a different SQLite DB, and then merged together sequentially into some master DB.
neither the prerelease
https://github.com/tursodatabase/turso/releases/tag/v0.3.0-p...
nor their latest release.
https://github.com/tursodatabase/turso/releases/tag/v0.2.2
The core/mvcc directory does have commits though, so maybe you can.
https://github.com/tursodatabase/turso/commits/main/core/mvc...
SQLite and Postgres/MySQL/etc. occupy different niches. If you need massive concurrent writes, surely that's what Postgres/MySQL/etc. is for? Their engines are built around that from the ground up.
SQLite is built around a file that stores data for a single application, as opposed to being client-server with many clients. I've used it a ton for that, but the idea that your application would have so many threads needing to write concurrently that it would be slowed down by a file lock... just doesn't make sense to me.
What applications need this, and why wouldn't they use Postgres/MySQL/etc. if they need such a high level of concurrent performance? This feels like trying to adapt a small sports car to tow a semi-trailer. It doesn't seem like it's what it's meant for.
The entire "problem" is a side-effect of using the wrong tool for the job.
Having separate tools for separate things makes sense when the things are different enough.
"Tools evolve" isn't a use case. So what's the use case where you need concurrent writes for a database that isn't client-server?
You can justify it as a migration strategy between a file and a service but you're just hammering in screws if you try to force Sqlite to be a multi-client database.
> If there are many client programs sending SQL to the same database over a network, then use a client/server database engine instead of SQLite[1].
I have an app that organizes my movie collection. I use IMDB public datasets as data source. Because versioning is hard, data changes over time (remember who directed Matrix?) it's just easier to rebuild the database every time.
A naive import of datasets is ~131M inserts. It takes 10 minutes (of which 100M takes exactly 6 minutes). Another 10 minutes to further clean up the data, set up indexes, optimize and vacuum.
It's fine for the use case, but still it would be nicer to be able to achieve that in half the time.
The app has no dependencies, database is a single file I can store on S3 and with a little bit of magic use it as a real database - meaning I can run a complex queries over http on that whole database and exchange 50kb.
The client is a standalone HTML file with some JS. No need of server for client, no need for hosting for server except of that S3 bucket.
Sure, it's definitely a niche application, but, at least for me, it makes perfect sense. Start your app with zero deps and add only what you really need. There's too much projects that start with google-esque architecture, with plenty of microservices, only to reach MVP with technical debt the size of US treasury.
Your use case makes perfect sense for SQLite. It's a great example of what it's for. It won't benefit. That's my point.
I don't think those solutions are necessarily that bad. There was one the other week that offered a means of guaranteed sync to a sqlite file in the cloud. Its nice to have the infra to allow users to hop around devices and have backups. What's weird to me, is trying to magic it into a performant multi-client db which the underlying technology was never designed to be.
A single application can need multiple concurrent writes
SQLite already supports lots of threads writing, they just all take turns. What application needs those writes to be concurrent?
D. Richard Hipp is a genuinely fantastic human being, and SQLite is a project developed at literally the level of planetary infrastructure given how broadly and everywhere it appears.
Forking his project while using the name to garner recognition and attention is poor form, and it is difficult to have faith in the results.
As long as the fork doesnt violate trademark (turso vs sqlite) it is working-as-intended?
I, for one, encourage this kind of behavior. We should have more forks. More forks = more competition = better results for everyone.
---
To make an analogy. Would you say the same thing if this were a for-profit company?
"I cant believe someone else is competing the same space as $x. $x is hugely successful, and so many people use it. I dont know why there's an alternative"
Plus, it's just technically bad. Most cases where you'd want to scale up sqlite are better served by a client/server database.
But not all of them, right? I don't need scaling across different hosts, but I need scaling between different threads on the same host. And I don't want for that PostgreSQL, MySQL, Oracle an other stuff like that.
The don't use the name. They use Turso. Even the HN title is wrong - the article title doesn't mention SQLite.
They refer to SQLite, but how could you not if that's what you forked from, and that's what has the functionality you're changing. That would be a very weird article if we didn't have that context.
“The next evolution of SQLite”
That is a material misrepresentation and absolutely trading on the SQLite name
How? The whole point of trademark is to avoid confusing users that an alternate product is the same as the original product.
By explicitly saying "Next evolution of SQLite", or "A fork of SQLite", or a "Better SQLite" .. all of this is saying our product is distinct and different from SQLite.
If the fork were called "nue-sqlite" or "sqlitest" or "fastsqlite", there's an argument to be made.
The problem is that the phrase “the next evolution of SQLite” conveys continuity and endorsement.
A reasonable reader could conclude that Turso is an official successor or the next release of SQLite — that it represents the official lineage of the project.
Phrasing like “SQLite-compatible,” or “a fork of SQLite” would be clear and factual.
Calling it “the next evolution of SQLite” isn’t factual; it’s marketing positioning, and it implies ownership of SQLite’s identity and lineage.
This reflects a broader pattern in how the fork has been presented publicly.
The messaging often treats the original project as something to be fixed rather than a foundation to be respected.
Referring to Turso as a product while leveraging the SQLite name reinforces that framing — co-opting a public-domain engineering gift into a commercial asset.
Our Finns are also not the standard Finns. I know Pekka closely for 15 years, and he smiled 3 times.
* You are right not to rush. You should keep using SQLite until Turso matures. Some use cases are more tolerant to new tech than others. It will take time for us to reach the level of trust SQLite has for broad use cases, but we are hoping to add value for some use cases right away. Never rush, tech matters!
* I have never met Hipp, but only heard great things about him.
* We never had a fight with SQLite over their contribution model (or about anything for that matter, I never even met Hipp or anybody else from SQLite). We just disagree with it - in the sense that we believe in different things. We don't think what they do is fundamentally wrong. Different projects take different paths.
* We are not using the SQLite name. We compare ourselves to SQLite because we are file and API compatible, and we do aspire to raise the very high bar they have set. It is hard to do this without drawing the comparison, but we are a different project and state it very clearly. I am not a lawyer (and neither you seem to be), but we believe we are doing is okay. If we ever have any valid reason to believe we crossed a line here, we will of course change course.
* We are not "startup bros". We spent 20+ years of our lives building databases and operating systems.
The issue I raised is that the phrasing and the way the fork has been presented create the impression of continuity and endorsement that doesn’t exist. That’s a reputational and ethical concern, not a legal debate. Calling it “the next evolution of SQLite” is, in practice, absolutely trading on the SQLite name.
There was a very public, one-sided disagreement about SQLite’s contribution model at the time, and you’ve been open about your criticisms of SQLite in the years since. That’s the context for my comment; it isn’t something I’ve imagined.
The disagreement about their contribution model of course happened, but the meaning you ascribe to it, perhaps is something you imagined. It boils down to what you understand "criticism" to be.
If I see someone doing something wrong, I will criticize them. That certainly never happened. What happened is that we pointed out pros and cons of an open and closed development model. We believe a piece of technology that plays the role of SQLite would benefit from having an open model. And exactly because they are absolutely not doing nothing wrong with not being open, we created our own thing. Hard to see how that is a "criticism".
I said that a billion times, and here's a billion and one: there's absolutely nothing wrong with a closed model. SQLite is doing nothing wrong. They contributed tremendously to the databases we used every day.
I do think an Open model yields so many benefits that should someone rewrite SQLite with an open model, even starting 20 years later, they would end up ahead.
There is now a very easy way to prove or disprove this particular hypothesis.
Stay tuned!
My concern is about presentation, ethics, respect, and about the co-option of a gift to the commons — particularly how your public messaging gives readers the impression of lineage and endorsement that doesn’t exist.
Regardless of your intent, that’s the effect of calling Turso “the next evolution of SQLite.” You’re welcome to disagree; it would be very strange if you didn’t.
Your claim that we are doing something ethically wrong seems to be informed by your pre-existing opinion of me, that itself derives from the "unpleasant fight" (that never happened).
As for the tagline we use, most people don't get the impression that there is any violation of ethics or respect. This is evidenced by other people's reaction here. You do, and you are within your right. I can't, unfortunately, please everybody.
Some people are more relevant than others, though: in this case, if the authors of SQLite expressed their opinion to me that this crosses a line in their view, I'd change it, without blinking an eye.
I have a tremendous respect for them, and we want our messaging to convey nothing but that!
The earlier disagreement with SQLite was one-sided because the other side saw no reason to engage. I expect that dynamic will continue.
Just to set some context, I remember talking to Glauber and Pekka last year, right around when the Turso folks started toying with the idea of reimplementing SQLite in Rust.
It’s a moonshot, but if anyone can pull it off, it’s these people. If you don’t know their background, checkout ScyllaDB. It never ceases to amaze me how belligerent some HN folks can be - they won’t even take five minutes to do a Google search before calling someone a nobody or “SV tech bro.” Not that I’m saying you should do that to anyone.
> We take our code of conduct seriously, and unlike SQLite, we do not substitute it with an unclear alternative. We strive to foster a community that values diversity, equity, and inclusion. We encourage others to speak up if they feel uncomfortable.
Sometime after the HN backlash[2], that paragraph was removed. Eventually, Turso released https://github.com/tursodatabase/turso without a code of conduct and hasn't had one since.
For me, the saddest part of the affair is that Richard Hipp and Glauber are both committed Christians. There should be no reason for envy, enmity, or memory games. But when business and money was on the table, it seems only one of them followed Jesus.
[1] https://github.com/tursodatabase/libsql/commit/3ac3ad263c0f0... https://web.archive.org/web/20221004144141/https://glauberco...
The charitable interpretation is that when people pointed this out, I have read it, and come to agree that the wording was too strong and not representative of what we wanted to convey. Therefore, it was taken down.
But why be charitable when one can just throw accusations around from the armchair towards people one barely knows ?
The one thing you are right about, is that I am failing to live up to the Lord. As much as I try, I keep falling short. It happens not only here but in all aspects of my life. If it wasn't for his grace I would be in Hell for sure.
Of course, you can take that — all of this — as an accusation. You can choose a narrative that's kinder towards you. The question I'd ask myself is if Jesus really wants to hear our storytelling, or if he wants to see us repent.
That said: on certain key aspects of life, I think you're at least closer to the Lord than I am. So Godspeed to you on your journey.
[1] https://github.com/tursodatabase/libsql/blob/3ac3ad263c0f092...