SQLite should have (Rust-style) editions
mort.coffee
mort.coffee
Too often it's just a list of issues and a wish that everyone else will change.
In (mild) defense of SQLITE_BUSY - busy_timeout just tells sqlite to sleep and retry up to the timeout when it receives SQLITE_BUSY. It seems like a sensible default for a library to leave that up the calling code - which may have something else it could do while it waits. However, that logic often gets missed!
Even if you're in one of those unconventional settings (say, a bare-metal microcontroller or something), you'd probably still start from edition 2026 and mutate your settings accordingly, rather than using the defaults that are 26 years old.
Given how many projects are potentially out there effectively relying on the current settings, and SQLite's general attitude to backwards compatibility, that would likely not be considered a good idea. Opting in with an edition flag for new (or updating) projects does seem like a good solution to this to serve all of old, active, and new projects, but it would increase potential bug surface area and therefor testing requirements, and the existing setting do allow all that to be opted in/out to/from already (and it is only four settings we are talking about here).
That FKs being enforced is set per-connection rather than at the database level is something that surprised me a lot when I found out. A way of setting that at DB creation (or via ALTER DATABASE after) seems like quite an omission because if you have multiple potential routes that can update the same DB any one of them could cause serious the others will encounter.
That's the key concept here. When tightening up the defaults, an "edition" mechanism is a good solution.
Now we need this for C/C++, which have much legacy stuff which ought to go away for new code. This is more feasible than it used to be, because "Convert this Edition 4 code to Edition 5" is something LLMs can do now.
I'd never seen all the rules for SQLite soft typing written out before. Those are more complicated than strong typing.
P1881 Epochs proposed to WG21 (the C++ standards committee) in 2019 by Vittorio Romeo
The committee found plenty of problems with this, and made it clear that if Vittorio did all the hard work to resolve those problems they would find more, P1881 was abandoned.
There was a Reddit thread https://www.reddit.com/r/cpp/comments/1tja9zr/c_profiles_a_c... which suggested that the "Profiles" idea Bjarne is pushing for C++ 29 could be used to deliver this.
So, you're not the first person to notice that this is a good idea, P1881 was written after Rust's 2018 Edition, but before 2021 Edition with its even more significant improvements. I firmly believe Rust's Editions unlock not only technical possibilities (though it does certainly do that) it unlocks an appetite from users which is good for your ecosystem.
In C++, certainly. In C, though, what do you not do in C23 that you was doing in C99?
"Modern C" by 1999 already included "Don't use gets()"
The "lock on write" problem is that in MySQL i could run a OLAP pipeline for a few hours and have a fully functioning database with degraded perfomance, on SQLite the same pipeline would lock the database for the full hour. (there are surely ways to solve this (eg using the main db as read-only and a secondary db for writes or splitting the writes in incremental transaction), but it is not a "myth".
If you run a transaction with writes for an hour on any database, the data you update will literally be locked. So your example only works if results are independent of the data other programs want to use.
Of course more granularity of locking is better and enables more designs that would not otherwise work. But somewhere you run into the same problem of writers taking turns.
Because it not tied to the data but to the code.
Instead, what I think should be is that the PRAGMAs become "data" that is always checked in full with "if manually set" and then on next "open" THEY GET APPLIED.
That is.
(and in the command line when open interactively they show up).
By default, SQLite transactions start in DEFERRED mode, acting as read transactions until an actual write operation occurs.
If another connection begins writing to the database while your transaction is in this read state, an immediate SQLITE_BUSY error is triggered regardless of what you set busy_timeout to
There was one case where all transactions were implemented using nested `SAVEPOINT bla` so `BEGIN IMMEDIATE` could not be used without more hassle, so this ended all “I know I'm going to write” transactions to instantly update a single-row table so that their lock would not begin as DEFERRED and eventually switch to `IMMEDIATE`; this way almost all `SQLITE_BUSY` side-steppings disappeared. (timeout was set to 30 seconds but all read/write transactions were instrumented to have less than 5 seconds duration).
Not a big deal though. Probably just need better ops to bundle the command-line utility that’s the same version as what’s used in your app.
Example: PRAGMA foo=1 is introduced in 2027. PRAGMA edition=2030 implies this foo pragma. Now you unnecessarily lock out three years worth of releases.
I’d be interested to learn if there are any db implementations that take this approach, or reasons this wouldn’t work.
https://github.com/future-file-format/F3
Bundle a wasm decoder as a fallback when native decoder isn't available.
For context, I have a game I'm working on where I build stand-alone builds with all dependencies bundled. The app bundle is 14MB right now. If I added sqlite, it would grow to 15MB. If I added sqlite-but-with-wasm, it'd more than quadruple in size to 60MB.
% curl -LO 'https://cosmo.zip/pub/cosmos/bin/sqlite3'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4845k 100 4845k 0 0 1384k 0 0:00:03 0:00:03 --:--:-- 1384k
% chmod +x sqlite3
% file sqlite3
sqlite3: DOS/MBR boot sector; partition 1 : ID=0x7f, active, start-CHS (0x0,0,1), end-CHS (0x3ff,255,63), startsector 0, 4294967295 sectors
% ./sqlite3
SQLite version 3.40.0 2022-11-16 12:10:08
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite>:And a lot of users of sqlite statically link sqlite within their executable, they actually recommend that instead of dynamically linking.
[0] https://sqlite.org/vfs.html ctrl-f for 'append'
Edit: I got confused between the sqlar command and the append vfs so fixed it.
I think this is the key.
From sqlite.org [1]:
> [Since 2004], the file format has been fully backwards compatible.
> By "backwards compatible" we mean that newer versions of SQLite can always read and write database files created by older versions of SQLite. It is often also the case that SQLite is "forwards compatible", that older versions of SQLite can read and write database files created by newer versions of SQLite. But there are sometimes forward compatibility breaks. Sometimes new features are added to the file format
---
Given editions (A) and (B), what does backwards compatibility look like? Must (B) be backwards compatible with (A)?
If yes -> editions are backwards compatible but not necessarily forwards compatible, which is the current status quo:
-------(A)----(B)--
---+----(A)--------
\
\--(B)--------
1. https://www.sqlite.org/formatchng.html
---
Interesting PS, grepping sqlite.org for "backwards compat": https://pastebin.com/Q7b7h4eM
Some parameters are properties of the database file itself, such as (I believe) journal_mode. These parameters probably result in issues with earlier versions of sqlite; if you create a file with journal_mode WAL, you're not gonna be able to open it in a version of sqlite without WAL support. Same with strict tables; if you have a database which contains strict tables, I assume you're going to encounter issues if you try to open it in a version of sqlite too old to support strict tables. But anything new enough to support the individual database file features shouldn't have any trouble. This is true whether we're talking about an edition pragma or individual parameters set the old way.
Some parameters are properties of the connection, like the foreign_keys parameter. If you're trying to open a database file in a version of sqlite which doesn't support foreign key validation at all, just don't set the foreign_keys parameter.
https://sqlite.org/forum/forumpost/1b9d073a37ca5998
I personally find this idea interesting and would like if SQLite meaningfully moved away from Postel's Law.
https://docs.google.com/document/d/1Qk0qC4s_XNCLemj42FqfsRLp...
The strong typing thing is really interesting. After using JavaScript for a while, I developed PTSD around dynamic types. I became convinced that static typing was the only way to avoid hell.
Then I used Python for a while, and... experienced approximately none of my previous pain. I found that quite odd. Turns out what I was actually after was a sane type system, not a static one. In other words, strong types rather than weak ones.
I do think there are additional benefits to static typing, especially for larger projects and serious work. But I was surprised that most of the pain-delta was in this first jump:
Weak -> Strong -> Static
So, my approach to dynamic features, global mutable state, etc... There's this big list of features which have caused me tremendous pain over the years. I am increasingly weary.
Sometimes you want or need them, though. You don't want the same level of strictness for every project (e.g. throwaway scripts or game jams), nor at every stage of a project -- e.g. being forced to specify invariants is something I'd love to be able to enable, but I wouldn't want that on while I'm still figuring out the basic structure of things.
So for game jams I'd put the language into #JAMMODE (which would be short for a bunch of other flags). But the point here is that jam mode should be opt-in, rather than opt-out. You should have to go out of your way to enable the footguns. And a file with #JAMMODE should be a little smelly. You should think, OK I'm actually shipping this thing now, let's get it out of #JAMMODE. (And strict files can't talk to jam files, and we probably want different strictness levels beyond Debug and Release, etc... someone told me with the strictness "zones" that I'm reinventing half of Ada, hahah.)
As SQLite is often used embedded, bloat matters.
So I suggest that "PRAGMA edition" to be only be a shortcut to a list of PRAGMA commands, that would be expanded at the library level: PRAGMA edition would never appear in the DB file. As such, the build of the library would just support a limited set of editions, with a removal policy in default builds. Maybe editions could even be defined at runtime (a system table?) as a way to load them dynamically if old editions are needed beyond builtin support (think about the state of SQLite in 2046).
The config options are already implemented, so the heavy lifting is done. Editions would amount to a few hundred bytes.
But figuring that out requires understanding how they are implemented, or reading the forum, which the author admittedly didn't do.
The rest are assumptions about best practices (also not shared by the developers of SQLite).
The developer still needs to ensure they apply their set of pragmas, whether that’s a single edition pragma or a set of pragmas. And they still need to understand and carefully choose the pragmas/options they use (an edition really makes this a little harder by abstracting/hiding something that needs to be directly understood and visible).
And these proposed new default pragmas are more incremental improvements rather than complete solutions, more convenience than critical. E.g., while the default affinity is goofy, strict tables don’t come close to a comprehensive validation mechanism — so if you need strong validation you’re likely going to need to implement that at a higher level anyway. (Also, there’s a decent separation-of-concerns argument that you should handle it separately.)
Likewise the busy timeout. The pragma is convenient but you still need to handle busy timeouts. (The author’s problem, “I didn’t realize busy timeouts could happen so the app didn’t handle them correctly”, is not solved by the pragma.)
I guess if foreign keys are handled properly then that's not a problem by definition? But it sounds wrong somehow.
If a parent table ID gets reused, then it's a potential to expose data to a wrong user -- security broked.
That's default behavior, but it can be altered when creating a table. See;
I would suggest if you are storing that much data, SQLite may not be the correct engine. (And you probably shouldn't be using an Int primary key.)
It's a good question to ask, but probably not a concern for most of us.
I'm not sure if you'd want to set one edition in stone every year. Perhaps every 3 years? Or 5 years? Especially for a long-term project like SQLite, that sounds perfectly acceptable!
But I suppose it would be nice to have a standard way to refer to those defaults, in a cross-runtime way.
Well, loose typing can be extremely useful, and having a type of "ANY" would not replace it.
I have built recently an accounting reconciliation system to find discrepancies in data coming from a large variety of sources: some from proper database engines (MySQL MariaDB), but most from proprietary systems that export to CSV. It's amazing how corrupt data can become: dates that are invalid, numbers that aren't numbers, strings strings strings everywhere.
Being able to store the data into tables that have types, but can accept anything, is simply great.
"Loose typing" enforced in a strict typing system can be useful in certain scenarios, but it is regrettable that it instead replaced the strict typing discipline for some time in software. Strict typing should be the default, because it is the most accurate description of data in the vast majority of cases.
...Meanwhile MongoDB being successful for years with no sign of decline.
You can already "store whatever you want" in a serious database that respects types by default. It's called a blob or if you must, a text/varchar.
The first two points are deliberate SQLite design decisions so they're unlikely to change.
So this was a write to a column that did not have INTEGER affinity. If it was intended to be used as a boolean, then it should have INTEGER affinity. I know because I've tried hard to enter integer- and float-like strings as strings in INTEGER affinity columns, and I haven't managed to; I could only insert them as BLOBs, or prefix the string with say '\' and check/remove at the application level. (That was for an ontology-like database, where table EAttribute.eatvalue could have any type.)
Today I noticed I could do `pragma foreign_key = ON`, and despite the pragma being wrong (it should be foreign_keys, plural), it reported nothing. In fact, it reports nothing with the correct pragma either. So check your pragmas!
https://www.postfix.org/postconf.5.html#compatibility_level
https://www.postfix.org/COMPATIBILITY_README.html
You get a warning whenever you depend on the deprecated old default until you either move forward or specifically commit to the old behavior.
Each "policy" they change can be manually set to old or new, and there's a global config to set them all at once based on the version of CMake.
https://cmake.org/cmake/help/latest/command/cmake_minimum_re...
Which makes a lot of sense for those projects! They can be built with CMake 3.2 because they don't need anything introduced in a newer version, so why not declare 3.2 to be the earliest supported version? It's not their fault that CMake interprets "minimum supported version" weirdly?
> But I do not recall a single instance where the bugs might have been caught by a rigid type system.
Which is a shame. Of course the author writes more than this, but this is IMO largely the gist of the argument. At this point it's beginning to feel like this is mostly a sort of stubborn sunken cost fallacy, where they've been arguing this for so long they can't take the "hit" of agreeing to change the defaults.
https://github.com/drizzle-team/drizzle-orm/discussions/2435
Those choices were made for specific reasons that make sense in embedded environment and when backward compatibility is no.1 concern.
But I wouldn't mind feature-sets. Editions are too wide of a concept and tell you nothing at glance what a given code is doing, "enable 2026 set of features" tells me nothing on what is actually enabled.
My proposal does not harm backwards compatibility in the slightest.
> Bad default #3: SQLITE_BUSY errors with concurrent writers
The reaction they're likely to have to that can probably be best described in megatons, like any other nuclear explosion ...
That way somebody can snag a copy of your data and be subject to the same constraints by default.
Then it argues for STRICT tables, recognizing that there are drawbacks without introducing a new feature (custom type aliases, CREATE TYPE alias = base).
If also doesn't even considering what it means for existing data to make tables strict, which is precisely why “there is no pragma to globally make all tables strict”.
Then it argues for setting a busy timeout, and picks 5s. Why? Why 5s and not 1 or 60s? SQLite doesn't decide, which makes perfect sense. Your OS or programming language also doesn't offer you locks with a default timeout: it's either indefinite, or an instant "try lock".
Finally: WAL mode is a different file format, unsupported on many platforms, in more danger of silent corruption. Why should it be the default?
The way I read it, this article does not advocate at any point to change the defaults for existing databases, but rather to start with better defaults for new databases.
Also, regarding the timeout of 5 seconds, I disagree with your premise "SQLite doesn't decide, which makes perfect sense". As the article explains, SQLite decides on the value zero (ie. instant error), which is arguably an inconvenient default.
I'm sorry, where?
SQLite schema is stored as text. If you change the default interpretation of CREATE TABLE with a PRAGMA, your existing tables become STRICT, but (1) they might now have columns with invalid types (which means you have an invalid schema, and your database fails to open), (2) they may have invalid data for their strict types (which you can only figure out with a full table scan, PRAGMA integrity_check will complain).
This was discussed previously on the SQLite forum, you can read the team's position there: https://sqlite.org/forum/forumpost/0248dcf7f0ece9fb
Regarding busy_timeout, why is 5s specifically a better default? You did not engage with my argument: that 5s is no different from 1s or 60s. How do you decide?
Also discussed in the forum, with the team laying out the rational; https://sqlite.org/forum/forumpost/f0da30efa661bd9c
I think the minimum is considering the arguments by the people who promise to maintain the software for the next 25 years.
PS: I actually really like the idea of `CREATE TYPE alias = type` for use with STRICT tables. I would champion that feature request on the forum. Given how schema is saved, I disagree with making it the default. Having to mark your tables STRICT is not such a burden, IMO.
> I'm sorry, where?
Here:
quote
[...]This should be a nice middle ground which avoids breaking backwards compatibility, but lets the database engine move forwards and not be bogged down by its own history.
end quote
> Regarding busy_timeout, why is 5s specifically a better default?
According to the post we are discussing, any number greater than zero is better than the current default:
quote
The default behavior [no timeout] has lead me to writing real-world bugs, where systems would sometimes just crash. I've manually written retry loops to fix it.
end quote
Your forum links may be quite useful as a response to this comment: https://news.ycombinator.com/item?id=48928441
> This should be a nice middle ground which avoids breaking backwards compatibility, but lets the database engine move forwards and not be bogged down by its own history.
Does nothing to address my criticism. Handwaving "this is a nice middle ground" does not address the issues. How does the feature work when enabled on an existing database, and on a new database?
Please understand that when you write `CREATE TABLE …` this statement is copied pretty much verbatim to the schema table. And that's the metadata that's saved for the table: a verbatim copy of the DDL.
If you don't mark it STRICT any current version of SQLite will consider it not to be.
So if you make assuming tables are STRICT the default (even with a PRAGMA), you'll have to deal with tables that you assume are STRICT, but aren't: they have invalid types, or invalid data.
The STRICT feature, as it was added, is backwards compatible in the sense that: (1) all old databases work with new versions of the library, and (2) all new databases fail fast (before corrupting data) in old versions of it, as they'll refuse to parse STRICT tables.
To fail fast for existing database files, you'd need to integrity check the entire databases.
So how do you implement it, what's the alternative, exactly? Add STRICT to every table you create since enabling the PRAGMA? You could do that but, to my knowledge, it'd be the first time you'd do that (modifying the schema before writing it) in 25 years of SQLite.
Whatever you come up with must be backwards compatible, as that is the promise SQLite developers have made.
New database ⇒ use the proposed new magic PRAGMA, start with sane defaults.
Existing database ⇒ don't touch anything, keep legacy "suboptimal" defaults.
CREATE TABLE t1(a DATE, b JSON);
PRAGMA strict=ON;
INSERT INTO t1 VALUES(a,b) VALUES(?1,?2);
Just, you know, explain what should happens when you do this, because the creator of SQLite doesn't know.
When doing so consider the file format, how schema is stored, and what happens when other versions of SQLite touch your file.
Should be simple, given that I'm overthinking it. Thanks.
It's like comparing old php with a strongly typed language.
There is not even a date type...
It’s a product that allows you to do sql like things without a database server. If you need to have database server behavior, you’re using the wrong product.
Different tools for different situations!
- no db user configuration - no installing multiple tenants in the same db - no phpmyadmin (ftp db files) - no remote database hacks - no backup tools
Still I quite a lot of question the use on servers if you have decided that I need a database.
Not that there isn't more valid use cases like local storage or self-contained information transfer for specific use.
It's a fully featured database though, with everything you expect from one, including actually working ALTER TABLEs.
A system-wide config file in /etc that changes defaults for every program would break any program that assumes the old defaults.
It also wouldn't solve the problem of having to manually find out what the current recommended defaults are. With editions, you can simply enable the latest one and know you've got the right defaults.
D will get caught into that edition trap too
All it does is fragment a ecosystem, bloat a source tree and makes maintenance a painful task only to please people who think maintainers should cater to their poor tech hygiene
The only maintainers it causes a burden for are the compiler developers (and tech debt within the compiler). But this is pretty much unavoidable if the language is to evolve while remaining backwards compatible -- e.g. it's not like the c++ compiler has a lower rate of tech debt accrual.
They even had an sqlite4:
It’s a great tool if you want to give a local app its own database. If you need concurrent writes and full ACID guarantees of an industrial strength database, use an industrial strength database.
Yes, other databases will require you to read more manual pages and configure a service. Higher up front cost. Not “lightweight.” But given enough operating time there is a certain unarguable lightness to using the right tool for the job.
- https://sqlite.org/testing.html
- https://sqlite.org/mostdeployed.html
EDIT: added links