SQLite's Automatic Indexes
misfra.me
misfra.me
There's an experimental module and CLI command in the normal shell to use this logic to dump out a suggested list of indexes to create for a given SQL command to speed it up.
One of the difficulties is indexes have a write cost so heuristically deciding on the trade off can be complicated.
https://sqlite.org/tempfiles.html#transient_indices
SQLite also creates background indexes for ROWID, of which I was aware.
The transient indexes will be created in a separate, temporary file. They will be erased on a commit, rollback, or crash recovery. They will not persist.
> SQLite comes with instrumentation (specifically the SQLITE_STMTSTATUS_AUTOINDEX verb for sqlite3_stmt_status() - http://www.sqlite.org/c3ref/stmt_status.html) that can be used to detect when automatic indices are used and alert the developer through a back channel to this fact so that she can fix the problem with an appropriate CREATE INDEX. In other words, SQLite provides you with the tools to help you detect and eliminate the use of automatic indices.
Either way SQLite doesn't have heap tables like PG (which only has heap tables), all tables are clustered indexes.
Also, a btree is created for rowid, according to these docs.
https://www.sqlite.org/withoutrowid.html#:~:text=In%20an%20o....
What do you think a btree index is exactly? Especially a covering index?
> It's just that the tables rows are entries in a btree, indexed by rowid.
That’s what a clustered index (/ table) is. It’s when the table's storage is organised by the chosen clustering key. In SQLite, all tables are clustered, and the clustering key is the ROWID, or the primary key for tables WITHOUT ROWID. You can think of it as the table being its own covering index (keyed on the clustering key, and INCLUDE-ing all other columns).
By opposition a database which uses heap tables (like postgres) needs a separate index for its PK (it also needs one to enforce unicity so that's two birds one stone).
The value of the rowid btree is the row.
That's what a clustered index is, the table is the index, the primary key is the key all the rest of the columns are in the value rather than the value being a row id that needs a second lookup to get to the data.
https://en.wikipedia.org/wiki/Database_index#Clustered
You can still have secondary indexes that point to the key of the clustered index.
SQLite is quirky in that even if you define a primary key, it has a hidden actual primary key called the rowid, unless you define it "WITHOUT ROWID" which was added later, this is what they say in the docs:
"WITHOUT ROWID is found only in SQLite and is not compatible with any other SQL database engine, as far as we know. In an elegant system, all tables would behave as WITHOUT ROWID tables even without the WITHOUT ROWID keyword. However, when SQLite was first designed, it used only integer rowids for row keys to simplify the implementation. This approach worked well for many years. But as the demands on SQLite grew, the need for tables in which the PRIMARY KEY really did correspond to the underlying row key grew more acute. The WITHOUT ROWID concept was added in order to meet that need without breaking backwards compatibility with the billions of SQLite databases already in use at the time (circa 2013)."
Nobody said you don’t have to create other indexes.
> Also, a btree is created for rowid, according to these docs.
As the GP says that btree is the table itself, that’s what a clustered table is.
This is the bit that reveal this information:
> As an ordinary SQLite table, "wordcount" is implemented as two separate B-Trees. The main table uses the hidden rowid value as the key and stores the "word" and "cnt" columns as data.
So one of the btrees is “the main table”, of which rowid is the key.
MSSQL and Oracle give you a choice for table storage to either have unordered heaps or ordered b-trees, depending on whether a clustered index is defined.
PG only has unordered heaps, indexes are always secondary data structures. PG could really use true clustered indexes as an option.
Funny you posted this now, I just recently researched Sqlite internals for my own project. Maybe you will find it interesting: https://twitter.com/high_byte/status/1607853384123703296
Interesting blog, I love that you keep consistency writing for so long.
It does amaze me the gaps that people have, though.
Then again, I have rarely run into performance issues with database engines. Either the ones I've had contact with (SQLite, MSSQL, Postgres) are very good at what they do (which I suspect they are), or my requirements were too puny for them to break a sweat (which I also suspect to be the case).
I did use MSSQL Studio's query analyzer once to fix a performance issue and was able to get the query in question from nearly an hour down to under a second. triumphant grin I don't remember the details of the query, but the solution was to use a hash join.
An interesting project to manage indexes on postgres that seems tangentially related:
Also, most of the time joins are made on foreign keys and these should always be indexed. So I guess this should be a rare occurrence if the database is properly thought-out.
I thought they just dropped the extra L because it made more sense without it.
Furthermore this is exactly in furtherance of your stated goals: sqlite supports indexes, by using automatic indexes it avoids the need to implement merge joins and hash joins.
https://news.ycombinator.com/newsguidelines.html
This could become a (very) good comment if you added enough explanation for people to understand what the issues are. But just posting a putdown isn't in the intended spirit of the site. If you know more than others, e.g. about nested loops in databases, that's great - but in that case please share some of what you know, so the rest of us can learn.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...