Hacker News Reader

Comment by matharmin | Hacker News Reader

1. Use WAL mode - this allows reading while practically never being blocked by writes. 2. Use application-level locks (in addition to the SQLite locking system). The queuing behavior from application-level locks can work better than the retry mechanism provided by SQLite. 3. Use connection pooling with a separate connection per thread - one write connection and multiple read connections.

Ideally, all of the above would be covered by a library, and not up to the app developer.

I wrote a blog post covering some of this recently: https://www.powersync.co/blog/sqlite-optimizations-for-ultra...

View on HN

Be careful of WAL mode. There are specific limitations.

https://www.vldb.org/pvldb/vol15/p3535-gaffney.pdf

“To accelerate searching the WAL, SQLite creates a WAL index in shared memory. This improves the performance of read transactions, but the use of shared memory requires that all readers must be on the same machine [and OS instance]. Thus, WAL mode does not work on a network filesystem.”

“It is not possible to change the page size after entering WAL mode.”

“In addition, WAL mode comes with the added complexity of checkpoint operations and additional files to store the WAL and the WAL index.”

https://www.sqlite.org/lang_attach.html

SQLite does not guarantee ACID consistency with ATTACH DATABASE in WAL mode.

“Transactions involving multiple attached databases are atomic, assuming that the main database is not ":memory:" and the journal_mode is not WAL. If the main database is ":memory:" or if the journal_mode is WAL, then transactions continue to be atomic within each individual database file. But if the host computer crashes in the middle of a COMMIT where two or more database files are updated, some of those files might get the changes where others might not.”

Yeah, wal can also grow very large, making read operations increasingly slow on wrong checkpoint configuration, which also makes checkpointing slower and it grows out of proportion.

Inefficient queries, no reader gaps and a bad manual checkpointing system can bring the system down.

Does WAL2 mitigate the performance reduction?

https://sqlite.org/cgi/src/doc/wal2/doc/wal2.md

That's neat, I haven't worked with SQLLite for a few years but this changes it. Of course you can still fall into the trap of reaching your checkpoints faster than the first wal file needs to write back to the database, but I think that's a less critical problem in pratice.

Reply on news.ycombinator.com