Hacker News Reader

Comment by bane | Hacker News Reader

SQLite is one of those awesome things that's the exact opposite of magic. It's beautiful, jaw dropping, engineering that exercises so many technical muscles.

The number of oddball, often critical, places where I've found SQLite being used would defy belief. As far as I can tell, the "expected" place for SQLite to work seems to be almost anything that's not your normal dB driving some web-based CRUD app...all kinds of embedded systems, easy to manipulate in-memory scratch pads for bioinformatics, lots of data analysis tools in mobile communications.

It's so good, and so obvious, that I think sometimes it makes other tools that might be simpler fits for many use-cases less likely to be used, like leveldb.

View on HN

> your normal dB driving some web-based CRUD app

That can totally be handled with SQLite.

I have several crud apps running sql. With moderate write load and a good concurrency write error handling code it works very well. Good when the product size is not worth a postgres full blown setup.

I backup this. I've been using SQLite for some low to moderate load CRUD apps, and this always worked like a charm. SQLite also make backuping, testing and moving apps so much easier.

How do you workaround the fact that SQLite only supports a single writer? For example, your app could be blocked when you run a long operation like creating an index.

For moderate load: just catch the exception and try again. It will rarely happen, and when it does, you can easily recover from it since your site is not hammered.

For bigger load, have a worker that does the writes with a queue.

Hello Sam and/or Max,

I'm French too and I read your blog sometimes ;-)

Is the following what you suggest:

You create an index, and the index creation takes 30 seconds. Then instead of writing directly to SQLite (which won't work since the index creation blocks other writers) you suggest to store the write in a queue (for example another SQLite database for durability), and have a worker apply the write to the main database when the index creation is done?

Unless you want to scale. I'm all for the rule of least power, but as soon as the app is exposed to multiple users I would ask and be sure about the expected number of simultaneous users before going with SQlite instead of going with a Client/Server RDBMS. Still, the bound is pretty high if you keep your transactions short.

If you have the correct architecture, multiple users shouldn't matter at all-- I have used SQLite to store and query activity data with something like 10k concurrent "users" without difficulty on a single machine.

I am beginning to suspect that MySQL, PostgreSQL, DB2, Oracle, BigTable, and others allow one to get so far with the wrong architecture, that maybe some even very experienced programmers believe that to go faster they have no choice but more threads.

Can you describe the right architecture?

How do you workaround the fact that SQLite only supports a single writer? For example, your app could be blocked when you run a long operation like creating an index.

SQLite supports concurrent reads and writes since 2010 with the introduction of the "write ahead log"

https://sqlite.org/wal.html

Yes, but with a single writer at a time.

From the link you shared: "However, since there is only one WAL file, there can only be one writer at a time".

Not if you need to use multiple app servers.

Reply on news.ycombinator.com