Hacker News Reader

Comment by techtalsky | Hacker News Reader

I like to take every possible opportunity to sing the praises of SQLite, which is one of the most widely deployed RDBMS's in the world. Not only is it open source, but it's public domain, and it's an incredibly stably developed and well-tested piece of software.

Please note that it's not for every application. It is not a client/server architecture. It simply a tightly optimized piece of C code that interacts with a file (or RAM) to create a lightning fast approximation of a SQL server that is ACID compliant.

One thing I've used it for the most, is for small, ad-hoc internal web applications. With the webserver as its "single user", it is incredibly speedy and reliable and holds a shockingly large amount of data (still works pretty well when data gets up into the Terabytes).

View on HN

> it's an incredibly stably developed and well-tested piece of software.

It also has a policy of safe by default, fast at your risk (mmap is itself an example, it is disabled by default), which is the only sane attitude when working with databases.

Contrast this to "modern" databases such as MongoDB, which is crazy fast out of the box, but then you start enabling durability, synchronous updates, etc... it becomes slower than standard DBs. This makes you win benchmarks, but in the long term causes a lot of headache and drives users away.

   it becomes slower than standard DBs

Sorry but either you're ignorant or being deliberately disingenuous. Either way this statement is completely untrue for many use cases. Can you not imagine situations where a document database would be orders of magnitude faster than ANY SQL one ?

Think about a document with hundreds of embedded documents which equate to joins in SQL.

You can use SQL as a document store, you don't have to use joins.

MongoDB queues inserts into memory before flushing it to disk. This means it's not durable in case of a power outage before the data is flushed. Turn on the safety of flushing-first and it's more or less doing the same stuff your sql software is doing.

Is is, in fact, much much slower than both SQLite and Postgres. It's even slower than a fully indexed hstore in postgres! (which is pretty much identical to mongo's data model, except consistent, durable, with transactions, etc.)

That doesn't make sense. Relational databases use relational algebra (which includes joins) to model data. The mere fact that you model it that way doesn't necessarily have to affect the physical representation. True, there is no such thing as an actual RDBMS on the market, but that's hardly Dr. Codd's fault. :-)

> which is one of the most widely deployed RDBMS's in the world

Being inside Chrome, Android, Firefox, Skype and other places (http://www.sqlite.org/famous.html), I'd argue that SQLite is by far the most widely deployed RDBMS in the world :-)

And, BTW, I love it too.

It is interesting how SQLite is so well known but de SCM built on top of it, Fossil, and by the same author did not catch so much.

Sqlite filled a desperate need with no contenders, while Fossil, released in 2006, was going up against git and hg (both released a year earlier). Though Fossil is much more than a SCM (wiki, bug tracker, blog) and while not mainstream, is still used by tcl.tk and others.

SQLite itself is hosted on fossil, as is Tcl/Tk as you mention (and it's various satellite projects), and I know NetBSD was flirting with it too (no formal decision made yet on migrating from CVS afaik). I use fossil for all of my personal projects. It's more than capable, and a joy to use. Everybody should try it for a week. http://www.fossil-scm.org/

It is also an excellent tool to have in your command line toolkit. Rather than | wc | uniq | sort etc, pipe to file and bring into SQLite for more extensive querying.

That's pretty interesting. Can you give an example?

Just make a quick, loose, all-text-fields schema and dump your data as a CSV: http://www.sqlite.org/cvstrac/wiki?p=ImportingFiles Then you can run SQL queries against it using the SQLite commandline tool: http://www.sqlite.org/sqlite.html

I've a different but related use case. I want to make a 5 GB file in a specific format, which requires that the data be sorted. I want it to run on my laptop machine, which has 4 GB of RAM. I dump the unsorted data to a sqlite database, tell it to use up to 1 GB of cache, create an index on the sort field, and pull back the sorted data, which I can stream directly to the file.

What about simply using the original NoSQL system? http://www.strozzi.it/cgi-bin/CSA/tw7/I/en_US/nosql/Home%20P...

I'm using Python, which has had sqlite as a standard module since Python 2.5. Therefore, I don't need to install any third-party module.

NoSQL provides command-line programs. sqlite is embedded in the same process. Error reporting across exec boundaries is much more complicated.

NoSQL requires standard unix command-line programs, and for at least one case "This program has only been tested with GNU sort" This makes portability to other OSes, specifically MS Windows but perhaps also OpenBSD, harder.

Hence there's no "simple" about using NoSQL for my use case.

One potential pitfall of SQLite is that datatypes declared in your CREATE TABLE statements have purely indicative value. You can in effect store strings or booleans in an integer column. I'd welcome a 'strict mode' which would do away with weak typing and behave more like traditional SQL databases.

I love using SQLite for websites. Unlike MySQL, SQL is a well-written and predictable database, there's no daemon, users or permissions to worry about, and the runtime is built into Python!

You have to be careful though, as SQLite doesn't support concurrent read/writes, so it blocks, which makes it rather difficult to scale up for websites, or at least it has in my experience with it.

Yes. In the Go language one can use channels to funnel the data to/from a single process that does the SQLite reads/writes. It's trivial to batch up data in transactions as it comes in from users (the current channel length is the number of rows to include in the transaction).

Could you explain this a bit more, or point me to a tutorial? That actually sounds really promising.

This is how you'd perpetually write data from all users using a single process, batching the data into transactions for better performance; there's not much new here that isn't covered in most tutorials: On app init create a channel into which data will be put as it comes in from users, and start a goroutine that is passed the channel. In the goroutine have an endless for loop, inside of which you wait for data to arrive in the channel, using a select statement. When data arrives start a SQLite transaction, then iterate the length of the channel in a for loop, writing rows to the SQLite table. (The length of the channel is the number of data values currently in it, which will vary depending on how fast you're putting data into it from elsewhere in the app.) After that loop you commit the transaction.

It supports concurrent reads/single writer.

I have to agree wholeheartedly. I find SQlite a great choice for web apps and small DBD websites as it's so much easier to deploy and more than fast-enought for most projects. Plus it is easy to upgrade if the need arises as the queries are for the most-part identical to SQL.

Yes, and I don't understand why Wordpress does not support it officially and we need to use MySQL instead.

I think WordPress just tries to cut complexity by using MySQL only so it doesn't have to deal with support for arbitrary RDBMS's.

More likely, the code is poorly written and heavily reliant on MySQL.

This is more or less the correct answer for Wordpress-the-ecosystem.

Wordpress-the-base-install has a thin abstraction layer over the database in the $wpdb object.

But plugins and themes can, and regularly do, use PHP's MySQL functions to access and manipulate the DB directly.

The theoretical answer is because SQLite is not the right tool for the sort of things Wordpress does. SQLite supports more concurrency than one might expect from an embedded db - it's still an embedded db, though. Writing, for instance involves a database-global lock.

The practical answer is - Wordpress is written in a MySQL-specific way. It doesn't officially support, say, Postgres or Oracle either.

> The theoretical answer is because SQLite is not the right tool for the sort of things Wordpress does.

Which probably isn't true.

SQLite is bad at concurrent writes. Your average wordpress install is almost only reads. And with 3.7.0's WAL, readers and writers don't block one another anymore.

The trouble is that there are many, many Wordpress sites out there that are not an average Wordpress install. People (disturbingly) write e-commerce sites on top of Wordpress. SQLite is not the right tool for the sort of things Wordpress does not because you couldn't write a blogging tool using SQLite but because Wordpress is crazy.

If one could write blogs and ecommerce sites using MyISAM (shrudders) then why wouldn't SQLite work? MyISAM had -- at least in theory, I have not done any benchmarks -- even worse concurrency than SQLite.

I hope I've completely forgotten any details of the convoluted locking behaviours and tuning options of MyISAM (which, in my head at least, is read as 'miasma') but they were there to at least attempt to support certain types of concurrency. And they were table-level whereas the SQLite write lock is db-wide.

Could you reasonably write blogging software that uses SQLite? Sure. It's just that Wordpress isn't that blogging software - it's written to be paired with MySQL, for good or ill.

MyISAM has reader/writer table locks[1] which meant that nobody could read while someone was writing to a table.

SQLite can be ran in two modes 1) database reader/writer lock or 2) in WAL mode[2] where readers do not block writers and writers do not block readers. Writers block each other though so there can only be one concurrent writer. WAL mode provides concurrency superior to what MyISAM did.

  1. The locks could sometimes be bypassed by inserts, but not reliably so.
  2. http://www.sqlite.org/wal.html

Sure but WAL in SQLite is 3 years old. Wordpress is 10 years old. MySQL with MyISAM is a lot older than that. I personally happen to think almost everything about Wordpress is wrong. It's just hard to blame them they didn't start or subsequently adopt SQLite. I doubt it's ever been as much as on their radar.

I believe you can approximate table-level locks in sqlite by splitting your database into multiple files. Transactions across multiple databases are atomic (provided you don't use WAL), so you're still in a fairly safe place. However, I'm not sure how practical it is to split a database into lots of files; or whether the sqlite engine deals efficiently with queries that span many databases, for instance - I've never actually done this.

But if you really need a particular table to be writable without locking the rest of the database, it's probably a fairly decent workaround.

Agreed, and I do not think the lack of SQLite support in Wordpress has anything to do with what SQLite can or cannot do. Only that it was written by LAMP developers who do not care for databases other than MySQL.

Given how easy it is to install Wordpress such that it will fall over at the slightest breeze of traffic, and it appears to be installed that way by default, it's hard to see SQLite making things any worse. Maybe it wouldn't handle crazy situations out of the box, but WP is already in that situation anyway.

WordPress doesn't support anything except MySQL because all the queries are specifically written for it. Developing for a single database enables the queries to be optimized for that database.

(That's the official WP line anyway, I'd love Postgres support myself.)

Reply on news.ycombinator.com