UnQLite – An Embeddable NoSQL Database Engine | Hacker News Reader

UnQLite – An Embeddable NoSQL Database Engine

unqlite.org

85 pointsrayascott56 comments

Hey all. I worked on the UnQLite development team back into 2013 ~ 2014. It was designed to be the primary storage for a home targeting router manufactured by a French tech conglomerate.

The API is broken down into two independent layers plus one for various utility stuff like cross platform mmaped files, RC4 random numbers generation and so on. The first layer serve as a general purpose key/value store for the host application blobs (i.e binary or text data). You can store whole disk files inside a single database to generate a cross platform TAR archive for example.

The second one is the document store layer similar in concept to what MongoDB offer but without the networking layer since everything run in the same process of the host application. Both layers are transactional and are able to recover after some external failure thanks to the SQLite journaling mechanism from which UnQLite is based on.

Since its release in 2013, there was four major bugs including two critical that involves data loss under certain load. All known data corruption bugs were fixed in the last release and no major bugs were discovered a year or so.

The library enjoy particular success among Python and C/C++ developers especially Chinese that used to bombard the original developers with various internal aspect of the library.

I authored a python binding and have some experience that may be relevant.

The library seems poorly maintained in my opinion and might be abandoned. There have been critical bugs leading to data loss. They have a similar project vedis which is like an embedded redis that I think is completely abandoned.

For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options.

https://github.com/symisc/unqlite/issues/41

> For the love of God just use SQLite if you want an embedded database for structured data, even if it's just eav. If you just need key/value there are battle-tested, well maintained options.

You can also use the SQLite Disk Btree as a key-value data store. I've just implemented a solution that does so in C++ with a LevelDB-like api.

You should use the backend Btree api (the same the VDBE/SQL engine uses to store stuff) and create 'tables' the SQLite uses for indexes. The integer-key based is used for ordinary SQL table .. (1 PK = 1 Record) and the blob-key type, where its agnostic about whats on the key. (Thats the one you use to create the KV store)

There are a couple of details, to make it right, but its not that hard, and its a great solution.

The Btree engine in SQLite is quite inefficient, and also unreliable on its own. (Needs the transaction log support to provide reliability.) LMDB blows it away.

https://github.com/LMDB/sqlightning

You've posted the same two-year-old closed issue twice on this thread. You're coming off as someone with an axe to grind.

People often get passionate about data corruption.

I just want to share my experience, which is extensive.

Agree here with the comment nothing compares to the solid codebase of SQLite. This project in my opinion trys to win your trust with QLite in name and does not deliver same quality.

The licensing is not altogether obvious. There seems to be a "one time fee" for downloading. The name sort of implies a connection with SQLite. No such connection exists. There is a parent company, name of "Symisc Systems". Which appears to be just one guy somewhere in France or Tunisia who likes to call himself "we". The code is not too kindly treated by several who have actually tried to use it.

I think I'll pass.

https://github.com/symisc/unqlite/blob/master/LICENSE

It does use a fair amount of sqlite source.

Charging for a 'production build' is certainly unusual.

I'm reminded of when OpenBSD used to sell CDs.

I think the FAQ needs another question: when should I prefer UnQLite over SQLite? When should I prefer SQLite over UnQLite?

That one seems straight forward: when you want SQL use SQLite, when you want NoSQL use UnQLite

From what I quickly gathered, it's just a Key-Value storage mechanism.

The title says NoSQL and the first paragraph on the page confirms that it's more than just a KV store:

> UnQLite is a in-process software library which implements a self-contained, serverless, zero-configuration, transactional NoSQL database engine. UnQLite is a document store database similar to MongoDB, Redis, CouchDB etc. as well a standard Key/Value store similar to BerkeleyDB, LevelDB, etc.

Ah, I didn't read it correctly then.

Choices are often made across many dimensions.

The question I'm more interested in is: what advantages does this provide vs creating my own document style schema in SQLite. Using the JSON1 extension to SQLite, you can even query inside of JSON blobs.

From reading the docs, I could spot the following differences

1. sqllite is 'sql' interface, unQlit is k/v.

2. Sqlite uses btree, unQlite is linear hash( doesn't support range queries).

> UnQLite is a standard key/value store similar to BerkeleyDB, LevelDB, etc. but, with a rich feature set including support for transactions (ACID), concurrent reader, etc.

This is slightly interesting, since the big reason I personally prefer applications using SQLite over those using BerkeleyDB (which used to be very common, btw) is that I’ve had lots of problems in the past with corrupted BerkeleyDB files. Like, almost all the time; if it used BerkeleyDB, it would sooner or later get corrupted. If UnQLite can be a “non-corrupting BerkeleyDB” for those who don’t actually need SQLite, it can serve an (admittedly niche) purpose.

Oh it corrupts... https://github.com/symisc/unqlite/issues/41

> This production-ready package is available for a one-time fee of $20.

Don't fault them for trying to make a bit of cash but going to have to pass since I'm too poor to spend $20 to see if it's a viable replacement for a little project I built on top of tinydb that's getting pretty slow since the db has grown past what a pure python db can comfortably handle.

Or...I suppose they could provide a link from their download page to their github repo for us somewhat lazy folks -> https://github.com/symisc/unqlite

Is there any documentation of the testing/verification process used in UnQLite? One of the things that gives me confidence in SQLite is https://www.sqlite.org/testing.html and it'd be interesting to see what approach UnQLite takes to the same problem.

From the FAQ:

"7. Is UnQLite thread-safe

Threads are evil.[1] Avoid them.

UnQLite is threadsafe and full re-entrant. But in order to be thread-safe, UnQLite must be compiled with the UNQLITE_ENABLE_THREADS compile time directive defined. If you are unsure if the UnQLite library you are linking against is compiled to be threadsafe you can call the unqlite_lib_is_threadsafe() interface to find out."

[1] http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.... "The Problem with Threads"

"Threads are evil" is not a useful remark to make in response to the question "is this library thread-safe"

sqllite has the same warning in their FAQ's :D

https://www.sqlite.org/faq.html#q6

SQLite can actually do grouping and sorting (iirc) in parallel, though it's disabled practically everywhere (Linux distros and other environments shipping SQLite seem to pick compile flags kinda arbitrarily, e.g. json1 and fts4/5 are fairly widespread on Linux, but not included in e.g. Python for Windows).

this is great - in my opinion there's a shortage of key value type data storage engines and there needs to be more competition in this regard. There simply aren't enough free/open source data engines to meet the demand of today's software.

few quick thoughts - how does this compare to leveldb? How would it perform in something like geth or parity as an ethereum client back end data engine in compared to leveldb? do you have a python interface library?

Found a python2/3 bindings library on the github -- https://github.com/coleifer/unqlite-python

It's based on cython apparently, might dust off pybindgen and see if I can do better...

I'm the author of that library. I've worked to make sure all the unqlite features are available (transactions, cursors, jx9 vms, collections) and the interfaces are pythonic. Performance is good. Why not submit issues or patches instead?

> Why not submit issues or patches instead?

Eh, not really a fan of cython...

Been messing with pybindgen for a few hours now and will probably have the thing done by tomorrow, kind of want a drop in replacement (or as close as I can manage) for tinydb since I like their interface.

Really? There seem to be a ton of key/value engines on the market, especially because it's such a simple model to implement.

There is plethora of embedded K/V stores, but most of them only solve the simple to implement parts. That means either no concept of concurrency control or global RW-lock, zero resiliency to power failures, no support for atomic transactions...

If you want on-disk K/V store for critical data you are pretty much stuck with BerkeleyDB.

BerkeleyDB still has corruption issues, and is quite slow. The only k/v store out there proven to be entirely corruption-free is LMDB.

https://symas.com/lmdb/

Unqlite doesn't support in order traversal of data because the default (and only included storage engine) is based on hashing.

Is there an "amalgamation file" like SQLite has for easy inclusion into other projects?

Yes. unqlite.h/c in the root at https://github.com/symisc/unqlite

Sounds pretty similar in target usage to LMDB.

thats log structured storage?

No, LMDB uses a copy-on-write B+tree.

https://symas.com/lmdb/technical/

The name though — what is it supposed to mean? “Un” as in “One” in French with QL Lite or “Un” as in a prefix for an opposite in English with QL Lite (like “No QL Lite”)?

Un SQL = no sql

And a pun on SQLite.

> a pun on SQLite.

And a bad one at that.

I normally don't chime in on HN, but this pun is a crime against humanity.

Not sure, but my first thought was Un(structured) Query Language

> An Embeddable NoSQL Database Engine

So, std::unordered_map?

Looks more like an out of core unordered_map<string,string> with thread safety and persistence to me, if you really want to make that analogy.

Not really. I mean, yes, but it also has a concept of document storage and a php-like scripting language they call jx9.

So what are the advantages of this over say python dicts? Genuinely asking thanks.

The data in a python dict disappears when you turn your computer off. The data in UnQLite doesn't disappear.

I could pickle the dict to a file right?

If you have 1GB of data, and want to change a small portion of it, that would require you to read the whole thing, then modify it, then write the whole thing back to disk. A database only requires a small amount of reading and writing.

This is a good example of someone twisting the term "serverless" to capture attention:

> Serverless Database Engine

> Most NoSQL database engines (i.e. MongoDB, Redis, CouchDB) are implemented as a separate server process. Programs that want to access the database communicate with the server using some kind of interprocess communication (typically TCP/IP) to send requests to the server and to receive back results. UnQLite like SQLite does not work this way. With UnQLite, the process that wants to access the database reads and writes directly from the database files on disk. There is no intermediary server process.

The fact that it's embedded doesn't actually have anything to do with what popular "serverless" architectures are designed to solve, i.e. writing code that does not need to run in a specific server process to be invoked on demand. By their definition of serverless, pretty much any library is "serverless"...

Edit: This is referring to "Classic Serverless" (https://sqlite.org/serverless.html) not "Neo-Serverless" architecture (my definition is more like "Neo-Serverless"). Thanks parhamn for pointing out the distinction.

That doesn't seem fair. The word you quoted isn't even in the section you provided. The use of "server" is interchangeable with daemon/isolated-process here. That section seems like a decent explanation of daemon vs embedded for the junior dev.

Huh? It's literally in the "Serverless Database Engine" section. I can update the quote to include the header.

Even then, "twisting" is a stretch (especially the connoted maleficence). I think most people would realize these are unrelated and there seems to be a quite of precedence for talking about daemon-less as "serverless" [1].

[1] https://sqlite.org/serverless.html

Fair enough, the "classic serverless" vs "neo-serverless" distinction directly addresses my point. I have added this clarification in my original point.

The UnQLite website has described it as a "serverless database engine" since at least 2013 (according to the Wayback Machine), long before it acquired its current buzzwordy definition.

If you ask me, using the word "serverless" to mean "a library that doesn't need a separate server process" makes a whole lot more sense than "a system that runs your code on a server that you don't control".

Reply on news.ycombinator.com