Better-sqlite3: A faster Sqlite library for Node.js
github.com
github.com
It's definitely a good example of buying into the "hype" of async without really understanding what's going on. Async is helpful when you need to wait on something in the future (like new packets come in from a socket) but don't want to block the whole process while waiting. It doesn't make sense at all for CPU work that's going to block it regardless.
I've lightly contributed to both better-sqlite3 and node-sqlite3 and the latter's async implementation makes it much more confusing and difficult to work with. And it slows things down considerably.
I switched node-sqlite3 with better-sqlite3 in my Electron app and found non-trivial performance gains. Make sure to run it in a separate process - never run sqlite in the main thread or the same renderer process as your app. If you run it in the main thread, it still blocks the renderer process. I wrote an article about this: https://medium.com/actualbudget/the-horror-of-blocking-elect...
I'm glad somebody finally made a robust synchronous API to sqlite.
What that article is saying is that libuv exposes a thread pool that you can opt in to with a native extension instead of bringing your own.
It doesn't mean a synchronous API won't block.
For example, https://nodejs.org/api/crypto.html#crypto_crypto_randombytes... without the callback blocks.
You said "Just because SQLite's API is synchronous does not mean that it must block the node process." I'm saying that's not true.
A function might use threads behind the scenes, but if you're going to synchronously wait for it, you're blocking the event loop.
I retract my comment about "no matter what", but it still just boils down to running sqlite on a separate thread. In my opinion it's probably a lot easier to run sqlite in a separate thread (or process) yourself at the app-level. The code for writing async native bindings in node is very complicated (not even considering the thread pool), and you probably don't need every single operation to be async. That just adds a lot of overhead, when you probably just need a higher-level layer that says "execute these queries on the sqlite process and give me the results".
"Asynchronous system APIs are used by Node.js whenever possible, but where they do not exist, libuv's threadpool is used to create asynchronous node APIs based on synchronous system APIs."
Isn't that entirely unsurprising?
The point of async is not that it's faster, it's that other shit can run while you do your IO. Of course your sequence of operations is faster when you never yield to other tasks.
node-sqlite3 uses an asynchronous API for CPU-bound functions.
An asynchronous API is only for IO-bound functions. CPU-bound functions should be synchronous.
It's definitely a bastardization of the terms a bit, but at its core it's basically a message-passing multi-process system that hides the implementation behind the well-understood (to most programmers that use node) async i/o
With async-await being common now, a synchronous API is also not required to have an easy-to-use API.
node-sqlite3 does have a massive performance problem when inserting 100 rows in a transaction - the overhead on the JS side can be around 10x more than the time required by SQLite. However, all that's needed to solve this is a batch API: a way to run 100 insert statements in a transaction with only a single callback at the end. The performance problem is really not about synchronous or asynchronous - it's the fact that we need the same overhead for every single insert in the transaction.
I've eventually worked around this issue by just combining multiple insert statements into a single one. For example, instead of doing 3 separate inserts, do one with `VALUES (?,?), (?, ?), (?, ?)`. While the code to achieve this is ugly, it gets similar performance to better-sqlite3, without being synchronous.
Who the hell decided that making sqlite asynchronous is a good idea in the first place? Node is so religious on async that creating everyday transactioned apps in it is PITA that never ends.
It certainly isn't the case for nodejs.
One could wonder whether that's an issue when IO are fairly short & strongly interspersed with code being run though, the overhead of switching tasks could be higher than the IO you're synchronising. I guess whether this is a good idea would strongly depend on how you're using sqlite (aka do you tend to make lots of very simple requests with small output or a smaller number of more complex ones).
IMO one of the core observations made by Dahl and the other Node.js folks was that this is very very rarely the case in modern web applications. Most webapps either a) spend the vast majority of their time waiting on IO, b) have non-IO code very regularly interspersed with IO operations (making event-loop based interleaving based on IO boundaries more useful) or c) both.
As a result, Node.js was built with the single-thread/IO-bounded reactor model. I think that this, along with (and perhaps even more than) its choice of language syntax, drove the platform's adoption. It means that code doesn't need to think about thread safety, and that there is only one IO model: asynchronous by default (yes there are exceptions, but they're a minority). That lets users get a lot of scale for free, provided their workloads meet those criteria.
If your api is not asynchronous, then any expensive call is going to lock the entire application which is bad and also why I won't be using this library despite the actual underlying implementation being much faster.
Did you measure the impact, subtracting harder development and run- time? Do you like to code in all hard ways before you meet any problems? Is sqlite a key performance point in your app? (Why not async-pg then?) How many requests per second do you have at average peak hours?
For example, that almost everything in Node is asynchronous, like in Go, is one of its best attributes. Compare that with the state of things in Java, Rust, Python, and most other languages if you decide you want to use the reactor pattern.
If you don't want an event loop, then you don't use Node. But not blocking the event loop is not "religious." It's how they work.
What if node-sqlite redesign it's library and gets even faster than this? Then it won't be better anymore? I think it's a bit of a bad name for a lib to have since stuff like this changes with time.
Anyway great work!
Not sure how I feel about it not being async...
Elm got this right. You shouldn't have to be clever in thinking of a package name just because the obvious one was taken and you have a different flavor to offer.
Rust didn't and it's another ecosystem where people feel the need to squat on names or suffix a '2' when they think they've written a better library, just like NPM.
If it's feasible to do so, you also push updates to both namespaces for a little while during the transition.
For example, instead of having to push updates to both packages, you push updates to the new one and, because you've annotated the old one with the move, users have a somewhat seamless experience when they run their update: they'll get a little heads up about the move, explain that oldpackage eol'd at 5.4 and newpackage picked up at 6.0, and would you like me auto-update your manifest?
And/or the package manager supports a rename flow that does this for you.
Github has some good examples of this when repos change hands.
I'm having the exact problem you're describing in a macOS app, where an app that was previously name.janedoe.app is now org.company.app. What a mess that caused.
The Java space, Maven, more precisely, did come up with the first namespaces package repository. For some reason other package managers don't do it, for simplicity I guess, and it always come back to bite them in the butt.
For reference, Maven 1 didn't have groupIds, Maven 2 added them. They did it for a reason and that was back in 2005!
Did you think nothing can be improved upon Java?
It's not that great unless it's required. For example, the bare "sqlite3" package name will always sound official/canonical next to "@JoshuaWise/sqlite3" and people will forever accidentally install it even though they want a scope. Just like how users type in domain.com when they really wanted domain.org.
It doesn't really work until FQN is required by all.