Why IndexedDB is slow and what to use instead | Hacker News Reader

Why IndexedDB is slow and what to use instead

rxdb.info

105 pointstypingmonkey45 comments

I ran into the IndexedDB slowness problem while adding support for it to Converse.js XMPP messenger[1].

I'm using localForage which lets me persist data to either sessionStorage, localStorage or IndexedDB (depending on user configuration), however writes to IndexedDB were excruciating slow.

I didn't want to give up on it however, because localStorage has a storage limit that power users were constantly coming up against, and IndexedDB is also necessary for progressive web apps.

Eventually I solved the problem by batching IndexedDB writes with a utility function I wrote called mergebounce[2]. It combines Lodash's "debounce" and "merge" functions to combine multiple function calls into a single one, while keeping and merging the data that was passed to the individual function calls.

That way, you can call a function to persist data to IndexedDB many times, but it only executes once (after a small delay). I wrote a blog post about this approach[3]. It's generic enough that anyone else who uses localForage for IndexedDB could use it.

1. https://conversejs.org 2. https://github.com/conversejs/mergebounce 3. https://opkode.com/blog/2021-05-05-mergebounce-indexeddb/

Thanks for the post, just a heads up the localForage link in your blog post leads to a 404 page.

Fixed, thanks!

> IndexedDB is also necessary for progressive web apps

localStorage doesn't work for progressive web apps?

Depends how complex your needs get, IMO. localStorage is a key/value store, IndexedDB is a (basic) database. (localStorage also ties up the main thread whenever its in use, which also becomes relevant if you're using it a ton)

It works the same way the glove compartment in your car works when driving for a holiday / to see relatives....

You grab stuff easily out of it, but not much'll stay in there.

Local storage is typically 5mb.

Not sure what PWA would survive on that.

I see, it has a lower quota and only IndexedDB has a mechanism to request persistence.

Thanks.

This mentions Absurd-SQL[0] which I only discovered recently. It’s seriously clever, worth reading the blog post[1] to see how it works!

If you are building an offline enabled web app for the cost of a 1mb wasm download it’s worth it for full sql support and the speed increase.

I think there is a good chance Absurd-SQL will get folded into SQLite.js.

0: https://github.com/jlongster/absurd-sql

1: https://jlongster.com/future-sql-web

2: https://sql.js.org/

What kind of sorcery is this!? (WASM)

Looks like only transactions are "slow", where "slow" means 2ms, which is perfectly acceptable since it needs to wait for disk commit.

So the problem is that the author is using it wrong, since you only need to wait for transaction commit when you need guaranteed durability, and you only need guaranteed durability when you are communicating success to the user or some other system, which only needs to happen every frame, which is 16ms at 60fps which is plenty of time to wait for a 2ms transaction commit (network communication can also be throttled to the framerate).

So in practice all you need to do is batch writes in a single transaction per frame and there is no problem.

And of course if you don't need durability you can just keep the data in the JavaScript heap (and use inter-tab communication if needed).

I'm more interested in read speed than write speed. I have about 2 MB of data that I fetch, parse and transform into a nested object for easy look-up by various types of keys. It consists of 6 other objects, and I'd guess it's < 50 MB in total size.

In my brief experiment, it was 12% faster to read from the web Cache API [1], re-parse and re-transform that nested object than to read the fully transformed object using IndexedDB via idb-keyval [2]. That surprised me! I went on to learn that IndexedDB does a structured clone as part of such reads, which I suspect is the main cause of slowness in my use case.

Related commits to reproduce that finding are in [3], specifically [4].

[1] https://developer.mozilla.org/en-US/docs/Web/API/Cache

[2] https://github.com/jakearchibald/idb-keyval

[3] https://github.com/eweitz/ideogram/pull/285

[4] https://github.com/eweitz/ideogram/pull/285/commits/90e374a0...

The conclusion of this article is "do not use IndexedDB as a database", but the reasoning is "if you bulk-insert by creating a transaction for every operation, it's slow".

If you were using IndexedDB as a database, you wouldn't bulk-insert data in this way, as it's the slowest way to do it.

The conclusion should be "use IndexedDB as you would use a database".

It isn't how you would do it if you were bulk loading a bunch of pre determined data, but if it were a bunch of random business transactions in an OLTP scenario that is pretty much how it would happen. The fact that IndexedDB scales poorly in the number of transactions it can handle is an important data point when considering what tech to use.

Well, that hypothetical OLTP database almost certainly has many concurrent clients, right? Your single-threaded JavaScript app is not going to be generating the same usage pattern.

> if it were a bunch of random business transactions in an OLTP scenario

Please provide a real world example of this in a client side/PWA context for the class...

I am still trying to figure out why anyone wants a database in their document display program. As far as I am concerned browsers were complete a while ago, and devs are just adding random stuff so they can keep getting a paycheck.

Unless I missed it, I assume the article is talking about the Chrome implementation of IndexedDB. As far as I know the performance issue with their implementation vs. say, Firefox’ one is a known issue[1].

Other comments mention using batch operations to speed-up IDB but maybe Chrome could also fix their implementation in the first place. It seems, though, that the Chrome team had no intention on improving the write performance of IDB as of 2019[2].

Edit: it seems that Chrome team did improve the performance of IDB since 2019 but it is still not at Firefox level of performance.

[1] https://dev.to/skhmt/why-are-indexeddb-operations-significan... [2] https://bugs.chromium.org/p/chromium/issues/detail?id=102545...

I don't understand what the big deal is with grouping a bunch of writes into a transaction? Isn't that how you use a DB?

That's the first thing I noticed when reading, it's complaining that if you don't batch your writes the performance suffers substantially.

It warrants an audible "duh". More work is always going to run slower.

I don’t think you should be expecting a local in-memory db to require batching for reasonable performance; the main reason it’s a problem for normal DB’s is the network — you’re constantly waiting for the roundtrip.

Which you obviously still have for an in-memory DB, but you should expect it to require a lot more picked low-hanging fruit before that becomes your bottleneck.

IndexedDB isn't an in memory DB?

Each transaction typically has a non trivial amount of overhead as committing it requires you to verify that there wasn’t a conflict.

Why would you expect that you wouldn’t need to do this just because it’s an in-memory database? You might still be modifying it concurrently through multiple tabs open to the same site modifying the same DB or because you are interleaving “concurrent” actions for some reason within a page. Has nothing to do with I/O to my knowledge.

Try this experiment out with any transactional in-memory tables on any database and they should all show some amount of degradation (although maybe not necessarily quite as severe as IndexDB - 10k inserted rows/s seems below what SQLite should be putting out).

I am fairly certain that the reason you want to batch writes in a normal database is not because of RTT latency, but rather transaction overhead; e.g. indexes have to be updated only once, rather than thousands of times. Disks have to be flushed only once, etc.

I am fairly certain the issues are similar for IndexedDB.

Sibling already pointed it out, but it's worth clarifying: IndexedDB is most definitely NOT and in-memory database.

Hm, but what if I have lots of transactions, going to lots of different object stores? Because every object store requires its own transaction.

(grouping there does not seem to help)

The solution would probably be, to not have lots of object stores, but this would have made my db setup much more complicated.

But FileSystem API seems to finally getting broad support.

You can "open" multiple stores in a transaction so there's no need to have dozens for a dozen stores. There's some onus on you to work out the batching as IndexedDB is quite low level for what we usually get in the browser.

> But FileSystem API seems to finally getting broad support.

I'm not seeing much beyond Chrome & it's associates for it?

That said that could just be because "filesystem" as a word is mentioned in a dozen different web APIs from allowing file input reading locally, to the deprecated API chrome had for local file access, to the new thing that came out of "project Fugu" for PWAs to have native file access and I'm missing which one is which in terms of support & docs.

Thanks, I will look into that. (Is this a recent addition to the API, or did I just missed that option from the beginning? My db code is pretty much unchanged since a few years)

And yes, file system api is confusing, I was also unsure for a moment to have looked at the wrong API, but I was refering to there, the local sandbox filesystem:

https://developer.mozilla.org/en-US/docs/Web/API/FileSystem

Which says it mostly works now, but I never worked with it, because of chrome only, despite it seemed so much better suited for my use case than indexedDB.

> When the window fires the beforeunload event we can assume that the JavaScript process is exited any moment and we have to persist the state. After beforeunload there are several seconds time which are sufficient to store all new changes. This has shown to work quite reliable.

On the contrary, I've found onbeforeunload far from reliable (albeit my use-case was a bit different). You should probably not rely on it for anything important.

This has been my experience as well. At a prior job, we experimented with deferring reporting/tracking functions until the browser was idle, and then firing the remaining tracking events on unload and found that a non-trivial number of events were just dropped (I believe due to this issue).

Funny, because that was almost exactly my use-case as well. I guess sending network requests during the onload event doesn't work well, but maybe for local things it works better, who knows.

One thing I don't see called out as an option: CacheStorage (window.caches). Just store your data as Response. Fast, easy, available in Worker scope...

Not apparently all that durable, though, especially when you hit the storage limit.

Doesn't solve rich data though as neatly as IndexedDB does.

Because IndexedDB uses structured clone, it can store javascript data as is, which includes storing correctly a number of interesting/useful types, including ArrayBuffers, Files/Blobs, CryptoKey instances (useful to avoid needing export flag on them), the cost of this is the performance hit compared to JSON.parse on a string.

Also the cache is less persistent than IndexedDB when storage pressure increases.

Not to take away from the points made in the article, but isn't inserting, hundreds or thousands of documents into IndexedDB at once kinda of an edge case?

Not only that, it's doing each insert in a different transaction, which isn't what you'd do for bulk-insert in any database.

I think that developers are constantly surprised and frustrated when writing data to IndexedDB takes much longer than downloading it. High transaction cost turns out to be one of several performance surprises with Chrome IndexedDB. If a developer discovers these one-by-one, then they will probably need to redesign a few times if they want IndexedDB to support a data-intensive or offline application.

Alternatively, there's localForage[1][2].

1. https://localforage.github.io/localForage/ 2. https://github.com/localForage/localForage

LocalForage is similar slow as IndexedDB

https://pubkey.github.io/client-side-databases/database-comp...

Maybe there is a case for LocalForage to use Absurd-SQL as mentioned elsewhere in the thread (https://news.ycombinator.com/item?id=29314766#29315976). It's data access layers all the way down...

I get the reasoning behind the browser makers' decision, but I really wish Web SQL had become the ubiquitous option. SQLite doesn't make a standard, but pragmatically I think it would have been a better option than how IndexedDB turned out.

A declarative storage syntax for a declarative display engine. Seems a good fit.

IndexedDB is pinnacle of bad API design and I'm not surprised that people will shoot their own feet with it.

I wonder why they don't mention dexie with all of the other options. Like they say, batched operations solves this handily and Dexie has great support for them and it's really darn fast.

It's kinda docs / PR / Look how great OUR solution is, not an unbiased piece...

I fixed some of the slowness issues in my application by writing a nice little cache adapter between my application and Dexiejs.

Reply on news.ycombinator.com