LokiJS – In-memory JavaScript Datastore with Persistence
lokijs.org
lokijs.org
What is important is being clear about what trade-offs they make.
As an example: Years ago I worked on a classifieds aggregation engine; we crawled hundreds of thousands of feeds, indexing tens of millions of listings.
All the queues in our processing pipeline were in-memory only: if we lost a few items (or a million) it didn't matter; worst case we'd pick them up no later than 24 hours later - in the case of more severe loss we'd simply trigger re-indexing of the feeds likely to have been "in flight" at the time. Failures would affect freshness and cause us to lose out on some updates, but as long as they were not severe enough or too frequent nobody would even notice, because the inventory was large and fleeting.
There are lots of cases like that where "performance over everything" can be a totally valid trade-off, because your data store is not the canonical source of information, doesn't need to be precise or comprehensive, and/or can be easily refreshed, for example.
This does require you to know and understand your requirements and what the datastore in question guarantees (or not), of course. If you don't, then you better stick with a "safe" option.
And even with that trade-off, I'd argue that there's a level of correctness that should be non-negotiable. Missing data might be OK, but corrupting data on the way in is bad, and failing to process everything might be OK (eventual consistency and all that) but you should meet your published guarantees and you shouldn't corrupt your results.
So I would argue for correctness over performance, with the caveat that "correct" has a technical meaning specific to the project.
And this can be done without compromising the storage or integrity of the data. The Observable pattern lets you update caches while you perform writes so the data stays correct, reflecting what is on disk. If caches don't exist then you pull from disk (slow) and write to the caches.
Honestly the wisdom I have has just been playing around with benchmark.js on V8 and noticing the Chrome team doesn't do as they claim (although latest Chrome Canary we saw major improvements), while FireFox is a different beast. You've probably done about just as much of this as me.
What we found was that every function call in javascript significantly decreased performance (even if it did nothing) at a quazi logarithmic rate. So on a theoretical level if you're able to reuse function results that are deterministic you can save a ton of overhead. However IDK how micro-optimizing you want to get as it can sacrifice code legibility.
Would love to chat more if you have time, I'll see if I can find your email and send you a message. Mine is (no spaces): a q u i v a @ google's email provider.
Well, no shit?
Is the syncing part as straightforward as Pouch[1]? I couldn't find anything about syncing on its docs. If a db comes with an easy offline-first and sync-when-possible model with granular permissions, it will be very interesting. Current solutions have (IMHO unacceptable) fixed dependencies on the backend or the architecture.
[1]: https://pouchdb.com/
Pouch, Firebase, and GUN[1] (disclaimer, I work there) are examples. If you bake permissions into the system itself, you get monoliths like Meteor. But you can still achieve granular permissions by rejecting/accepting network requests (like with ExpressJS or Hapi, etc.), which gives you the balance of using whatever backend you like.
Who wants to replicate their "whole" database locally?
So, probably no-one wants to replicate the whole database locally but there is a strong benefit to replicate parts of it for performance and offline-capabilities.
Disclaimer: I was one of two developers on the mobile app mentioned above.
What I actually want is MeteorJS with neither MongoDB or the JavaScript backend, or PouchDB without Couch, or Jaydata + ASP.NET WebAPI with more open source compatibility...
Have you noticed such quirks building loki? Do you imagine being able to make full use of, say, a 64gb ram server? Is this something being taken into account and optimized in the project somehow? @joeminichino
I was storing time series data.
I ended up moving to redis which has worked very well as long as I make sure it has enough memory for the date range I am keeping.
I hope this project keeps at it and gets a handle on performance under heavy load.