Show HN: IronDB – a resilient key-value store for the browser | Hacker News Reader

Show HN: IronDB – a resilient key-value store for the browser

github.com

73 pointsgrun27 comments

> When a value is retrieved via its key, IronDB... > Looks up that key in every store. > Counts each unique, returned value. > Determines the most commonly returned value as the 'correct' value. > Returns this most common correct value > Then IronDB self heals: if any store(s) returned a value different than the determined correct value, or no value at all, the correct value is rewritten to that store. In this way, consensus, reliability, and redundancy is maintained.

I'm not sure any of these properties are ensured the way anyone would want. Why do this? What are the failure modes this protects against? If only one of the backing stores is active when a result is written, and then the others later become active with no data, is blank data returned for the prior result? It seems like recording timestamps would fix this problem nicely on a single system and make this thing overall quite reliable.

Cookies are frequently cleared by users, and the other datastores -- IndexedDB, LocalStorage, and SessionStorage -- can be unceremoniously purged by the browser under storage pressure.

See my other comment, here: https://news.ycombinator.com/item?id=18297177

I'll explicate this in the documentation. Thank you for your feedback, bazizbaziz.

Thanks for the response! FWIW I'm overall really interested in this, as I maintain an application that uses localStorage to keep very important data while the app is offline until it can be uploaded.

I recently tested localForage [0] to see if it could be more reliable and get around storage limitations of localStorage. Unfortunately, LocalForage has a very annoying problem where it picks a single storage backend to use, but the one it chooses can switch on page reloads (yes, even on the same device/browser.) and this switching causes data loss as keys stored in the other backends are unaccessible. I'm very interested to see if IronDB can help here! Thanks for working on this.

[0] https://github.com/localForage/localForage

Wonderful.

Don't hesitate to let me know if there's anything else I can help with, bazizbaziz.

What if users don't want you taking control of their computer?

> Why do this? What are the failure modes this protects against?

I have a use case for this: I've written what is basically a crud web app for a non-profit. That's an offline app (appcache) and right now uses localstorage for keeping data offline in the browser while in the field. That is pretty brittle as is, but dedicated application installations didn't fit the budget. The app is used in the field by volunteers all over the country. With volunteers fluctuating and many people being privacy conscious, it happens regularly that they change Firefox preferences to delete cookies etc. on shutdown, and now newer Firefoxes deprecating localstorage, potentially resulting in data loss.

Something like IronDB would actually help me.

Awesome.

If you find any bugs, or have any ideas to improve IronDB, don't hesitate to let me know.

Confused. Why would you name your K/V database irondb when there is already a time-series database called exactly that?

Think this is their site: https://www.irondb.io

Yup, I was wondering how the hell a timeseries database would run in the browser.

Nope, this is just syntactic sugar on top of calling (up to) 4 JavaScript functions.

Maybe this is useful tool, but several things strike as weird to me:

- save all values everywhere: why not select the one that would last the longest (that people delete least often)? Or maybe two, if you want a cookie but also want to survive over "delete all cookies". But then again, if a user wants to delete cookies then you really shouldn't try to keep it.

- "Data is stored resiliently but can also be voluntarily purged if the user designedly clears cookies and application storage." So you delete data when the user wants to delete data. Which database doesn't have this feature?

- doesn't use Flash, Silverlight, or Java -- which do?

- "healing" is usually used in the context of distributed computing, but here it means that it gets around user wanting to delete stuff from it's browser, but perhaps isn't in-the-know enough to delete everything but just the cookies.

No offence to the author but all this sound malicious. Perhaps this is why evercookie isn't maintained?

> save all values everywhere: why not select the one that would last the longest (that people delete least often)?

For improved resiliency.

You can also configure IronDB to only use any two datastores of your choice, if you so desire: IronStorage's constructor takes an Array of storage implementations of your choice. See https://github.com/gruns/irondb#api.

> So you delete data when the user wants to delete data. Which database doesn't have this feature?

A database where the data therein can be deleted without warning. Browsers unceremoniously delete IndexedDB, LocalStorage, and SessionStorage under storage pressure. See https://developers.google.com/web/fundamentals/instant-and-o...

> doesn't use Flash, Silverlight, or Java -- which do?

Evercookie, a similar library, uses Flash, Silverlight, and/or Java.

See https://github.com/samyk/evercookie.

> No offence to the author but all this sound malicious. Perhaps this is why evercookie isn't maintained?

No offense taken.

Thank you for your feedback, kreetx. Don't hesitate to let me know if I can answer any other questions, or if there's anything else I can do for you.

It's not clear to me how this is more privacy-respecting than Evercookies, except that it doesn't use browser extensions.

Evercookie's goal is to store data indelibly, regardless of user intention.

IronDB's goal is to store data reliably, e.g. in the face of storage eviction, but not against user intention. If the user clears all browsing data, that willful action is respected.

Thank you for your feedback. I'll clarify this in the docs.

Why do we need a library to store data reliably? Aren't browsers API already reliable? Does the LocalStorage API for example sometimes fails to store its data properly?

Clearing cookies is a common user action. Even for relatively non-technical users.

And under storage pressure, browsers evict data stored in IndexedDB, LocalStorage, and/or SessionStorage. See

https://developers.google.com/web/fundamentals/instant-and-o...

IronDB is resilient in the face of such events.

Thank you for your feedback. I'll add an explanation about such in the documentation.

That's the answer I was looking for, thanks. The intuition I have is that if someone deletes something from localStorage (or clears localStorage altogether), they want that data gone.

What about https://developers.google.com/web/updates/2016/06/persistent...

Persistent storage requires explicit user permission.

IronDB doesn't.

See https://developers.google.com/web/updates/2016/06/persistent....

Great point, though. I'll highlight this in the documentation. Thank you.

Those docs are dated unfortunately. For example, in my tests on multiple android devices and Mac OS the storage limit on Chrome is closer to 20% of total disk size and you don't need to prompt the user for persistent storage like you prompt for access to gps.

If anyone could point to an up to date docs around storage eviction in browsers it would be greatly appreciated. It's been a big hit to our team's confidence in the "use the platform" mentality that this is not accurately documented.

I remember reading a 5MB limit some place (ages ago) and I'm not able to find the reference at the time.

My team has been assuming that is true (for local,session)Storage and so far haven't been bitten.

Only occasionally have we even gotten close to that size. You can also shove loads of data in the DOM.

Edit: one of the old refs - 10MB

https://www.html5rocks.com/en/tutorials/offline/quota-resear...

What’s the point of storing the same data in three distinct storages? What kind of disaster are you trying to avert?

ps: this will back fire real bad when safari and friends add permission prompts for each specific storage that this library fills up in a for loop.

The usecase I've seen more recently is that paywall news sites now store the number of page reads in multiple places. You end up having to clear more than just your cookies to 'reset' the read count.

What are the differences with localForage (or similar solutions)? The fallback to cookies if no localStorage?

https://github.com/localForage/localForage

localForage only uses a single datastore. E.g. LocalStorage or SessionStorage or IndexedDB. Thus, stored data is lost forever if the browser unceremoniously purges that datastore due to storage pressure.

https://developers.google.com/web/fundamentals/instant-and-o...

IronDB redundantly stores data in multiple datastores -- cookies, IndexedDB, LocalStorage, and SessionStorage -- and self-heals if the data in any thereof is deleted or corrupted. For example, stored data is safeguarded in the event the user clears their cookies or IndexedDB is purged due to storage pressure.

tl;dr: IronDB is resilient in the face of data deletion. localForage isn't.

Thank you for the great answer. :)

It's kind of ridiculous that a library needs to exist to guard against this browser behavior, although I empathize with the browser vendors' argument, which is that browser storage should essentially be used for caching and that browsers should somewhat aggressively defend against disk space consumption that most users don't even know about.

It would be nice to have an in-between where if a user authorized an app to permanently consume up to X amount of disk space the browser would allow it. But pulling off that user interaction well (i.e., getting the user to make an informed decision rather than blindly clicking yes or no) seems quite hard.

Reply on news.ycombinator.com