ImmortalDB – A resilient key-value store for the browser
github.com
github.com
> clearing cookies is a common user action, even for non-technical users. And browsers will unceremoniously delete IndexedDB, LocalStorage, and/or SessionStorage without warning under storage pressure. ImmortalDB is resilient in the face of such events.
When a user (especially a non-technical one) clears cookies, their sole objective is to remove their identity and "start fresh". That's the expected outcome and this library violates that expectation. Also, the reason the browser "unceremoniously deletes" some storage areas is because they are not meant to hold 100% persistent data. So in both cases, the "right" thing to do is respect the end goal of the action and not restore it.
Furthermore, this library should not be used for sensitive data but likely will. This is because sensitive data should not be stored in local storage but rather in cookies [0] as they have better security options with "httponly" mitigating leaks via XSS and scope narrowing with Domain and Path. Local Storage is available to any script on the domain including any 3rd party script in the page.
> Strikes an equitable balance between reliability and respect for the user. That's a fancy way of saying "Doesn't fully respect the user".
[0] https://www.owasp.org/index.php/HTML5_Security_Cheat_Sheet#L...
Is it just me, or is that a novel interpretation of "respect for the user"?
"I'm going to continue to track you even after the usual and expected method of preventing tracking in the browser is used, but if you jump through an unexpected hoop that's not documented to the user, then I'll grudgingly delete my tracking data..."
> browsers will unceremoniously delete IndexedDB, LocalStorage, and/or SessionStorage without warning under storage pressure
If you have an app that stores a small amount of data client-side (preferences, data to allow operation while unable to connect) but is not used very often it's stored state could be evicted because of a few large apps because the browser drops data on a "last used" basis because it has no way of knowing that this bit of data might be more important than the data touched more recently. The user might not want a small amount of practical data for an app that they used on their morning commute being evicted because they've touched some large games over the weekend. For a large device this isn't going to be an issue as you are unlikely to be that limited for space, but for a cheap phone with limited storage it could be a real inconvenience.
Any app doing this should have a very easy and obvious "purge my data from this device" option, to maintain the user control. In fact to do it properly right, the local storage should be opt-in ("Do you want to keep some data locally to speed up this app and allow it to be used offline? Yes / Not now / No and don't bother be again") with the app continuing to function without the local data cache.
> I'm going to continue to track you even after
Of course it will get used for this, it would be naive to think otherwise, it is after all essentially a structured evercookie, but that doesn't stop the idea having good uses too. The existence of arsonists doesn't mean you need to ban all uses of fire.
I get why it's doing this, but it strikes me as fairly user-hostile to react to the browser deleting data under storage pressure by duplicating the data so it now takes up 4x the space it used to.
E.G: on my browser, you can only store 81920 bytes per domain in cookies, so if I want the max redundancy, I have to limit myself to 80ko. Besides, cookies are sent in each request, so I want to limit myself. Let's say you want more, local storage is limited to 10 MB. It's not huge.
But the browser doesn't care about the size of the data, only how recent it is. So it may evict yours, even if it's very small, compared to the huge cache of hundred's of MB from the manifest file or the webworker from other open tabs.
With many user hostile and invasive solutions, browser makers should take a stronger stance on clearing all local data simultaneously (cookies, local storage, IndexedDB, etc.). If someone wants a resilient client side data store, apps are one solution. Websites shouldn't be doing this or allowed to do this in a (standalone) browser.
Safari in Private Browsing also used to do this too, which happens to completely break a surprising number of websites.
The user can still clear the data manually (clear cookies and other local data) or through a properly written app (which implements a "remove my records from this device" option, and preferably makes the local storage opt-in anyway).
If you are thinking of using this maliciously then you wouldn't use this: you'd use the already existing evercookie instead as it survives more (at the expense of potentially imparting more load on the users browser).
Don't hesitate to let me know if you have any questions or comments about ImmortalDB. Feedback is how good products become great.
thank you for open sourcing your work
On smaller devices, a few kb of important data in LocalStorage was periodically deleted without user intervention.
> do you believe it will be used for nefarious purposes?
If one is nefarious of heart, https://github.com/samyk/evercookie supersedes ImmortalDB.
Also, if you are storing larger strings in local/sessionStorage, websql or indexdb, you should use lz-string[1] to try to minimize storage impact.
Aside: if you want something with a nicer interface for underlying storage you may want to look at pouchdb[2], which has a localized couch interface and supports remote synching to coudchdb, which is pretty nice.
[1] https://github.com/pieroxy/lz-string [2] https://pouchdb.com/
I'm guessing a mix of localstorage and sessionstorage isn't al that useful (I can't imagine a circumstance where localstorage is being cleared when sessionstorage isn't also or hasn't already been) but localstorage+indexeddb might give you some useful level of protection.
If a user clears all app data for a site/domain then it will be cleared. If you want to work around this, you could utilize resources/assets from other domains in conjunction but it will be more complicated. I find that most of such cases are abused for unscrupulous tracking.
> For example, clearing cookies is a common user action, even for non-technical users. And browsers will unceremoniously delete IndexedDB, LocalStorage, and/or SessionStorage without warning under storage pressure.
> ImmortalDB is resilient in the face of such events.
I guess that’s a good thing. I was worried about the next browser supercookie being made with a library like this.
AKA
>Its goal is to identify users after they've taken actions to not be identified
sigh
Interestingly, neither Apple nor Google were pro-deprecation; Mozilla, and to some degree Microsoft, spearheaded the move to standardize on IndexedDB.
Moving forward now that Edge is based on Chrome, maybe there's hope for WebSQL after all. Not that it will be undeprecated, but if 98% of the browser market supports WebSQL then it's effectively a non-standard standard.
An attempt was made to break down the SQL dialect and exact behavior of that version of SQLite, but the objection was that it's not really great to reverse engineer a standard from a single vendor implementation even if that vendor's product is open source. (eg. what happens when sqlite changes? do browsers stay fixed on an ancient exact version?)
Though many browsers still support it. https://caniuse.com/#feat=sql-storage
function hasSemicolons() {
console.log('hello world');
}
(function() {
console.log('this is broken, JS will call hasSemicolons()');
})()
See also: https://stackoverflow.com/questions/1873983/what-does-the-le...
console.log('this doesn't work!')
(false ? console.log : console.error)('this will fail with a TypeError')
>> Uncaught TypeError: console.log(...) is not a function
console.log('this works!')
;(true ? console.log : console.error)('hello world')Since everything in js is a statement the correct semi-colon syntax for the above is
function hasSemicolons() {
console.log('hello world');
};
(function() {
console.log('this is broken, JS will call hasSemicolons()');
})();I assume, still, it should not be used to store JTW refresh tokens, or anything where secure storage is required.
I see a handful of blog posts on the internet arguing that HttpOnly cookies are a more secure storage place than web storage, because HttpOnly cookies are only vulnerable to CSRF attacks and not XSS and CSRF is easier to defend against. This argument is, as far as I can tell, completely flawed: if an attacker is able to execute XSS, they're then able to use that XSS to execute same-site forged requests - they no longer have to bother with the "cross-site" part of CSRF. They can just fetch your anti-CSRF token with a normal XHR after having executed an XSS, and then they can make arbitrary same-origin requests against your website, each of which gets sent with the HttpOnly cookie.
I do not know of many web apps that store passwords in browser storage.
Unlike passwords, JWT refresh tokens are created programmatically (not entered by user interactively).
Perhaps,I am just missing a link, but I though browsers do not allow an application to programmatically store passwords securely
Storing a JWT in localstore isn't the same as storing a password in plain text. JWTs are indeed non-revocable, but they are also non-durable (ours expire after 24 hours) and cannot be renewed without a refreshToken. Most people also do not rely on JWTs to write important information (like password reset) because there are more secure alternatives for these specific use cases.
As an aside, if someone gets script access to your SPA, you're already very far up shit creek. If your user types in any sensitive information like a password to login you've lost the war. Securing the JWT is the very least of your concerns at that point.
Typically the matching hostname can view local data/cookies though (along with any extensions you have that have access to it - which is most of them). That's why you shouldn't ever copy/paste stuff into the console of a website, it lets people extract your session.
While the HN cookie is HttpOnly, this isn't really putting it in more-secure storage: an XSS attack against HN can just issue whatever API requests to HN it wants, even though it can't easily walk off with a copy of the cookie.