Hacker News Reader

Comment by kevincox | Hacker News Reader

I do find it disappointing that there is no reliable local storage system for the web. I get the resistance to trackers but there should be a way to request permission to store data that isn't deleted except for explicit action by users. It means that you effectively have to store the data in "the Cloud" which means 1. I have to pay for it 2. If I shut down the service you are screwed and 3. I have at least some access to it (encryption aside).

I would also like to see a synced version since most browsers these days support syncing settings and passwords. But creating a generic syncing solution that is actually useful is hard.

View on HN

> there should be a way to request permission to store data that isn't deleted except for explicit action by users

I think that's "request permission to use files on disk". This is in progress as https://developer.mozilla.org/en-US/docs/Web/API/File_System..., though there's still more work before there's a version all the browsers like (Mozilla likes the ability to work with files, but thinks cross-site access should not be included: https://mozilla.github.io/standards-positions/#native-file-s...)

> but it is wrapped together with aspects for which we do not think meaningful end user consent is possible to obtain (in particular cross-site access to the end user's local file system)

This is a really difficult problem to solve, and I get Mozilla's hesitation. I'm also frankly very hesitant about Google leading the charge on this, not because I'm paranoid about them sneaking in tracking, just because I think Google tends to create less thoughtful web specifications sometimes.

But... cross-site file access is really important for data portability and open standards, and Google's current proposal isn't bad, it might be rough but it's definitely workable. Mozilla really should try to figure out a way to move forward on this.

We've seen the difference in data portability between mobile and desktop apps, and a big part of the difference between those two platforms is being able to very easily have multiple sources working on your data at the same time. Siloing data has downsides. It's tough to embrace a Unix-style philosophy without allowing programs to operate on the same data. And having Unix-style smaller webapps that work with each other is a good way of fighting against data silos and in some cases a good way of fighting against anti-user and anti-privacy services in general.

I'd love to see more progress made on this, but who knows how that will work out. Caution is probably warranted for the moment, I'm just disappointed that the language suggests Mozilla would never consider a proposal that included this.

It's also very important that this expose user-accessible file system access and not just a virtual filesystem in the browser; otherwise it just becomes another data-silo in the web browser. This is something that Google's proposal really gets right, and it's disappointing to see what appears to be pushback on the idea that users should be able to open up the directories that a web browser is writing to, inspect the files, and open them or move them around the filesystem, or even write to them from native apps. That to me is an essential part of the proposal.

A few years ago a tried using the browser's IndexedDB for my invoicing app and loaded a 1000 documents into it at a time. It did ok up to about 2-3k documents but choked the browser to the point of being unusable at 5k documents. That was on a Late `09 Mac Mini so maybe newer or more powerful PCs would do better, but that's not an issue at all if you're using CouchDB to store the data on the client side.

I use CouchDB installed on the client side to implement Offline First data storage. This works for Desktop PCs and if you run it on local device that's accessible via your local network, like a Raspberry Pi in-house for example, you can also use it with mobile devices in-house too.

The local CouchDB will sync with the Cloud based CouchDB as soon as they're both online. CouchDB will decide which version to keep and deliver it from that point.

It's certainly not perfect and doesn't provide "real time collaboration" but so far that's been out of reach anyway and may not be a very good approach at all. The notion of several people editing the same document at the same time seems to me to be chaotic no matter how you approach it.

The biggest downside to this approach is that the user has to install and configure CouchDB. I made a simple web app to help with this, but it's a bit too much to expect users to install and configure it.

What we need is a client side DB pre-installed that any web app can access and the data for that app is sandboxed and can only access the DB assigned to it. But it's not reasonable to add that to a web browser. CouchDB can do that now.

`navigator.storage.persist()` will prompt the user to allow persistent storage for your site, so data won't be evicted under storage pressure (for IndexedDB, service worker registrations, localStorage, sessionStorage, etc.)

https://developer.mozilla.org/en-US/docs/Web/API/StorageMana...

That doesn't work on iOS, nor on Safari on Mac for any released version as far as I understand:

https://caniuse.com/mdn-api_storagemanager_persist

The problem runs deeper than what the GP complains about.

If you rely on that, and write important things on the browser storage, a week later your user access some moronic site that doesn't work, and their support tells him to clear his browser cache, your data will almost certainly be cleaned with it.

The problem is that browsers do not even consider that they may be storing important data. There is no clear way to recover or backup that data, and it is tangled with what comes from every other site.

> I do find it disappointing that there is no reliable local storage system for the web. I get the resistance to trackers but there should be a way to request permission to store data that isn't deleted except for explicit action by users.

I've built a number of apps in the last year or two that use browser local storage. It also annoys me that the idea around "storing data in your browser" is automatically attributed to ad tracking. I have to go out of my way in my apps to inform users that yes this app uses localStorage/cookies, but no this is not used for any ad tracking, rather for actually storing your app's data.

How do you deal with the face that the browser may just decide to wipe the storage at any time?

The most important thing to do is get your users data somewhere safe as quickly as possible. For the vast majority of users that means your cloud database.

As the user generates new data, spray it to your servers as well as writing it to the syncable IndexedDB local storage, and to an in-memory buffer. Make your backend handle writes idempotently, and retry all failures a few times. (Eg, IndexedDB disk might be full or flaking out, so retry writing the memory buffer to disk.)

As long as the write path is quick, users can tolerate the browser nuking offline storage cache because they can re-download all the data that made it up to your server.

Hopefully soon the browser vendors will allow more durable file system access with appropriate user controls. Chromium built out the file system access API (https://web.dev/file-system-access/) but it’s not supported in Firefox or Safari.

Is this (generally) reliable?

I know terrible, awful bugs eventually doomed WebSQL from getting any traction and IndexedDB seems to be a more competent replacement, but the fact that Google is leaning on FSA seems like a non-starter.

It just feels like there's no way in hell Webkit will ever implement this stuff - not because of the divide between the App Store and PWA's - but due to the implications for privacy.

Hard pass.

I’m not sure exactly what your reliability question is about. I haven’t actually used the file system API I posted, probably same as you I’m waiting for it to ship outside of Chromium.

On the subject of WebKit, IndexedDB bugs are also pretty bad especially on iOS; we have debated about turning off IndexedDB write buffering in Safari and just do in-memory there. The best thing to do on Apple platforms is to make the app they’re trying to force you to make. Then you can make a little adapter so your web app can write to disk using SQLite and enjoy a nice relational API without needing to worry about the whims of the browser.

From the people who have used it, does it do what it says on the tin?

If it's the only real option and it's half baked (performance-wise), fine, but I'd be really concerned if the replacement for buggy code didn't actually do what was promised.

On the plus side, SQLite seems to be pretty stable on iOS, so at least there's a chance of it working out.

For a SaaS style (aka client-server) application, the right way to think of client-side storage is as a persistent cache, for a few reasons:

* it can be deleted at anytime (by browser, or even by user!)

* you generally want the server to be authoritative. if there's a bug client-side, server view of state should win.

* it's not possible in the general case to store all user data offline, it's always a subset.

Once you realize that the client-side state is a cache, potential uses of it become a lot more clear.

That's the thing. I want to make apps where the client-state is more than a cache. I want it to be able to be authoritative.

Sure, you probably want to put some sort of syncing on top, but that isn't even always necessary.

Alright, there is some naming collision then.

"offline-first" (terrible name, but here we are) generally refers to a classic web application that wants to be able to run offline either for network resiliency reasons or for performance.

"local-first" is a term that has been coined for something close to what you are talking about: https://www.inkandswitch.com/local-first.html

You can do that with CouchDB installed on the client's PC. Syncing that with a Cloud based CouchDB is trivial, and fast and efficient, and requires no user interaction.

What's wrong with offering the user a file to "download" (actually creating a file on the fly) or "upload" (actually loads it into the local web app?)

Edit: Here's the stack overflow copypasta I used to achieve it: https://github.com/EamonnMR/Flythrough.Space/blob/master/src...

Not much more to say other than Noms was my favorite project (https://github.com/attic-labs/noms) for a while until acquisition and the engineers are now the ones behind Replicache (https://replicache.dev/).

I think this is going to be the next "Realm" that works everywhere.

aww, thanks.

Looks like replicache is pretty expensive though.

If you have more than 500 users, the price is $500/mo (and it goes up from there).

sqlite bindings in browsers came and went in the time between when I learned about them and finally found a nail that needed that hammer.

I started on a design and literally within a few weeks Firefox announced it was deprecated.

Can you allow the user to load and save files?

I mean sure, you can get the user to download/upload files, but this is very awkward and not suited for storing every change. (But is good for migrating and backing up the data). Having to get the user to manually break the data out of the browser is not a good UX for day-to-day work.

I know that Chrome is pushing for a filesystem API but I don't know if that will be exempt from the usual ephemerality. IIRC it is just a private storage space with a filesystem-like API.

Chrome has an API that allows you to save to files without a new download every time. They made a pretty nice library that wraps that API on Chrome and it gracefully degrades on other browsers.

https://web.dev/browser-fs-access/

i noticed this behavior in a drawing app called excalidraw. first save opens a file dialog, subsequent saves just update the file, basically like a standard local text editor.

i keep doing 'save as' to create new files because i don't trust it lol

Reply on news.ycombinator.com