Safari now supports File System Access API with private origin | Hacker News Reader

Safari now supports File System Access API with private origin

webkit.org

230 pointsjvican79 comments

This is brilliant! Being called a “file system access” api will confuse many people into thinking it’s about traditional file storage for “people” to use, like a file picker/save dialog. It’s not, this is about providing a block storage that can be used for other things.

The one I am most excited by is for persistent SQLite with proper acid transaction in the browser, not having to load the whole db into memory. Absurd SQL [0] currently does this by creating a VFS on top of IndexedDB. This would let it do it properly, and is likely to be upstreamed to SQL.JS which is the main SQLite WASM project.

0: https://github.com/jlongster/absurd-sql

Many new in browser DB engines are going to get built on top of this. Others that I could see happing are:

- Relm from MongoDB being ported to WASM and use this for storage.

- If I were Supabase I would be looking to create a “Mini Supabase” for mobile, and make it work in browser too.

- Couchbase Mobile as an alternative to PouchDB

I don’t see how this is fundamentally any different from IndexedDB: both are asynchronous transactional key-value databases. One is byte-oriented and the other object-oriented, but it looks to me like they’re essentially completely equivalent in what’s possible with them—except insofar as File System Access may allow you to avoid loading the entire thing into memory. (I say may because you can’t control the browser’s implementation; my feeling is that it’s not unlikely that the browser may keep the full thing in memory in FSA where I wouldn’t expect it to with IndexedDB, which if so would return them to basically the same position.)

Implementing SQLite on top of this would require that a commit write the changes, close the file, and then reopen it, since writes are only performed when you close the file. That could perform tolerably or terribly (there’s no way of knowing), but certainly won’t be as good as what you get natively, where you can truly write only part of a file, especially once you get to large databases where it will certainly be drastically slower. If you want performance out of any of these sorts of things, you’re going to need to stop putting everything in one “file” and split it into many, so that you can avoid touching most of them on most writes.

I haven’t had a chance to dig into the details of what part/version of the spec Safari have implemented so far, and I’m aware the spec is somewhat a moving target. But I do know the developers working on the type of projects I’m talking about are talking to the working group behind the spec to ensure the spec meets these type of needs.

The point of this over “abusing” IndexedDB, jlongster who created Absurd SQL had to perform some pretty unholy hacks to get it to work. When porting these db engines to WASM they expect a FS to look and behave like a FS, that’s what this does that IndexedDB doesn’t.

Quite true you could build a new SQL/DB engin on top of IndexedDB with a storage architecture designed for it. But that’s not what the existing engines are expecting.

This FS doesn’t much look like an FS either—certainly it’s way off what SQLite needs. For mutating a file, you get the options write, seek, and truncate, but they don’t do anything until you close the file. It’s essentially equivalent to the write-to-a-temporary-file-and-then-atomic-rename-overwrite pattern.

You can implement exactly the same thing on top of an ArrayBuffer that you’ll put in an IndexedDB with no fuss—and if you were doing it that way, then you could even decide whether to go for something like a rope for intermediate editing, or mutating the ArrayBuffer directly, whereas if you use FSA you have no control over which of the two approaches it might use, or something else altogether.

It seems this is up to the implementation as the spec leaves it open. Safari is not mapping this to the actual file system, but what about other browsers?

For Origin Private File Systems, I would not expect any user agent to map things to the actual file system: there are notable functional problems especially on Windows, and it’s a very significant security hazard, all of which is neatly avoided by putting it all in the likes of a SQLite database.

First, the functional problems: file names are going to be a bit of a bother, to the point where if you back it by the actual file system you will require some kind of escaping mechanism, which loses you the exact one-to-one correspondence. Names are sequences of 16-bit code units, which means they need not be valid Unicode (ugh, I wish new additions to the browser platform would just start rejecting malformed Unicode), which will cause trouble on some platforms and will probably not be looked on favourably on others as having the potential to cause trouble in software not written to cope with that (e.g. on Windows I don’t think I’ve ever encountered a name with an unpaired surrogate, though it’s legal; on Linux, paths that aren’t UTF-8 are decidedly more common, though they still have a habit of breaking things). It also allows various file names that are forbidden in Windows at large, though in almost all cases you could with difficulty work around that on a per-process basis by throwing it into POSIX mode and/or using fully qualified (\\.\ or \\?\) paths.

Second, the security hazard: putting files with arbitrary names and contents on the file system is dangerous for a few reasons. The simplest is that there’s various buggy software that automatically reads files that get added to the file system, and more than a few times such software has had bugs in parsers that have led to privileged arbitrary code execution. A key purpose of Origin Private File Systems is that you don’t need to prompt the user for permission, because it’s supposed to be safe. I have no idea if this sort of vector was ever used back in the days when IE persisted its internet cache directly to files on the disk, but I can easily imagine such a thing happening now, and it could be really bad, given the right bug.

Great comment, this expands the issues really well for me.

I didn't notice before that this was supposed to be a safe api usable without permission, that indeed changes things.

Antivirus software often reads all files coming to the system, run with admin / kernel driver privileges, and still has lots of parsing and other vulnerabilities (project zero blog wrote about some of these). They tend to try to unwrap all container formats. I wonder if they'll start diving into SQLite files used by browsers and that way manage to expose their privileged attack surface to malicious web content.

I'm not sure that SQL in the browser is such a good idea in the long run. Database migrations are hard enough when there's just one instance of the DB; managing and performing migrations in each individual user's browser is going to be incredibly painful. Performance is also likely to be an issue.

your phone already has some tens or hundreds of SQLite databases on it. it requires some care, but many already go through the effort in their apps, so i can imagine web developers going to the same effort if they need it.

though i can also imagine them going through that effort when they don’t, so…

I would take the hassle of carefully constructed database migrations over a no-sql database with records in multiple unknown states from different versions requiring handling routines scattered throughout my codebase.

SQL will always win, as will the web, the potential of SQL in the browser for PWAs is incredible. It also allows alignment of mobile and web apps for local storage.

It was also the correct decision to drop WebSQL, it would have limited the api and held it back. With WASM SQL we can all use whatever sql db we want with whatever extensions we want. Row/Column oriented? Our decision (SQL.js or DuckDB right now). Need a specific full text search extension, or a custom one? Just compile it in. Have an idea for a funky CRDT based replication system? Write an extension and add it.

> I would take the hassle of carefully constructed database migrations over a no-sql database with records in multiple unknown states from different versions requiring handling routines scattered throughout my codebase.

Couldn't you also carefully construct the data migrations for a no-sql store like IndexedDB? In fact they provide a mechanism for it vs. having to bolt one into your SQL system?

    managing and performing migrations in each 
    individual user's browser is going to be 
    incredibly painful

What's the alternative? Typical NoSQL slop where we basically just wind up doing the "painful" SQL stuff anyway in what ultimately winds up being an even more painful process?

What's the alternative if you want to efficiently store data in the browser? SQL isn't the problem, every database requires migrations.

I think this is an ok permissionless fallback if you dont want to bother the user, but tbis feels so radically useless compared to local filesystem access.

It would be be so much more powerful & useful a capability if your users could see the sqlite db, of whatever other store you have. It feels like such a Safari-ism that this great capability does nothing for the user, does not help the web export & share itself. It keeps the web & the system boxes fully isolated. Safari's murderous intent.

This is incorrect. The File System Access API 100% does contain a showOpenFilePicker API. As it says in the blog post, they’ve just begun this process by beginning with the private origin portion of the API, but they’ll be adding the other portions later.

I’m curious how much use something like this would ever get; the Origin Private File System is basically just a file system implemented atop a database: it doesn’t provide any new fundamental functionality, but can be shimmed perfectly atop IndexedDB (though performance characteristics will probably differ). When people hear “file system access”, they’re going to expect that this means you can access arbitrary files and folders on the file system, whether unbounded or within a certain scope, but the Origin Private File System is not that. There need be no correlation between the file system this API exposes and your file system. (In theory there perhaps could be, but it’s not the preferred way and there’s no obvious reason to do it that way, and the question of valid file names would be rather messy, even if the spec kinda makes a weird half-compromise for backslash in file names; and I think the stuff about atomic operations could make it difficult to do sanely, too.) It doesn’t put things in a private folder so that you could open its contents in other locally-installed apps.

So this is basically the harmless but also fairly useless part of File System Access. The only real reasons I can think of straight away for it potentially being useful are (a) if it performs better than IndexedDB, and (b) for simpler API compatibility with something using the dangerous parts of File System Access, which Chromium has implemented but which Firefox and Safari are both utterly rejecting.

I’m also a bit concerned about browsers shipping this stuff given how much flux there still is in the draft spec, which is still thoroughly in its incubation phase and hasn’t been adopted by a working group.

In the end, I can’t understand why they’d implement this part of the spec. It will get some people’s hopes up, only to dash them, since they’re not implementing the useful but dangerous part which is what people actually want.

"since they’re not implementing the useful but dangerous part which is what people actually want."

I want it like this. There are lots of use cases for a file system to store user data locally in a safe way.

Even though it all can be also done with indexedDB, too - it makes this task incredibly more complicated. I did basically this, implement a file system on top of indexedDB - but with great pain.

and I did, because there was no other option - but only with great unnecessary pain.

I haven’t tried, but I think a minimal polyfill of exactly this atop IndexedDB might take 300–400 hundred lines of code (and I’d be interested in the difference in performance characteristics, especially on large “files”, and with different types of edit strategies). A mildly simplified but close enough to feature-complete form of it could probably be done in under 200 lines, though I doubt 100 is attainable honestly (IndexedDB is comparatively verbose).

It’s not that this is nothing, but I think it is and will be being fairly heavily oversold by some people.

Well, by now you can probably find some ready avaiable solutions, that would save you time.

But having it native, would get you also native debugging tools, where you just could see the data in a common file explorer.

What I have instead now, are tons of abstract dbs and object stores, when all I want to see are folders, pictures and text. Which is why I made a node server, which after sync, gets me actual files I can easily debug and store and edit.

But this does not work locally anymore then and the structure is unnecessarily complicated.

If a browser syncs a private file system access across devices, I could see this being quite useful; e.g. you could have a Google Docs -like browser based document application that doesn't depend on a Google hosting and snooping on it. I don't know about whether it adds anything above indexedDB.

Origin Private File Systems should be in exactly the same boat as IndexedDB, Local Storage and cookies here.

Are you saying that a Firefox user with an account which is synced across multiple devices does not sync the IndexedDB, Local Storage etc. That's news to me (though I have not looked into it particularly).

I don’t know what any browsers’ synchronisation systems do with any or all of these, but for syncability and sessiony purposes, these are all equivalent.

That's correct: I'm not aware of any browser that syncs those.

If one started doing that it would at least initially trip a lot of bot detection systems when a cookie for one device suddenly showed up on a different device.

Could be challenging for application logic as well, if their state depends on these resources.

Firefox Sync doesn’t copy of those.

https://support.mozilla.org/en-US/kb/how-do-i-choose-what-in...

Apple doesn’t even back up its local storage to iCloud, so if you have to restore an iOS device, anything stored there is lost (e.g. wordl progress)…

It's much easier to lose localStorage than that: if you browse for 7 days without visiting a specific site, Safari will discard that site's local storage.

(That also applies to this new API)

Unfortunately, this does not seem to be the "local file access" that Chromium offers and that Safari lacks.

With better browser support for the File System Access API, web applications that store their data in files might become a common thing.

Currently, when building a web application, you usually build some backend system that lets the user log in and then stores the data. Or you use IndexedDB and let the browser handle it. In both cases, the user does not have good access to the data.

If instead on the first run, the application asked the user "Where do you want to store the data" and the user selects a file or a directory, that puts the user in full control.

Then they later can backup the data however they like, edit it with other tools, version it etc etc.

The browser is an awesome platform. I love to write local tools in HTML. It is just so easy to tweak browser based applications to your needs. Open the html file, change a line or two, save - boom! you got what you want.

> that puts the user in full control.

This is entirely contingent on the format used. Netflix or any other media streaming company would store DRM compatible formatted content locally.

I think the feature in of itself opens up interesting possibilities but not sure how it will get used.

> If instead on the first run, the application asked the user "Where do you want to store the data" and the user selects a file or a directory, that puts the user in full control.

It doesn't, not realy.

Both Safari and Firefox are now very wary and weary of asking the user for anything. There are so many things browsers already ask the user for: camera, location, notifications, microphone... In case of Chrome, additionally: USB, Serial, HID, ...

All the discussions about such features inevitably boil down: it's impossible to properly explain to the user what the hell is going on and what the implications are. At one point the user will just click "ok" without reading.

Yes, so many potential dialogs. But I sure as heck want to be able to store my work product by some means which gives me the most control.

The problem is that browsers can't only target your work product :)

> It is very common for an application to interact with local files. For example, a general workflow is opening a file, making some changes, and saving the file. For web apps, this might be hard to implement. It is possible to simulate the file operations using IndexedDB API, an HTML input element with the file type, an HTML anchor element with the download attribute, etc, but that would require a good understanding of these standards and careful design for a good user experience. Also, the performance may not be satisfactory for frequent operations and large files.

> Based on the implementation of different browsers, one entry in the origin private file system does not necessarily map to an entry in the user’s local filesystem — it can be an object stored in some database. That means a file or directory created via the File System Access API may not be easily retrieved from outside of the browser.

Honestly, this makes no sense to me. The motivating example is exactly about being unable to interact with the host file system, but then they present a solution that does something completely different.

If this is supposed to be yet another storage API, then so be it, but this won't be able to solve the UX problems that the first quote was talking about.

Edit: Aha, the spec clears this up a bit: The standard defines different implementations of the file system API. Two of them are the "local" [1] and the "origin private" [2] file systems. The former is indeed a view of the host file system, with picker and all, while the latter is completely independent of the host.

Looks like this feature announcement was exclusively about the latter.

(The motivating example still make no sense to me as that is a clear use-case for the "local" filesystem implementation, but that seems to be more an issue with the announcement, not with the feature itself)

Edit2: Another important distinction is that the local file system requires user interaction to use (with good reason) : You have to show a file picker before you can use it, and subsequently can only access the files and directories the user selected in the picker. So you cannot use this as a behind-the-scenes storage mechanism.

In contrast, the origin-private file system requires no permissions and no user interaction.

[1] https://wicg.github.io/file-system-access/#local-filesystem

[2] https://wicg.github.io/file-system-access/#sandboxed-filesys...

Thanks for clarifying this for me. I'd gone through some of the specs of the File System API, but not nearly thoroughly enough to pick up on this distinction.

Why does the implementation detail around origin file systems affect the motivating use case from the announcement?

Web apps can now create their own file systems and treat objects as files (and it seems directly point to such files) with this api. That wasn’t easily possible before.

The type of files accessed is radically different: The local FS acesses already existing files that users typically manage by hand, whereas the origin-private FS accesses files that the webapp created itself and will typically manage internally.

In Unix terms, the former gives you access to parts of /home, the latter to parts of /var or /tmp.

I'm also not sure if calling them different "implementations" (as I did in the GP) is really the best way to put it. Both filesystem objects implement the same interface, but you call different APIs to obtain the objects in the first place and would use them in different contexts.

My issue was that the motivating example was about rebuilding the "open"/"save as" functionality from desktop apps, which the local FS API was tailor-made for - but the announcement is about the origin-private FS API, which has a completely different purpose.

I've read this a few times, but I don't see how you can regard this as anything other than false advertising. In short, it's basically a file-like wrapper around the hard-to-use IndexedDB. So it pretty much competes with localForage and idb-keyval.

The promise of File System Access API is in interop. There is no interop with Safari's implementation.

I understand the desire to have file access as a developer, but as a user, Safari and Firefox's decision to not implement the rest of it is a great decision. It's not worth the attack surface.

Firefox is not implementing. Safari is misleading people about what they have actually implemented. Big difference.

It is possible to simulate the file operations using IndexedDB API, an HTML input element with the file type, an HTML anchor element with the download attribute, etc, but that would require a good understanding of these standards and careful design for a good user experience. Also, the performance may not be satisfactory for frequent operations and large files. … Based on the implementation of different browsers, one entry in the origin private file system does not necessarily map to an entry in the user’s local filesystem — it can be an object stored in some database. That means a file or directory created via the File System Access API may not be easily retrieved from outside of the browser.

So what is the point of using this instead of IndexedDB, then? In IndexedDB we can even save private non-extractable keys!

Shameless plug for my soon to be outdated Safari iOS extension https://filepicker.app which implements these APIs through an extension.

Fortunately for you this shouldn't be outdated for a little while longer, since this only covers "origin private" filesystems. Which, in other words, are just a file-structured IndexedDB and not actual local file access.

This is fantastic! How haven't I heard of this before?

Does anyone know if there's a list of small, little-known extensions just like this?

This is awesome. What other web apps are out there that use this API that I've been missing out on?

In the document I don’t see how they plan to avoid this being used as storage for persistent evercookies.

In the same way, indexedDB has to be manually cleared by the user across all browsers?

(and safari does this automatically after some idle time, I think)

At least this would allow a transparent download your files option which is a work around for safari on both mobile and desktop, since a user can prompt to download files in Safari (particularly useful on mobile).

However it still relies on the user knowing about this or the application having to somehow prompt it from the user, rather than transparently writing to the file system directly or mounting it as a drive.

Step in the right direction for sure, still way too much hassle though. This singles to me they just made sure not to provide a first class file handing option for web developers that would potentially compete with native apps, especially on iOS

This sounds like a possible solution to localStorage being cleared after 7 days of no user interaction on a site in Safari, though I'd guess this has some persistence limitations as well.

Damn, "its storage lifetime is the same as other persistent storage types like IndexedDB and LocalStorage". So a maximum of 7 days if the users doesn't return to the site. Not very useful.

Gross. Imagine going on vacation for a week and not using a particular website during that time, and losing all your data.

Safari should have a big warning on top of that blog post saying, “all files will be deleted after 7 days unless these API’s are being used within a PWA”.

Also, if this API behaves like Local Storage, it’s not backed up to iCloud either.

Safari's rules on storage persistence are aimed at limiting tracking. For it to be effective, they need to apply that limit to all forms of client side storage.

I understand that perspective but in reality it does way more damage to those of us who want to provide services that are specifically trying to protect privacy by not having user accounts or storing user data server-side - and our users. There is no way for me to safely store user data, for a site that most users interact with once or twice a month, without having to store that data server-side and requiring some way to authenticate them to retrieve it when the come back.

If Safari provided an API to store data longer after a user prompt I would be more understanding.

It seems many apps these days contact more trackers than a typical website[0]. And the apps manage to circumvent most of the counter measures despite Apple PR on privacy.

[0] https://blog.lockdownprivacy.com/2021/09/22/study-effectiven...

A prompt would make a lot of sense here. Perhaps as an extension of https://developer.mozilla.org/en-US/docs/Web/API/Storage_Acc...

Or do it a bit like the engagement factor was for autoplay. A site gets an extended interval before clearing because it receives user interaction and the user spends X minutes on it. I'd actually be fine with sites that receive no user interaction or the user spends less than 1 minute on having immediate evacuation of storage when the user leaves.

I don't think gating on interaction does much of what they want, because they are trying to prevent cross-site tracking and it is quite common for people to interact with both sites.

This is a mitigation that specifically addresses tracking that runs entirely on the client. If you set a cookie from the server the time limit does not apply, which means you can store whatever information you want on the server keyed by an HTTP cookie.

i felt previous coverage of this topic was much more on target. 11 points, 4 days ago, 6 comments: https://news.ycombinator.com/item?id=30335261

Anyone know if it mentions a file size limit?

Will there be limits to the File System Access API in terms of file size and file persistence?

I wonder how symlinks work in this context.

Not applicable; Private Origin File Systems are just a key-value store made to look a bit like a file system, and symlinks don’t exist.

(There is also the rest of the spec which provides real file system access, and that will require decisions on what to do with symlinks, but only Google has implemented that—Mozilla and Apple have strongly rejected it as a currently-unfixable security hazard.)

What could possibly go wrong?

Do they limit using space based quotas or other systems?

Better late than never?

Why are you saying this is late? The specification is a draft, the only other rendering engine to support it is Blink, and Firefox isn’t going to implement the full spec.

Because over a decade ago the Apple devs poo-poo’d all over the premise of this API when it was first proposed. I’m referring to its earlier incantation that we added to WebKit and shipped in Chrome but could never get Apple devs to support. I’m delighted to see it supported now. I just think of all the lost time for the web and what could have been.

Who is the “we” you are referring to?

> I just think of all the lost time for the web and what could have been.

This doesn’t seem like it enables anything new though? It’s a convenience.

> Who is the “we” you are referring to?

Darin used to be something like my boss's boss's boss's boss when I worked in Chrome.

> Because over a decade ago the Apple devs poo-poo’d all over the premise of this API when it was first proposed.

What I read is: Apple devs came back with a list of problems and issues as long as Jupiter's equator, Chrome said "we don't care" and shipped it?

> I just think of all the lost time for the web and what could have been.

With Chrome not rushing it's own broken insecure implementations of internal APIs, not pretending they are standards and not gaslighting other browser vendors? Oh, the Web would be a much, much better place.

Cause Chrome implemented that eons ago. Of course Firefox isn't going to implement anything, Firefox has been a thorn on the side of web standards for a good decade now, thanks to Mozilla's management.

Chrome implemented and shipped an early draft and has, I believe, broken it multiple times since. What it shipped has significant security implications that are handled poorly at best (mostly not at all).

Firefox and Safari have said “you what!? No way we’re implementing that as it is”. And Safari are now implementing the safe and boring part of the spec, not the part that people actually want.

> Cause Chrome implemented that eons ago. Of course Firefox isn't going to implement anything

If Chrome matters, Firefox doesn’t, and Safari is measured against when Chrome supports something rather than when the specification becomes stable, it sounds like you think the web is defined by “whatever Chrome supports, whenever Chrome supports it”.

Good web standards are whatever simplify my work as a developer. Firefox used to be at the forefront of web standards.

I can guarantee you Firefox will eventually be running Blink because Mozilla has no vision for the future of the web anymore. Mark my words.

Do you remember the last time we had a near monoculture for browsers? People with attitudes like yours were all too happy to think of Internet Explorer as the only browser that mattered. Microsoft decided they had won and mothballed Internet Explorer development. We then had a dark ages for web developers with five years of no progress in browser functionality.

Positioning Chrome as the definition of the web is incredibly harmful for the long-term health of the web. A web independent of a single vendor is vital.

You're blaming me for Mozilla's failures? How about you stop being complaisant of Firefox mediocrity?

> You're blaming me for Mozilla's failures?

Where did I say that?

> Microsoft decided they had won and mothballed Internet Explorer development

the difference is that Microsoft wanted to protect the Desktop platform. Google isn't in that position with the Andorid platform, but requires improvement on the Web.

That said: I fear the power Google has in that realm.

Good web standards take implementer consensus and users into account too, including their security.

Google has ploughed ahead with File System Access in order to meet a developer desire (and occasionally user desire). Mozilla and Apple have declined to implement it (or at least the dangerous half) because no one can find an acceptable solution to the security problems.

File System Access is not currently a good web standard—in fact, it has not even progressed to being a web standard, because of these problems.

Reply on news.ycombinator.com