It does not stream in content, but I would guess the API might support that some day.
2,856 karma · joined February 3, 2009
Third party kernel extensions on macOS are on the way out. On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location. Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
4. All sorts of other stuff, happy to keep going
Third party kernel extensions on macOS are on the way out. On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
But that’s okay, they’ve got something better they’re working on. Just keeping it a little quiet right now since it’s not 100% done.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.
Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
4. All sorts of other stuff, happy to keep going
I run ExpanDrive (www.expandrive.com). We use a derivative of macFUSE (https://osxfuse.github.io/) on macOS.
Third party kernel extensions on macOS are on the way out.
On M1 machines any third party kernel extension requires a reboot into recovery mode, lowering the security settings, and then finishing the install.
This is a terrible and unfriendly model for users (by design). This is the primary reason these Google/Box/etc aren’t shipping M1 versions of their software.
But that’s okay, they’ve got something better they’re working on. Just keeping it a little quiet right now since it’s not 100% done.
The replacement for apps like Google & Box Drive (and everyone else) is their new File Provider framework (https://developer.apple.com/documentation/fileprovider), which they quietly launched but didn’t publicize at all. It’s basically a technology that enables something like Dropbox Smart Sync (project infinite) built directly into macOS fully managed by APFS.
It’s a pretty big departure from the macFUSE style interfaces we've been using for years now. File Provider apps are userspace extensions that work with with placeholder files and on-demand hydration of remote files by the system via APFS. It’s a pretty cool framework, but pretty new (and a little raw).
Box/Google/Dropbox/Microsoft will all eventually be using it, but the rollout has been slow.
The whole macOS File Provider project is an off-shoot of the iCloud drive and iOS file provider projects, but with one key component they’ve added called the replicated extension that has an apple-blessed extension to APFS called “FPFS” (presumably file provider FS) that can intercept read/writes/etc to these on-disk placeholder files and download content (and thumbnails!) on demand. Provides data directly to spotlight indexes, etc.
Whereas FUSE is a really low level API exported into a safer user-space process, this is a little higher level. You give the file provider framework lists of directories, help it monitor changes, provide item info and then it will in turn issue callbacks into your extension to initiate downloads to a temporary location.
Benefits
1. All on demand, driven by placeholders. “regular” sync downloads contents ahead of time, this is not the case with file providers. It’s more like network filesystems in this regard
2. Doesn’t eat up free space. Providers can mark their content as “evictable” which lets APFS know it can toss out the data if space gets low. But what I think is extra interesting is that when you mark the content as evictable it doesn’t even register as being used against free space. You could bring down a 10GB file from the cloud and your free space remains the same
3. Integrated with all the higher level APIs so that applications that open a file with swift/objective C don’t beachball while waiting for a download (for an open) to complete. They appropriate waits and expectations are in there now
4. All sorts of other stuff, happy to keep going
That being said, I wish B2 performance was better. Throughput is dramatically slower than S3.
I have kext signing certificate for ExpanDrive, Google has one for Google Filestream, I suspect many others have one as well. Rightfully, Apple doesn't hand them out as easily as they do with regular developer certificates, but if you want one and do a reasonable job representing that you're not going to panic end-user systems, you can get one too.
FUSE for macOS remains open source, fork it if you want. Benjamin merely decided not to work on it for free anymore and essentially providing bug fixes etc for those who pay for it.
Lastly - FUSE of macOS is not going to be around in the current form much longer. Apple has made it abundantly clear that Kernel Extensions are on the way out, and that macOS 10.15 will be the last release to fully support kexts without compromises. Check this slide from WWDC
What are best practices for using/selecting an ESP if you were to use a project like this and want to ensure reasonable deliverability?
Renaming a folder than has a million files/folders inside is a single operation in SFTP, but 2 million operations on S3.
Does it handle writing at arbitrary offsets within a file? Does it download the file first then let you start writing?
What about just writing a few bytes at the beginning of a large existing file and then closing your SFTP handle?
How about 2 users accessing same file via SFTP at the same time?