Hacker News Reader

Comment by jez | Hacker News Reader

A lot of people are taking this as yet another example of Dropbox changing things for no reason, but I think it’s important to make clear that in this case Apple is forcing their hand. Dropbox is being told by Apple that apps must integrate with File Provider API[1] if they want to present files that don't exist on disk until they’re first opened.

This API is hamstrung and limiting, full of instabilities and quirks (designed to fit iCloud primarily but not necessarily everyone else).

In my opinion, the fault lies with Apple here.

[1] https://developer.apple.com/documentation/fileprovider

(Full disclosure: I interned at Dropbox for 3 months 8 years ago.)

View on HN

Apple can’t (yet) stop you syncing a folder somewhere on the disk. Other services offer this feature - sync.com for example, which I switched to after Dropbox announced this change.

Dropbox wants to offer some features that are only available through Apple’s API. They want to be on the App Store. But they need to weigh that against what still is their primary feature, and literally how the product was described when sold. “A folder synchronized across all your devices.”

I am not a product manager, but I simply would not throw out my main feature and the thing that differentiates me in the market, in order to provide some features Apple wants, where the only difference between me and OneDrive is the icon.

I agree, but I think it’s important to point out: I believe the main feature they need is to sync the full file tree without having to download the actual files until needed. I don’t know how others do it, but I assume that you’d have to manually maintain mappings between a subtree in the cloud to one on disk, or alternatively run out of disk space if you want to have everything synced.

Non-techies will find it incredibly annoying to map individual subtrees leading to missing files, upfront sync costs when you need those, and running out of disk space. Dropbox cannot afford to simply throw their hands up and blame the users for not being technical enough.

That said, this hack (pretending your files are on disk) is arguably a fragile one, and perhaps shouldn’t exist in the first place (assuming it’s not a networked file system, which I believe it’s not). Or at the very least, they could have attempted to publish a standard. But if dropbox is just another proprietary piece of software running hacks, I can’t really blame Apple.

What I find surprising is that Dropbox is apparently willing to just fit in the narrow space Apple has carved out for them.

Especially now, when companies are cutting their spend, this is an extraordinary misstep. If I am already paying for iCloud, Google Apps, or Office 365, I can get exactly what Dropbox offers for free, and better integrated with the other productivity apps.

So why exactly would anyone use Dropbox now? There is nothing to differentiate them.

It’s hard to see this as anything other than the beginning of the end for Dropbox.

So why exactly would anyone use Dropbox now? There is nothing to differentiate them.

Their sync is just must faster and they actually do block-level sync, which the other major providers still haven't implemented yet after over ten years. I understand why people go to the competition, but I think it's a mistake. The FAANG offerings just provide enough (especially disk space) to lure people into their ecosystems, but they don't really care much about file sync, and therefore they are all quite miserable.

It is a bit like Microsoft Teams. Companies choose Teams because they have an Office365 subscription for Microsoft Office anyway and it is cheaper than Slack. On the surface, Slack does not seem to have much of a competitive advantage anymore. So, if you are simply bean counting, it's clear that you should switch to Teams. But in the meanwhile, everyone using Teams feels miserable to be trapped in it.

OneDrive does block level sync

Ah, cool, I missed that they added it in 2020. Google Drive still doesn't seem to support it.

Google Drive also used to (maybe still does) change names of your files if they contained parenthesis.

I think most people are using block-level sync. Hell, even Syncthing has block-level synchronization now.

> Hell, even Syncthing has block-level synchronization now.

I thought it (BEP) was always a core feature?

Because the rest of the company is on Dropbox already, and it's easier to stay there than migrate everybody. Single personal-use accounts aren't their bread and butter.

perhaps shouldn’t exist in the first place

Why not? It's a very handy feature for a lot of users who don't have the disk space available to store their complete Dropbox folder. And it's much easier to understand and work with than manually managing selective sync.

Or at the very least, they could have attempted to publish a standard. But if dropbox is just another proprietary piece of software running hacks, I can’t really blame Apple.

Right. The issue is that you need to intercept file I/O to make transparent sync work. Dropbox did this before using a kernel extension, so that it can hook into the kernel kauth framework [1]. However, having third-party code running in kernel space is a huge liability, besides opening a huge possibility for state-sanctioned backdoors, it opens another venue for security vulnerabilities that give kernel-level access. So Apple decided that the kauth API should go (probably for other reasons), but more importantly that thirt-party kexts should go.

In general, I am very happy that Apple is on a mission to ban kexts, I don't want closed source third-party code to run in the kernel. However, it seems that Apple provides replacement userspace APIs, but is not really receptive to issues that developers have with them.

[1] https://dropbox.tech/infrastructure/going-deeper-with-projec...

> In general, I am very happy that Apple is on a mission to ban kexts, I don't want closed source third-party code to run in the kernel

Will you still be able to run open-source or first party (for example, written by yourself) kernel extensions? Or will future macos versions require jailbreaks to get a real root permissions? Genuine question, I'm not too familiar with Apple products and I'm trying to understand why is this a good change. It looks like a limitation imposed on users.

FWIW, running code in the kernel is not "real root permissions", it's kernel-mode privledges, which are much different and arguably much more dangerous.

For my money, running custom kernel-mode code is not a concern. Apple is infatuated with remote device attestation for some reason though, so preventing OS modification is paramount for them right now. It's a limitation, but one that's hard to get mad about when Macbooks have a mostly-open bootloader.

To answer this question explicitly, it's currently possible to jump through some hoops to run kernel extensions, but Apple has been slowly tightening the requirements over the past few years. here's what Apple currently has to say about them.

> Kexts are no longer recommended for macOS. Kexts risk the integrity and reliability of the operating system. Users should prefer solutions that don’t require extending the kernel and use system extensions instead.

"System extensions", in this context, refers to extensions that the OS loads into userspace. Apple is developing userspace extension APIs to perform actions that were traditionally handled by popular kexts. The File Provider API is one of these system extensions. Dropbox used to rely on a kernel extension to intercept file access. File Providers allow them to do the same thing, but, as evidenced by this article, there are limitations as to where the files can live, and there are some bugs.

So if the File Provider API is so bad, and it's sill possible to load kexts, why doesn't Dropbox continue to use a kext?

- In 2015, Apple updated the OS to require that kexts be signed by a certificate in the Apple Developer program. This allows Apple to decide who can sign kernel extensions, and allows them to revoke the signing certificate for individual extensions. - In 2017, Apple updated the OS to require a multi-step user confirmation before installing a kext. This initially caused some headaches for system administrators automating the deployment of multiple machines, but eventually solutions to this were implemented. - In 2020, Apple started shipping ARM-based Macs, and by default, they don't allow the loading of any third-party extensions. Instead, users have to go through a process to enable them, and this process can be disabled on managed machines. Here's how Apple describes that process.

> Kext management by the user requires a restart to recoveryOS to downgrade security settings. The user must press and hold the power button to restart into recoveryOS and authenticate as an administrator. Only when recoveryOS is entered using the power button press will the Secure Enclave accept the change of policy. The user must then select the checkbox Reduced Security and the option “Allow user management of kernel extensions from identified developers” and restart the Mac.

All of these requirements can also be bypassed (if the machine is unmanaged or the sysadmin allows it) by turning off System Integrity Protection, which is a similar process as described above. However, disabling SIP disables a lot of security protections in the OS, and turning it off shows a lot of scary dialogues to dissuade users from doing it.

And for most users, SIP is a good tradeoff. It protects against a lot of things that malware can do. It prevents code injection, so dtrace doesn't work with SIP on, but if you're not using dtrace, there's little reason to turn it off.

Either way, the process is complicated enough that not enough users will do it in order to provide a mass market for Dropbox. Apple is also pushing Dropbox to use the File Provider API with the implicit threat of revoking Dropbox's signing certificate for their kext. Apple will not issue signing certificates for products that can be implemented using the userspace system extensions instead, and it's likely that they will continue to restrict the loading of kexts further in the future. Even if users can disable the signing requirement by disabling SIP now, there's no guarantee Apple will allow that to work in the future.

> Why not? It's a very handy feature for a lot of users who don't have the disk space available to store their complete Dropbox folder.

Yeah I wasn’t fully clear.. I don’t mean the feature itself, just the way it was implemented.

> I believe the main feature they need is to sync the full file tree without having to download the actual files until needed. I don’t know how others do it, but I assume that you’d have to manually maintain mappings between a subtree in the cloud to one on disk, or alternatively run out of disk space if you want to have everything synced

I thought the whole point of file syncing software was to sync all files. If I do not have internet access, then would I not be guaranteed to use all the files in my Dropbox folder?

I only use iCloud Drive, does it do the same?

iCloud Drive does not guarantee all your files are available offline.

Interesting, I assumed anything I put in iCloud Drive was downloaded to my laptop.

I too switched to Sync.com. I've been quite happy with it so far.

I had frustrations with Dropbox about a lot of this when I realized they had made little public effort for Apple Silicon, but I think the fact that Dropbox is kind of stuck moving in this direction essentially highlights that Apple’s approach to cloud works for Apple … but not really anyone else.

I think for tools like like Box or even OneDrive, this move makes sense for, but Dropbox is intended to be a living backup of your primary folders, and that model just doesn’t fit with this.

This is almost bad enough to the point where the right play might be to have a machine on another platform, like Linux or Windows, manage your file sync, then to just access it on your local network. Apple really messed this up.

Well from a user and third party developer perpective it actually does make some sense for an OS to provide standardisation over cloud storage.

"If in folder X expect content to require fetching depending on application Y settings and act/manage exceptions accordingly. You can use that API if needed than standardize sync states across providers"

It is easier to exclude or target specifically for backup, easier to know you shouldn't store there for offline use, etc...

Being forced into OneDrive on Windows10 for corporate need since 10 month, I can tell you it's unbelievably easy to misconfigure OneDrive into a slugish unresponsive nightmare. Especially if you dare mix Organization OneDrive and Organization Sharepoint "mount points" in not exactly the good non obvious way. (never ever for hell's sake use the sync button in SharePoint website, use add to your OneDrive button)

> Well from a user and third party developer perpective it actually does make some sense for an OS to provide standardisation over cloud storage.

There’s standardization and there’s treating every cloud service like it should only work in one way.

Dropbox for years sold itself as being intended to work as your hard drive that could follow you as you switched devices. But now it’s working on a platform that actively discourages its use in that way, one that is operated by a company that is a direct competitor to Dropbox (albeit one that is not cross-platform). It feels like they’re getting crunched by standardization that is meant to disrupt their business model.

Now is that business model struggling for other reasons? Sure. But it shouldn’t be missed how this shift is arguably more disruptive to Dropbox than many of its competitors.

I’m not saying that standardization isn’t a good idea. But the API clearly was built without a lot of input from users of these services to understand why they want them to work in certain ways.

Not familiar enough was Dropbox was able to mount on "/" ???

As others and the suport page pointed you, you can still put a symlink at any available location you want.

It's still fully usable but it's now garanteed to not mess around with others contents. I personally am an iCloud Drive user at home because if I already chose to trust Apple for the OS and all, using their (contractually paid for) sync service doesn't require me to trust another party.

Standardization and sandboxing of cloud providers actually make me think about subscribing to a third party (for diversification let's say, or for separation of concerns when sharing on pro/other context). Because if standardized I can be assured by OS policy that theses sync services won't fuck around anywhere else than in their designated folder.

So yeah, maybe from now on you'd better go for rsync based tools that don't target App Store for pro use. But for 90% of the target audience this might actually be a desirable feature.

> Not familiar enough was Dropbox was able to mount on "/" ???

No, but what the OP meant was that Dropbox users think of it as a replacement for an external hard drive with all your projects. (Myself, I consider Dropbox, OneDrive, and iCloud Drive, to be the same thing with a different corporate overlord, and I migrated from Dropbox to OneDrive without any changes to my workflows…)

> It's still fully usable but it's now garanteed to not mess around with others contents.

How would it mess around before? Do you have any examples?

I can't point out a specific bug/vulnerability because I don't follow them that closely on platforms I don't use.

It's just that globally I'm not fond of background processes that run without being properly sandboxed through the OS. I'm less regarding when using FOSS software but for commercial closed source solution like dropbox I'd rather have them run with limited privileges.

These are two different arguments. You’re arguing for a level of security; I’m arguing that Apple’s implementation did not consider real-world use and leads to end-user compromises that lessen the experience.

In particular, I point to the move that you have to now keep the storage on the main drive, and the move to block access to the Mac’s photos library. Neither of these moves were necessary for sandboxing reasons. Both degrade the experience for Dropbox users that may have a folder far larger than the 256GB many base Macs come with.

Apple could have accounted for use cases of this nature to allow them to make sense for existing Dropbox users. They didn’t.

Sandboxing can be good in theory but completely suck in execution. That, I argue, is what happened here.

Leaving a third party Interfer with Photos library can easily lead to corrupted database and sync error if iCloud Photo library is enabled or worse if someone tried to access "Apple managed Photo library" from 2 different devices at the same time (the latter being a typical use case of dropbox). So this make a lot of sense, if you want to share a photos with drop box you should configure "Apple Photo library" to keep files in place instead of including them in database and sync that folder. Or you are even better using an other photo management app that support ACID database consistency accros dropbox file sharing (which is actually not an easy problem). Actually to share photos with dropbox you are maybe even better off syncing flat jpg files and use applications that don't rely on database at all on your different devices. A friend of mine do the latter and he's happy with that.

I can agree however on your last point, it could have been nice to allow cloud storage to be moved to an external HD. Maybe they should bring back this possibility in a futur update.

But for the managed photo library nothing currently prevent you from storing your database in an external drive, it's only cloud storage that is forbidden for the previously mentioned concurrency issues.

Please keep in mind that Cloud storage is not an external hard drive storage, it's not local, it's not a backup either (I personally backup regularly files on my cloud storage in 2 separates offline external HD and everyone should at least get one. Otherwise you actually have no backup).

> it's unbelievably easy to misconfigure OneDrive into a slugish unresponsive nightmare. Especially if you dare mix Organization OneDrive and Organization Sharepoint "mount points" in not exactly the good non obvious way. (never ever for hell's sake use the sync button in SharePoint website, use add to your OneDrive button)

I am using both of these, direct sync of SharePoint libraries (what you call Organization Sharepoint) as well as linking the libraries to the personal library (what you call Organization OneDrive) and have never had any issues. What are your problems with it?

The only thing one has to know is that it's not possible to do both at the same time for a given directory: a synced folder (and its children) can't be linked to personal and a linked folder (and its children) can't be synced. Note that "linking" also "syncs" it, just in a different location.

Well my organization current moto is to create hundred of different sharepoint sites that can contain multiple document libraries to compartmentalize (I personally think it's a really bad idea but i have to comply).

So when you happen to work with 10+ Sharepoint sites there is two way to operate if you must sync.

1. Click sync on the document library. Then each sync while need a configuration entry in One Drive and each sync will create it's own background worker. Prioritization between sync/mounts point must happen somehow but if you open 10+ I experienced lot's of sync error/delay. Sync parameters to each sharepoint have to be defined manually and independently on all your devices.

2.You can add a "shortcut" pointing to the sharepoint into your organization (personal) OneDrive. The configuration is then stored on your OneDrive profile, it's synchronized across all you device and the OneDrive webapp. You can move the shortcut around (inside your OneDrive) and rename the folder as you which. It only require one background worker that don't compete for system ressources with others.

I was initially configured using solution 1 and experienced many conflict/lag/bug. Since I implemented aolution 2 everything is (almost) flawless.

So the question remain why do they offer two concuring implementations one of which can lead to madness... legacy support I guess.

> the right play might be to have a machine on another platform, like Linux or Windows, manage your file sync, then to just access it on your local network.

I mean, the Internet has known for a long time that you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem…

I know you’re being sarcastic and highlighting an infamous comment but I sort of feel like that’s what we’re being driven back to.

I want all my Dropbox files to be actually present on disk at all times, so this change has only downsides for me. I’ve been with Dropbox for over a decade at this point, mostly out of inertia, but forcibly changing the location of the folder will likely push me to finally switch away.

Do you use a custom location? Or default to ~/Dropbox?

You can always symlink to whatever location Dropbox forces upon you…

I don't see how Apple can force the matter. Why would Dropbox comply if it breaks things?

They can't force but they can't retract from the AppStore. You'd still be able to deliver outside the AppStore with a signed App on MacOS but with way less visibility.

Who's discovering DropBox from the App Store? Seems like they could drop that and be just fine. Much smaller devs have.

Who installs anything to a desktop computer from an app store instead of the developer's website?

I prefer installing things from App Store on my parents' computers cause it will handle automatic updates and provides certain guarantees (i.e. can rely on Apple not letting malware onto their App Store).

Dropbox currently uses API X to have files that are available online but still visible from the Finder and other apps and can be downloaded on-demand. Apple wants to retire API X and offers API Y (FileProvider) as a replacement. If Dropbox want to keep the download-on-demand feature, they must switch.

It’s because their implementation of “files downloaded on demand when you click on them” required a kext, and kexts can’t be loaded on Apple Silicon without weakening (disabling?) system integrity protection. Google Drive and OneDrive had to wait a while before Apple had these APIs ready before they could make non-Rosetta versions, and I’m sure Dropbox was the same way, we’re just hearing about it now.

What is a kext? What is system integrity protection? What is Rosetta?

I don't use Mac

- Kext == Kernel Extension - System Integrity Protection (or SIP) is a feature where various core operating system files around the boot process and kernel must be signed by Apple or the machine will not boot. It can be disabled, but this leaves your machine in a much more insecure state and can cause certain features to be disabled. I know for example I would get reamed by my companies InfoSec team I had SIP disabled on my corporate machine. - Rosetta 2 is an transparent and efficient x86 -> to ARM64 translation layer on MacOS. It’s used on Macs with apples m series of processors to run old software that hasn’t been recompiled for the new architecture. It runs quite efficiently actually with ~15% performance overhead in my experience. Rosetta 1 was and older version of the same thing from the last Mac CPU architecture change (PowerPC -> x86)

It doesn't allow the developer to sign their own kernel extensions? That's seems mighty asymmetric.

A developer can sign their own kernel extensions, even on M1. It just requires you to initially set the machine into a specific mode that allows it. The people this puts pressure on is not individuals trying to write or sign their own kernel extensions if they know what they are doing (essentially reboot into recovery and run some some shell commands).

It does prevent

- Unsuspecting or otherwise ignorant users from being tricked into installing a malicious kernel extension. - Prevent large companies from distributing custom kernel extensions for every day users

KExt: Kernel EXTension.

AFAIK loading custom kexts has become harder since the launch of TouchID on macs, but for an average user its better to live inside the secure world of Apple's tight security controls & encryption - allowing Apple to claim security & privacy as core selling points.

Loading kexts require disabling SIP (System Integrity Protection).

Rosetta 2 makes x86_64 binaries run on Apple Silicon (arm64).

You can dive deep into Apple's platform security guide + arch: https://help.apple.com/pdf/security/en_US/apple-platform-sec...

This really is a full disclosure, huh.

Reply on news.ycombinator.com