AeroFS is x87 faster than Dropbox in LAN syncing [video] | Hacker News Reader

AeroFS is x87 faster than Dropbox in LAN syncing [video]

blog.ram.rachum.com

33 pointscool-RR59 comments

People often misunderstand what the Dropbox LAN Sync does. The file needs to be already in the cloud before it can be synced over LAN. This means that when you add a new file, it will first get uploaded to Dropbox servers then transferred to other computers over LAN. Considering that most home connections are asynchronous (slow up, fast down) the entire processes gets bottle-necked by that slow upload rate.

You've given an excellent answer to the question Why is Dropbox's LAN Sync so slow? but the more interesting question is Why don't the Dropbox engineers change how LAN Sync works so it won't be so slow anymore?

Also, uploading the data to Dropbox server, and then downloading again, wastes valuable quota for those under a data cap.

No, it doesn't download it from the master server. It transfers directly between the computers on the LAN. The only thing that has to go through the Dropbox servers is the master file list.

I can understand why, particularly given Dropbox's aim of being as simple as it can possibly be. At the moment the rule is simple. At the moment the cloud can be relied on as the authorative copy which everyone syncs with. If I have 3 or more local clients, all talking and sharing over LAN, many issues get very complicated.

It seems like it could be mitigated by having the client hash the file and notify the cloud of the hash of the file it intends to upload first. But I think your explanation for why they don't currently do LAN-only sharing is correct.

It does do that; I don't understand what you think it'll fix when it comes to multiple clients.

Do you want Dropbox to make a ghost entry with the contents and the hash, then start transferring it over LAN right away? That's going to be confusing when a ghosted file doesn't behave the same as a normal unsynced file. Especially if you start doing collaborative work and the version history is broken because that's cloud-based.

This is also how SpiderOak's LAN sync works. Both Dropbox and SpiderOak guarantee that the single point of truth is the cloud, for good reason.

Making exceptions to that approach (like allowing individual machines to sync with eachother directly before transferring to the cloud) creates opportunities for conflicts and inconsistencies that are very difficult to resolve because all the information might not be available.

Your second paragraph reads like a defense of CVS. git-annex handles this fine for me on my network at home with multiple machines and multiple copies of the repository/archive/drop-box-folder on each machine.

With any VCS, commits are explicit. Dropbox and the like need implicit sync.

Are you familiar with git-annex and its helpful assistant?

While it looks cool, it seems to me that it isn't entirely automatic. Sadly the vast majority of people will not go for anything less automatic than Dropbox.

"any changes you make to its folder will automatically be committed to git, and synced to repositories on other computers." [1]

I guess its not automatic in the sense that it does not install itself, but then again neither does dropbox...

[1] http://git-annex.branchable.com/assistant/

Nitpicky, but you meant the typical home internet service is asymmetrical, not asynchronous.

Note that technically the file contents don't need to be in the cloud, but rather the metadata and indexing do. I'm guessing that the indexing is how Dropbox breaks the file into blocks, checksums them and does some form of duplicate detection to save on storage costs.

I love AeroFS. Although it is missing many of Dropbox's features (e.g. web storage of files, better version history, etc), the fact that it is extremely fast and there is no cap on P2P sync makes it my weapon of choice when sharing files/folders with colleagues.

Sounds like exactly what I need - and what I usually use Dropbox for.

If anyone could spare me an invitation...

I'll be happy to invite you, but I don't have your email address.

robert@chromablue.net

What's the best way of writing this. Is it x87 or 87x faster? Thank goodness it wasn't x86 faster as that would have confused me.

On a related note, I've been using Livedrive for a number of years now and they have a lan transfer option to obtain a file over the network than over the internet. I suspect however a lot of use cases are separated by the internet and generally aren't on the same network.

What would be awesome however is if after the sync both computers were able to contribute to uploading the file to the live server. So I could sync my files over lan to my laptop, go to work, then have my home/work computer upload them to the net, thus 2x quicker to store it in the cloud. But that would be a bit of an edge use case I think.

I've always thought it's 87x as in 'eighty seven times' - the other way around makes it seem like a long lost Intel instruction set.

x87 is the original FPU instruction set of the x86 family, so time to be confused again. But now you can be .75 confused instead of 1 confused.

> But now you can be .75 confused instead of 1 confused.

A confusion index of 1.0 would be described as perfectly confused. But, as is often said, nothing is perfect. :)

Just tried this myself with a couple of 8.5MB MP3 files. One appeared on the second machine in about a minute, the other in about a minute 30.

Not sure why this is taking upwards of 12 minutes for the OP.

I'm jealous that on your computer, instead of being incredibly slow, it's only slow.

OP probably didn't disable dropbox speed limiting - this is the default:

http://i.imgur.com/wTWzG.png

For what it's worth, I haven't changed those settings on either of my machines.

Thanks for that tip. I disabled the limiting now and it sped up from 11:42 minutes to 01:40 minutes! Now Dropbox is only x12.5 slower than AeroFS.

The real solution, of course, would be for Dropbox to transfer the file over the LAN immediately before waiting for it to be uploaded to the Dropbox servers.

That's not a solution, it's a design tradeoff. Dropbox's servers are the "truth", and the software's clear goal is to get a copy on the servers. There are many reasons for this--weird conflicts, inconsistencies, and so on--that are much harder to resolve than if there is a copy on the servers.

Once that happens, other clients will ask to download from the servers, notice that it's on the LAN already, and take it from there. I bet if you install a new client on the same LAN, or share a folder between clients on the LAN, or add a folder to selective sync, once the files are already there... it will be a competitive speed.

This is like comparing apples and oranges. I'm inclined to think that the creator of this video is biased towards AeroFS.

Funny, I'm actually biased towards Dropbox. I use it for most of my stuff because of the versioning and the Packrat feature. But there's no reason for Dropbox not to do direct LAN syncing, and no reason for AeroFS not to offer infinite versioning. It may be more complex to design, but definitely feasible. So I don't think that this is an apples-vs-oranges comparison.

> But there's no reason for Dropbox not to do direct LAN syncing, and no reason for AeroFS not to offer infinite versioning.

Really, no reason? None at all?

For "infinite versioning", it's obvious - there's no upper bound on the amount of data the server has to store. Storage space costs money.

As for no direct LAN syncing, someone's already mentioned the reason - consistency. Syncing is already a hard problem, and it's dangerous to allow more things to go wrong because the clients synced but the server didn't receive all the data.

So what if there's no upper bound? Treat it like normal data and charge the normal price. Spideroak already does this.

What would this get someone who already has Dropbox and Air Drop on OS X?

Answer: compatibility! Now that walled gardens are going up, there's pain in getting through the walls. If they can make this compatible with Dropbox, it would be a great combination. There's also the possibility that Dropbox could add Bonjour/Rendevous to their client side software, reducing AeroFS to just a feature.

Not really, the attraction of AeroFS is not having a connection/reliance on S3. So for something like 1Password syncing, it would be great not to have a copy sitting somewhere in the cloud.

If you use a strong master password, and given that 1Password uses a secure key-derivation function, does it really matter if your 1Password file is sitting somewhere in the cloud?

Yes.

My take on it is if there is no/little benefit of putting something in the cloud, there is no reason to do so particularly if it's something security related.

This is one of my major annoyances with Dropbox — I'd love to shift everything to my Dropbox, but the slow sync speed is painful.

I signed up a long time ago for an AeroFS invite but had no joy. Anyone got one they could share? I'd love to try this out.

Sent you an invite :)

Received! Thanks! I noticed from your profile you wrote LOGO for python — I did the same for processing! http://terrapin.sourceforge.net/ Great minds :)

Can I have an invite too? Pleeaassseee. :)

Sure, I'll just send it to the email address that you've conveniently included in your profile. Oh wait :)

Ok. Updated. (Also, sorry!!)

Update: Got more invites, sent you one.

Thanks man!! You're awesome. :)

can I have one as well, please?

Sent :) and this was my last one.

Many thanks!

hello, can i ask for an invite? my address is rymngh[at]gmail.com..

Sent.

thank you sir.

If anyone else has an invite, I wouldn't mind trying this out

What protocol does AeroFS use?

I would love an invite if anyone could spare one. My email address is hotspam{at}gmail.com. I've been on the sign up list for over a year but haven't heard a thing. Thanks in advance!

AeroFS looks cool but I'm not too keen on having to run Java just to run AeroFS given that I had disabled Java on MacOS due to the recent exploits.

I wouldn't worry about it.

The Java to be disabling is the Java browser plugin, which allows un-trusted web sites to run applets in a sandbox on your machine. As you know, there have been a lot of issues with Java sandboxing.

However, AeroFS runs locally (as far as I understand, I don't have an invite) so should be fine as long as you trust the provenance of the code.

I believe those exploits are Java applets breaking out of the browser-plugin sandbox, which is designed to limit application access to even simple things like files and class loading. To be clear, these aren't strictly JVM vulnerabilities, and more plugin security-model vulnerabilities.

Java "desktop" applications don't have such a sandbox and thus they already have access to things that the vulnerabilities enable (such as manipulating system properties). Really, we ought to treat Java desktop apps no differently than native binaries, bounded by the the OS' security model.

It's nonsense, they are 2 different technologies

Reply on news.ycombinator.com