Introducing streaming sync: supercharged sync for large files | Hacker News Reader

Introducing streaming sync: supercharged sync for large files

blog.dropbox.com

27 pointslalwanivikas8 comments

The technical details of what they changed are here: https://tech.dropbox.com/2014/07/streaming-file-synchronizat...

Looks like it still doesn't fix the LAN scenario, where one has to wait for a server upload before the file is synced through the LAN.

With this new version, it seems like the LAN sync will happen in parallel, but the file is still confirmed only after a full upload to the remote server and thus local peers don't see it appearing immediately in the shared folder.

That seems like a policy decision, not a technical limitation. If Computer 1 uploads a file to Computer 2 and then Computer 2 deletes the file before it gets uploaded to the server, it's going to be difficult to sort that out. Letting the server be the final authority on a file's state is a lot simpler.

Why? The server already knows about the file, who created it, etc. Why would they need its contents?

If the server doesn't know that Computer 2 had the file earlier, it might re-send it when the upload is complete and it tries to sync other computers. Also, what if Computer 2 modified the contents?

If the server doesn't know that Computer 2 had the file earlier

My point is that you could keep the server as the master, without having to upload the contents before pushing to other machines. Imagine this exchange:

  Computer 1 > server: I have a new file X, created on timestamp Y, with hash Z
  Server > computer 1: ack
  Server > computer 2: there's a new file X, etc. Computers [1] have it.
  Computer 1 > server: ack. I'm requesting it from computer 1.

The server doesn't actually need the contents, just the metadata.

By the way, this is all fairly archaic. Nowadays with DVCSs, you can do much niftier things. I have a network of five git-annex nodes, some of which have two degrees of separation (my Android tablet syncs with my personal laptop, which syncs with my VPS, which syncs with my work laptop) and it all runs smoothly.

This may be be related: https://www.dropbox.com/developers/blog/48/how-the-datastore...

wonder what's on the backend? :P

Reply on news.ycombinator.com