Hacker News Reader

Comment by victorhooi | Hacker News Reader

I'm not sure what value that would add here...

The customer's issue seems to be that they have a very large number of files, and are trying to sync it with a 32-bit desktop client, and are hitting memory issues.

Dropbox themselves state this on their websites:

https://www.dropbox.com/help/space/file-storage-limit

> Note: The performance of the Dropbox application may start to decline when you store above 300,000 files.

I've done some comparisons across various companies sync clients, and it is a tricky problem.

(Disclaimer: I work for Google).

View on HN

1. Read list of files present on remote & write to file A

2. Read list of files present locally and write to file B

3. Sort files A & B

4. Read files A & B to update remote and local files

This can be done with less than a meg a memory.

You are forgetting about monitoring for changed files to resync those quickly.

Get notification of file change, binary search the list of files, make adjustments. This can also be done on a small amount of memory, independent of file count.

I'm sure it's tricky to update the existing software, but as a generic problem syncing files with O(1) or O(logn) memory is not very hard.

Windows' notification API gets an overflow (ERROR_NOTIFY_ENUM_DIR) if you change many files quickly, like during a copy/paste in explorer. You then have to re-scan the dir manually.

While rescanning, the files may continue to change. You may or may not get a subsequent event for a file that changed during your rescan.

One example of related issues: https://github.com/libuv/libuv/issues/626

I'm aware of how awful notification APIs can be, but that's a separate problem entirely from per-file memory use.

Where's your sync client then?

(I've been very slowly designing my own, and the first thing for me was coming up with a list of awkward cases ..)

This seems like a good project. Thanks.

> Where's your sync client then?

I guess I was unclear here. It's hard to make a polished user application. The not-hard part is just the core algorithm selection. I have enough minutes in the day to consider algorithms, but not enough tens of hours to write suites of quality software products.

The problem is unfortunately not as simple as a naive first pass might assume.

There's a lot of nuance involved here - and the fact you have to do bi-directional sync, with a reasonable expectation of "real-time" makes it tough as well.

Also - the fact that other desktop clients (e.g. Dropbox, Box.net) exhibit similar performance characteristics (e.g. Dropbox advertises a 300,000 file limit - I believe this customer was going on 700,000 files) would also probably indicate it's not quite that simple.

At 1 meg, are you not counting storage of the file names and paths?

The user has 700000 files. Each is probably >10 char of UTF8.

In step 4, you also want to catch things like the user moving 50000 files to a new subdir, or renaming a top level subdir, while the sync app is not running.

I'd understand if you don't want to discuss details but for the life of me I cannot understand how the total number of files can affect anything. What function does a sync app do which depends on total number of files? I can just do the sync file by file. It will take 10x more time to sync 100 files than 10 files. But why the brokenness?

The question is whether "declines" literally means "declines", or it's code for "breaks".

Reply on news.ycombinator.com