1. Read list of files present on remote & write to file A
2. Read list of files present locally and write to file B
3. Sort files A & B
4. Read files A & B to update remote and local files
This can be done with less than a meg a memory.
2. Read list of files present locally and write to file B
3. Sort files A & B
4. Read files A & B to update remote and local files
This can be done with less than a meg a memory.
There's a lot of nuance involved here - and the fact you have to do bi-directional sync, with a reasonable expectation of "real-time" makes it tough as well.
Also - the fact that other desktop clients (e.g. Dropbox, Box.net) exhibit similar performance characteristics (e.g. Dropbox advertises a 300,000 file limit - I believe this customer was going on 700,000 files) would also probably indicate it's not quite that simple.
The user has 700000 files. Each is probably >10 char of UTF8.
In step 4, you also want to catch things like the user moving 50000 files to a new subdir, or renaming a top level subdir, while the sync app is not running.
I'm sure it's tricky to update the existing software, but as a generic problem syncing files with O(1) or O(logn) memory is not very hard.
While rescanning, the files may continue to change. You may or may not get a subsequent event for a file that changed during your rescan.
One example of related issues: https://github.com/libuv/libuv/issues/626
(I've been very slowly designing my own, and the first thing for me was coming up with a list of awkward cases ..)
I guess I was unclear here. It's hard to make a polished user application. The not-hard part is just the core algorithm selection. I have enough minutes in the day to consider algorithms, but not enough tens of hours to write suites of quality software products.