I ran fdupes on a random directory once, very interested to see what it would do. After spending an extraordinary amount of time checksumming everything (I don't even remember if it had a progress bar) it spat out a
very confusing text file I wasn't able to make sense of at all.
My plan is to write a multithreaded disk [re]indexer (since full indexing sadly seems to be the only solution at the moment; no filesystems have a "fast dump" mode yet), along with a bunch of utilities that try to make sense of the generated index.
(By "[re]indexer" I mean that, if you pass the indexer an old database, and any file has the same size and mod/ctime as what's in the old database (or matches any other rule you specify), the checksum for that file won't be recomputed. You could also pass in an old database and simply tell the indexer to scan a specific set of subdirectories on disk that you know have changed, and the indexer would copy simply copy the old database to the new for every path/file outside the ones you've specified.)
Building an index and then working off of that has a lot of benefits - instantaneous visualization without needing to rescan the disk on every program start, realtime similar-file searching, listing every file with the same checksum as you browse around, etc.
Obviously this tool will wind up on here when I eventually (:P) do write it. I probably should think about starting on it already but in all seriousness I'll likely be motivated to start working on it once it looks like I'll finally be able to put my ZFS pool together. (Hopefully that plan doesn't backfire, and I don't get the remaining $ all at once and abruptly wind up with a bunch of disks and no tool... and we wind back to the first approach, of just starting now)
Local storage is definitely the goal, once I can afford it (expensive medical issues that prevent me from straightforwardly getting a traditional job are really fun, FYI xD). I don't really find cloud storage flexible enough for my requirements at this point.
Sadly the NBN has been sufficiently politically meddled with that it's not likely to be 1Gbps - and indeed current installations are only 100Mbps with no path past that. But I heard recently on a news story (on the radio of all places) that if people need an upgrade path with the NBN, that that will be available. Nobody mentioned how much said upgrade would cost, but I'd be willing to drop a few hundred on it once I have a job. What on earth issues do you have with your connection?! lol