Hacker News Reader

Comment by pizzafeelsright | Hacker News Reader

data loss is the largest concern

I still do not trust de-duplication software.

Agreed, "I made a deduplication software in my garage! Do you want to try it?" is a terrifying pitch.

I've been writing a similar thing to dedupe my photo collection and I'm so paranoid of pulling the trigger I just keep writing more tests.

Dedupe seemed more interesting when storage was expensive, but nowadays it feels like the overhead you get from running dedupe, in most cases, is priced-in. At least with software like CommVault for backups, dedupe requires beefy hardware and low-latency SSDs for the database, If there is even a few extra milliseconds of latency or the server can’t handle requests fast enough, your backup throughput absolutely tanks. Depending on your data though you could see some ridiculous savings here that make it worth the trouble.

I’ve heard many horror stories of dedupe related corruption or restoration woes though, especially after a ransomware attack.

Even using sha-256 or greater type of hashing, I'd still have concerns about letting a system make deletion decisions without my involvement. I've even been part of de-dupe efforts, so maybe my hesitation is just because I wrote some of the code and I know I'm not perfect in my coding or even my algo decision trees. I know that any mistake I made would not be of malice but just ignorance or other stupid mistake.

I've done the entire compare every file via hashing and then log each of the matches for humans to compare, but never has any of that ever been allowed to mv/rm/link -s anything. I feel my imposter syndrome in this regard is not a bad thing.

Now you understand why this app costs more than 2x the price of alternatives such as diskDedupe.

Any halfway-competent developer can write some code that does a SHA256 hash of all your files and uses the Apple filesystem API's to replace duplicates with shared-clones. I know swift, I could probably do it in an hour or two. Should you trust my bodgy quick script? Heck no.

The author - John Siracusa - has been a professional programmer for decades and is an exceedingly meticulous kind of person. I've been listening to the ATP podcast where they've talked about it, and the app has undergone an absolute ton of testing. Look at the guardrails on the FAQ page https://hypercritical.co/hyperspace/ for an example of some of the extra steps the app takes to keep things safe. Plus you can review all the proposed file changes before you touch anything.

You're not paying for the functionality, but rather the care and safety that goes around it. Personally, I would trust this app over just about any other on the mac.

More than TeX or SQLite?

Best course of action is to not trust John, and just wait for a year of the app out the wild, until everyone else trusts John . I have enough hard drive space in the meantime to not rush into trusting John.

Having listened to John for 10 years, he’d be the first to encourage you to wait around to trust his app.

> I'd still have concerns about letting a system make deletion decisions without my involvement

You are involved. You see the list of duplicates and can review them as carefully as you'd like before hitting the button to write the changes.

Yeah, the lack of involvement was more in response to ZFS doing this not this app. I could have crossed the streams with other threads about ZFS if it's not directly in this thread

Question for the developer: what's your liability if user files are corrupted?

Most EULA’s would disclaim liability for data loss and suggest users keep good backups. I haven’t read a EULA for a long time, but I think most of them do so.

I can't find a specific EULA or disclaimer for the Hyperspace app, but given that the EULA's for major things like Microsoft Office basically say "we offer you no warranty or recourse no matter what this software does" I would hardly expect an indie app to offer anything like that

Reply on news.ycombinator.com