Comment by jimmies | Hacker News Reader

Comment by jimmies | Hacker News Reader

If you plan to use it for your personal traditional storage HDD, you have to keep in mind that it will render most (if not all) data recovery tools naught. Tools such as PhotoRec rely on the file headers to determine the file type to recover the data.

I have a long-term storage HDD that I thought I would never need data recovery on it until I fucked it up good by a wrong partitioning command. Normally those types of mistakes can be recovered by TestDisk but not that time. I realized my only hope was to use PhotoRec to recover what I can from the garbage that I overwrote on the disk. Thanks to PhotoRec I could recover most of them with messed up names. I was so thankful I didn't use any fancy compression algorithm.

Tools such as LVM, SSD backing storage, btrfs, compression and stuff are all nice when you understand their limitations. For now, I don't use all of my storage so I just create ext3, ext4, and exFAT partitions to store my data for the maximal chance of recovery, whether it is due to hardware or software.

I'm not saying you're wrong, but "what about data recovery" has been used to argue against every advance in storage, even massive ones like SSDs. Have backups and unshackle yourself from fear.

I'm also old enough to remember how buggy DoubleSpace was and, truth be told, that still kinda puts me off disk compression.

NTSF compression is pretty transparent and robust. I usually have 60% of my main disk compressed (laptop, I need the space especially with multiple Visual Studios-installed) for many years and no failures in any way.

As I understand it you've never had to recover your compressed files.

I run off Google Drive or GitHub for all files. I think most people should. Local data is ephemeral.

On the other hand, you should make backups anyway.

On the other hand if you are using full-disk encryption then you will be fucked anyways.

Depends. If you mean soft compression (not ATA level crypto) you may be able to back up the header externally and still decrypt it. The data will be scrambled where it would have been scrambled anyway, and unmodified regions will remain unmodified.

Using something like VDO for personal storage would be pretty pointless.

Unless, of course, you have a ton of similar virtual machines or other data with high amount of 4 kB block size aligned redundancy.

If your data directories have much redundancy (so deduplication can help), or are nicely compressable, then VDO can also make sense for home setups. Like companies, you would consider things like "does it make sense for my kind of data", "do I have enough CPU cycles/ram to spend on getting storage down to some degree" and "am I ok with the additional layer, so more complicated setup".

imo VDO makes sense where you deal with high speed, low capacity media. I've used BTRFS compression to hold huge disk images in RAM or an SSD while I fully strip/TRIM and repack them (say from RAW to qcow2 or similar). Of course this is usually a case in which I've imported a physical disk image of a 500GB drive, but the actual data on it was only ~50GB or less.

Edit: Sorry not the disk image, the temporary overlay for zero blanking. But sometimes smaller disk images for the whole thing.

Recently I've also experimentally been holding some older games in RAM on a compressed image.

Reply on news.ycombinator.com