Digital hoarders: Our terabytes are put to use for the betterment of mankind | Hacker News Reader

Digital hoarders: Our terabytes are put to use for the betterment of mankind

arstechnica.com

92 pointsfeross45 comments

I hoard because shows, youtube videos (youtube-dl) and websites disappear. I use some GTD methods to organize my data (e.g. folders a,b, c...z) I use recoll to find data https://www.lesbonscomptes.com/recoll/

I would like to use an online backup service but I would like to have my data encrypted.

I tried https://www.hetzner.com/storage/storage-box

But data throughput to an encrypted storage was abysmal. I tried several methods, even included running an encrypted filesystem in a 10TB file. I tried https://www.cryfs.org/ but nothing worked good and reliable.

I looked into https://www.infomaniak.com/en/kdrive/ but the encryption method seems like snake oil.

Any suggestions?

You can keep a local Borg repository (https://www.borgbackup.org) and copy it anywhere in the cloud with something like https://rclone.org.

To quote the website, BorgBackup gives you:

- Space efficient storage of backups.

- Secure, authenticated encryption.

The encryption is performed on your machine, and you don't need any special features from the provider(s) beyond simply storing a lot of small chunks.

On Linux ZFS has has an encryption feature developed by datto.com.

They developed this feature for their own hosted encrypted backup service.

If you have you data encryped in ZFS like that locally, you can use zfs send to send them encrypted snapshots of your data while only ever having to transmit the diffs.

I am in the process of setting up something like this with my own backup server right now. I'm not 100 % confident yet that I'll get it working though.

Have you looked into tarsnap.com?

I use rclone for that purpose.

https://rclone.org/crypt/

There's tarsnap (tarsnap.com). It always pops up here on HN when talking about really secure encrypted backups. Not sure it solves your use case, but I figured I'd mention it since nobody else did.

I use restic with backblaze. It works great and super cheap. Kinda slow, but you can always order one of their portable drives and ship it back to them for the initial upload.

60 USD for unrestricted per year? Can I store 10 TB there?

You can. I haven't checked their products in a long time. Look out for:

[1] Old file versions and deleted file retention - default was 30 days I think. You pay extra to make it a 1 year, and more to make it forever forever.

[2] Keep connecting the external drives or lose that data. Because the last time I checked if you don't connect your drive (or maybe the computer/laptop too, not sure) regularly the files are marked as "deleted" and [1] will kick in even though you'll keep paying (unless you are paying extra).

[3] They didn't have restore option from the app - either use their website (I am not sure how the partial/interrupted restores were handled from a browser) or they will send a restore drive to your home (it was free iirc).

If it works for you you can give the service a shot. Their Mac app, at least, is really really good (way better than competition including the relic called Time Machine).

Why do you need to encrypt YouTube videos?

I think it is just good practice to have all your data encrypted.

I don't understand the negativity the article associates with "hoarding" data.

It is the nature of the internet that any content or website you see can disappear at any moment without a prior warning or an explanation. From an author deleting their earlier work they were later dissatisfied with, to copyright claims, to people losing interest in their hobby project, to site owner forgetting to extend their domain, to website becoming impossible to find in search engines.

It always pays to be properly paranoid and have backups in case the worst happens. Storage is cheap, so what's exactly the downside?

My personal archive may not bring "the betterment of mankind", but it saves me from disappointment when looking up older content that has since disappeared, and also allows me to help other people find what they're looking for.

One that caught me three or four times was Udemy, where I'd paid for a course heavily discounted on sale meaning to study it at some point later. Much later, I'd go to the course page and find that the author had withdrawn it and Udemy were offering me a refund of the discounted price. All within their rights, but it feels annoying.

When you think about it is quite insane that you can find rock tablets from thousands of years back with text on them but we can lose all the data on the internet just like that.

Is that insane? It’s not like every stone tablet or parchment or scroll survived, many were lost. Some were lost immediately, some after a few years, some after a few centuries. Not every word on the Internet survives for long but not every word written on stone tablets lasted very long either.

We really should go back to having people copy things on stone tablets, or at least encode them on sapphire discs..

I "hoard" because there is a lot of media I own that I can't find elsewhere. I had stopped buying physical media and ripping them to my disks in the last 2 years and moved full time to streaming - Spotify for discovery, Apple Music for maintaining the library (both are cheap here in India).

But recently I saw that a lot of music I owned was missing from both services. That made me decide to go back to my old ways. Also, having spent the effort in cataloging and ripping my collection has made me intimately familiar with my collection and has been an activity I can partake in to relax/wind-down.

>But recently I saw that a lot of music I owned was missing from both services.

It's not just that, it's also that apple "transparently" changes the version of the music that you "bought". I had a version of a musical album with a particular song in it. When my ipod died and I re-downloaded that album on my new computer, I found that they had helpfully "upgraded" it to the newer release (without the song I wanted on it). And there's no easy way for me to get it back unless I find a physical copy somewhere and buy it again.

Added to that concern is that streaming services often feature only recent versions of albums that are pitched as “remastered”, and they might replace earlier audio with whatever the label now sells. The supposed remastering is not any improvement on earlier digital-era releases, it just reduces the dynamic range because labels know that people are listening to music on their phones in noisy places these days. So, if you have a nice stereo and listening environment and you want to get a version of the album with actual room to breathe dynamic-wise, your only choices are buying a used CD or torrenting it.

I'm in a similar position. The nice thing about the NAS is that it doesn't have a "leaving soon" section.

Another good thing about using it for most video consumption is that disruption of internet service becomes a temporary issue upstream, rather than game over. Quality downgrade because of a drop in bandwidth doesn't happen either.

Similar to you, I'm converting CD to Flac one cherished album at a time, taking a bit of time to enjoy it and tweak the artwork, etc.

How do you handle listening to songs on mobile devices across the internet? I'm currently using Resilio Sync (trying to hack around for selective sync support in Syncthing) meanwhile to get albums I want onto my phone.

I haven't yet looked into things like Plex, Kodi or Emby.

I haven't looked into playing back Flac files from Mobile devices. I'm currently using a client on the Mac called Audirvana, but intend to use something on the Raspberry Pi like Volumio for general listening, connected to an amp and speakers. The Flac files are just available on the LAN from the NAS. I think Synology does support Plex too, which would let you listen to tracks away from home.

I also remain a keen member of filesharing communities because I want to actually own my music and films, and I know it could disappear from streaming sites.

But another advantage of having my own collection is that it lets me enjoy all this media at my country cottage, where the only internet access is 4G that gets throttled to hell after just a few gigabytes per month. I can torrent Blu-ray images in the city with my fiber connection, and then keep a backup hard drive at my country home. That has proven a lifesaver during this long virus lockdown when I am stuck in the countryside and can’t travel back and forth.

I do not like the shaming tone of this article. Petabyte server today is a small box that fits anywhere. Meanwhile some people have hundreds of shoes or clothes, but nobody calls them "hoarders".

Anyway, I have offline storage, because my internet connectivity is slow. I do not like waiting, so all my content is downloaded overnight with a few cron scripts.

People with lots of clothing and shoes and other things absolutely might be criticised for their behaviour, and may be called hoarders. (I have direct personal experience of such people.)

There is definitely a mindset —at its most extreme, a pathology— associated with hoarding behaviour.

Digital hoarding may mostly avoid the negative life consequences of hoarding —as you say, you can store a lot of data in a relatively small physical space— but this doesn’t change the nature of the psychology behind the behaviour. (Although I’d wonder how many digital hoarders really have a small neat modern NAS as you imply, versus a room full of old racks?)

Interestingly, the rationalisation given in the headline (“betterment of mankind”) isn’t so very far away (although maybe more grandiose) than the rationalisations that physical hoarders offer to justify their behaviour, around the perceived intrinsic value of their belongings.

> this doesn’t change the nature of the psychology behind the behaviour.

The "psychology" behind the behavior isn't the problem with hoarders, it's the actual hoarding. The reason we try to change hoarders' thinking is because of the physical consequences of their hoarding - we define this as illness because of its distressful results, if it had no consequences it wouldn't be illness.

edit: I think you're accidentally using circular logic to rationalize a bit of magical thinking (person who obsessively collects garbage is ill, therefore person who obsessively collects anything is ill.) It's not the obsession that's the problem, it's the garbage and the distress.

I know that some of the members of the organized data-hoarder community are spending huge amounts of money on disks month after month, and it really bothers their spouses. So, even if there is no physical mess being created in their homes, their obsession is still affecting their lives and other people around them.

You raise some interesting points. A few thoughts.

Generally, psychological conditions can be considered on a spectrum, an analogue scale (i.e. from you have none of this trait or condition, to you display huge amounts). You're absolutely right that the point at which Hoarding Disorder becomes diagnosable (currently, per DSM-5) is determined by the real-world effects on someone's life becoming severe and displaying a number of features. [0]

However, I'd argue that for this subject, considering only the DSM-5 diagnostic criteria is too simplistic:

a) There are plenty of people whose hoarding behaviour falls short of the formal diagnostic criteria, but which do still cause some level of problem for them in their life, or for other people who are in their life. (For example, you could have someone whose home conditions don't meet the diagnostic criteria, but who no longer receive visits from family or friends because of the state of their house.)

b) The diagnostic criteria to a large extent represent a traditional, common-sense, real-world societal view - i.e. it's fine to collect things, until it starts to make you unwell, and unsafe in your physical environment.

c) The diagnostic criteria are updated seldom, and slowly. [1]

As such, it's not infeasible that DSM-5 might lag behind (potential, arguable) alternate forms of hoarding disorder - such as digital hoarding. Consider someone who spends huge amounts of money on storage media and electricity, to the detriment of their financial well-being and maybe close relationships. Or someone who spends so much of their time downloading and categorising their digital belongings, that they miss sleep and make themselves unwell, or avoid social engagement and have detrimental effects on close relationships. These cases might not be diagnosable within DSM-5, but I'd argue that they would still be representative of psychopathology... and may in time (in DSM-7? :) ) be diagnosable in their own right.

[0] https://www.ncbi.nlm.nih.gov/books/NBK519704/table/ch3.t29/ [1] https://www.psychiatry.org/psychiatrists/practice/dsm

Hmmm for a petabyte you'll be closer to the size of a desk than of a small box.

True. You can fit 150TB in a 4U rack mounted server fitted into an IKEA coffee table though, using entirely off the shelf consumer parts.

Supermicro's got a top-loader 90-bay chassis that fits into 4U. Filling it with 16 TB drives will give you 1.44 petabytes (not pebibytes, because hard drive salespeople), minus whatever redundancy you'd like to build in.

I wouldn't consider that chassis to be an "off the shelf consumer part" though. Also comes in 45 and 60 bays. :)

I'm not entirely sure that IKEA coffee table would be able to withstand the weight of 45 or 60 drives plus chassis, either.

I don't see why not.

750g/drive * 60drives + 45kg chassis = 90kg. I've stood on top of IKEA Lack side tables, and a) I weigh more, b) with the chassis, the weight is going to be better distributed (lower mounting and no flexing). It might slack a little.

Anyway compared to the price of the disks, a cheap small rack (they come in living-room compatible form factors as well. don't ask) wouldn't break the bank.

Only inconvenience would be the top-loading aspect. The Lack fits 19" exactly, and besides can't provide the needed counterweight, so you can't have sliding rails. Can't mount lower either, a Lack would only barely fit 4U.

1 Petabyte in a small box? I think the biggest drives are 15 Gigabytes or so? It would take more than 50, that is a loooot of weight!

Yup, a lot of weight, now imagine 3 Petabytes[0]

[0]: https://www.youtube.com/watch?v=EtZXMj_gUjU

Yes, hoarder here. Started with scanning old family photos, progressed from there. Was psyched to find youtube-dl after I had discovered many of the YouTube videos I had playlisted years ago, are now missing. Perhaps not very interesting things to most: research footage testing some aspect of the X-20 Dynasoar space plane, etc. Many things gone missing I don't even know what they were since YouTube doesn't indicate what the missing video was. So, yeah, hoard it is. "Japanology"? Got all I could find. "How It's Made?" Yep. Live footage of the band "The Embarrassment" sleeps on my drives as well....

2nd youtube-dl

Feels insane to store youtube vids, but quite often they are simply gone for one reason or other after a few years.

I'm definitely a digital hoarder but not on the level of the people in the subreddit. It started when I was in grad school and a web site I wanted to reference went down before I was able to archive it and it wasn't in archive.org and youtube wasn't a thing. So I started downloading copies of anything I thought relevant to my research. I know I have some rare stuff that is gone from the internet. One of these days I'll get around to posting them on archive.org. Until them I practice 3-2-1 backup to make sure I don't lose anything.

My only regret about digitizing my music over the years is that I didn’t care about the quality as much when I was younger. I’m older now - with better equipment - and would love if everything was FLAC rather than mixed .mp3

Years ago, I was a member on a private file trading site that only exchanged old and unusual art films. The kind of stuff that is entirely non-viable commercially and will never be found on a streaming device. It shut down - but I miss having that small corner of the Internet.

Just picked up two more 36 bay sata 4Us for the garage on eBay - only $500 each!

Shoot me an email if you have bulk old/used sata drives for cheap. I’m buying.

Are multiple used smaller drives really more cost effective than fewer new larger drives once you take into account cost of power, HBA, backplane and lifetime of used drives?

I'm a data hoarder too (proof): https://i.imgur.com/j9jk9OP.jpg

But always been curious about cost so let's do some sums...

With a used 3TB setup. Assume 12x3TB = 36TB storage. A reasonable 2U server would be needed to support that which would include a backplane and HBA. Power consumtpion would be 12x 5W (HD) + 10W (HBA) + 65W (server). Total is 135W. Power cost in UK for one year at 24/7 is approx £176. Assuming used 3TB drives give us two years of reliable operation and cost £30 each to buy (going price on ebay a few months back) total cost to buy would be £360 of just the drives (not including the server). Add cost of electricity for two years operation and we get a total of £712. Average the cost per year (£712/2) and we come to £356 per annum for the 3TB drive setup.

Now with new 12TB drives. 3x12TB consumes 3x 5W of power + 30W for the server (no backplane or HBA needed). Total is 45W. Power cost in UK for one year at 24/7 is approx £60. Cost of drives would be 3x£300 = £900. Assume we get 5 years of life out of these new drives, plus electricity costs of £60 - over 5 years total would be £1200. Averaged the cost per year (£1200/5) and we come to £240 per annum for the 12TB drive setup.

(I have not counted the additional cost of the 2U server for the 3TB scenario since that would complicate the math since servers tend to last longer than drives.)

So I'm wondering what the appeal of large servers with lots of small drives is when back of the envelope math suggests it costs more in the long run. Unless of course it's because you like those blinkin' lights :) https://i.imgur.com/4ypumBQ.jpg

What level of redundancy are you calculating? I’m not comfortable with less than RAID6/Z2 on large drive arrays, so 3 12TBs is only 12TB usable. 12 3s I’d probably do 2 6-drive volumes, each with 2 drive redundancy, so 2x4x3 or 24 TB usable. That’s enough to flip the math.

> Are multiple used smaller drives really more cost effective than fewer new larger drives once you take into account cost of power, HBA, backplane and lifetime of used drives?

Resilver times for drive replacement is generally much faster for smaller drives.

Oh, I got one of them to be a disk duplication machine for the next Defcon. Only one of them will be in full file server service, and I intend to do an 8-20 disk array of 12TB disks.

That will still leave me with over a dozen slots for a secondary array of cheap refurb disks of any kind I can come across.

Power is cheap here; my current hobby bottleneck is drive cost. Ideally I’d be able to buy a few dozen extremely cheap disks that have been pulled from service that someone wants to dump as a lot.

The calculations might depend on whether the drives are powered on all the time.

That's my sense too. Unless someone has to have access to that data relatively quickly all the time I don't see why the system couldn't be left off.

Even if the data is accessed frequently, lets say once every day or few, it could still be left off most of the time if someone is OK with waiting a few minutes for everything to power up.

Reply on news.ycombinator.com