Hacker News Reader

Comment by freeqaz | Hacker News Reader

So 300TB * 365 = ~110PB for a mirror to have 1 year of retention.

That's pretty insane lol. How many mirrors are there that can actually manage that much storage? If you're using 20TB disks that's ~5500 disks per year with zero redundancy. Double or triple that for a bare minimum... not counting the load of actually serving that data to everybody.

How is this economical for anybody at this point? Or are these Usenet mirrors all massive businesses that can support running hundreds of PBs or storage and I'm just naive?

View on HN

> Or are these Usenet mirrors all massive businesses that can support running hundreds of PBs

ding ding ding. The market consolidated toward only a few large backbone operators. Here's a picture of what the network looks like: https://svgshare.com/s/14tF.svg

I'm also curious -- how come you hear big stories about governments going after torrent sites, which don't even host the pirated content.

But I've never heard big stories about the government going after big Usenet providers. Do they? If not, why not? Or does it just not make the news? Or am I just not paying attention?

Usenet files get taken down all the time. Rights holders send notices to the Usenet providers and they remove enough of the binaries where they can't be repaired by the PAR files.

Indexers are separate entities. Those are the ones who catalog and host the .nzb files. You see those go offline far more often but Usenet is still pretty small compared to torrents so they don't get reported as much.

> But I've never heard big stories about the government going after big Usenet providers.

The most litigious government has a 'safe harbor' provision most of the major Usenet backbones comply with: https://www.copyright.gov/512/

And they do take down a substantial amount of content every day, so while everyone knows piracy is common there, what more are they supposed to do? At this point the main effective thing governments can do to make it annoying to use Usenet for piracy is target the indexers (and they do this, but more always move in to replace the ones that are shut down)

AFAIK, files uploaded to Usenet still need to be ASCII encoded using something like uuencode or base64, which increases file sizes to something like a third larger than their original sizes. So you can decrease the amount of data you need to store quite significantly by only storing the unencoded versions.

After that, deduplication can probably bring you down at least another 50% - how many of these files are just the same things being posted over and over? Probably quite many of them. Store a file once with a database tracking its SHA1 hash, and whenever you see a file with the same hash come in, throw it away and instead store a reference to the first file.

Most posts are yEnc encoded these days https://en.m.wikipedia.org/wiki/YEnc

Reply on news.ycombinator.com