Hacker News Reader

Comment by cantrevealname | Hacker News Reader

Anyone remember Napster? There were extremely obscure musical recordings that were instantly found on Napster that are impossible to find even today. It took a long time for anyone to rebuild that data to the level of usefulness that Napster had.

If Sci-Hub is forced to shut down due to legal pressure, technical problems, running out of money, or the founder losing interest, it would take a long time for anyone to duplicate Sci-Hub if they started from zero.

I think a good way to keep the cause alive is to backup or mirror the 85,483,811 papers hosted on Sci-Hub. That is, if she's willing to allow it (maybe it's already being done, but how widely?).

In the case of Sci-Hub, you'd need between 85 and 850 TB for each backup or mirror depending on whether the average paper size is 1 MB or 10 MB.

View on HN

> I think a good way to keep the cause alive is to backup or mirror the 85,483,811 papers hosted on Sci-Hub.

There's already an effort where the data is broken up into 100GB chunks [1]. Looks like the entire database is duplicated now at least 10 times. [2]

> Anyone remember Napster? There were extremely obscure musical recordings that were instantly found on Napster that are impossible to find even today.

That existed (and still exists) - there was oink.cd, then what.cd, and now RED. But each incarnation has lost things. Each time you burn down the library, you're bound to lose something. It's disappointing there isn't a legal way of cultural preservation. We need something like a crowd sourced digital library.

[1] https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_...

[2] https://phillm.net/torrent-health-frontend/stats-filtered-ta...

Yes p2p really had the largest catalog and still surpasses what’s available on Spotify or Apple Music. You could get obscure live recordings, the smallest bands, everything was there. If you consider bootleg recordings, the volume of music produced is many times larger than what any one centralized service could offer.

As far as backups for future generations go, I'd hope archive.org has a backup of it all, regardless of whether or not they make it available. I certainly hope they have a program for keeping copyrighted (or otherwise private) data backed up for safekeeping for future generations without making it publicly (or privately) accessible; sort of a digital time capsule, if you will, that's kept securely locked until a future date.

For me soulseek was amazing to find obscure recordings.

Napster was ok. Oink on the other hand...

Or its spiritual successor, What.cd

The closure of what.cd should be up there with the burning of the library of Alexandria.

Isn’t there a torrent available? I thought I had heard that scihub is immortal because of torrented copies.

Only the existing bits. New publications wouldn't be included, so eventually the archive would become less relevant and stop being shared. There's significant cost for the required storage (currently 77TB according to andyxor's comment).

108T 104T 3,9T 97% /home/scihub

77TB (or 104 TB) sounds expensive, but many people reading this could buy that kind of storage with just one month's salary. (SeaGate IronWolf Pro 16 TB, currently $521 on Amazon)

77 / 16 = 5.5 (6 disks) at $521 per disk turns into $3126. Seems only 11 countries in the world has a higher salary than $3126[1], and probably people wouldn't buy something like that with their entire monthly salary, so at least $5000 a month is required to be able to just dispose $3126 for some drives. That leaves 5 countries with a average salary higher than $5000.

Granted, the source might not be the best, nor using average salaries as a signal either, but to say that many people can spend something like that on a whim seems ignorant at best.

- [1] https://www.worlddata.info/average-income.php

They said "many people reading this" and HN's biggest audience is US tech workers. It wouldn't be ignorant to say that many US tech workers can afford that.

The table doesn't list the same as the map. For instance Norway and Sweden is included in the map, but not in the table. And I don't think the average reader here should be compared to the average income around the world. Rather a top single digit percent.

This is why broad averages aren't always the most useful statistic. The number of complete mirrors required to make the data trove fairly robust is much lower than the number of people in the world, and the prior commentor's qualifier was "people reading this", which is likely a biased sample and includes many people working in technology in the richest countries in the world. It's unlikely that said population's income is clustered around overall national averages, and only a relatively small number of outliers are required to set up adequate mirroring.

you could do it for about $2300 by shucking external hard drives. I picked up a couple 10TB WD drives last year for $210 each for my NAS.

Off topic but I didn’t realize you could save that much! Why are externals less expensive?

There's a bit on the r/DataHoarder subreddit about it[0]. They say it comes down to a shorter warranty, more demand and "Binning of performance parts".

[0] https://www.reddit.com/r/DataHoarder/wiki/hardware#wiki_shuc...

You can get amazing deals on storage at different points in the year. I have a Seagate 16TB external hard disk that I bought during Black Friday sales of 2020 for about $270, half of your quoted price. Currently, it is $450 new.

At the reduced cost, we'd only need 7 of those for 112TB of storage, and that's $1890 + taxes, close to $2000. Not a lot of money for a backup of humanity's knowledge.

I think there was some reddit data hoarder initiative that did that a few months ago.

There are 850 scihub torrents, each containing 100,000 scientific articles, to a total of 85M articles (77TB) http://libgen.rs/scimag/repository_torrent/

index for all the papers in these torrents (SQL db): https://libgen.rs/dbdumps/scimag.sql.gz

also see https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_...

But are they searchable, by users with little technical ability? Will they be updated with new papers if Sci-Hub shuts down?

Nope, but at least, they aren’t lost. They are just there, ready to be made available by the potential successor of Sci-Hub. That’s what matters.

It's not too hard to find the article's doi (programatically) for newer articles. Once you have the doi you have all the important article metadata like author, publish date, etc. I imagine this is all that's required to make the articles searchable.

However, this is certainly more challenging for older articles.

Reply on news.ycombinator.com