Hacker News Reader

Comment by th0br0 | Hacker News Reader

They backup to Library Genesis as far as I understand it, Library Genesis is available via Torrents & Usenet - http://gen.lib.rus.ec/repository_torrent/

View on HN

The IPFS network is ideal for this kind of situation.

I should say I'm bias as I run ipfsstore.it but to host all 26gb there would not be expensive.

ipfs overhad is still way to big. also 26TB not 25gb.

Overhead in what aspect?

Adding files to IPFS creates a copy of the file in their blockstore implementation. The new filestore should have solved this problem but I haven't used it yet.

besides the mentioned duplication of all files, the dht network chatter to announce a large repository by default produces many TB of traffic a month.

In which way is IPFS ideal?

Wouldn't IPFS make it easier to access individual files compared to a Torrent file?

26TB is a massive amount of data.

Having downloaded (very) large dumps in the past with torrents, I've found it always a bit annoying especially when attempting to access a subset, torrent client interfaces are less than ideal here. Not that someone can't build a good document browser on top of torrents - ala those movie torrent streaming apps.

I'm not familiar with how IPFS works in detail but being able to access it via a file system sounds much better UX wise. Hopefully it can support that scale.

But to your point the primary end-goal of "censorship-free" persistent access is largely the same AFAIK.

You don't have to download an entire torrent to access a single file. Most torrent client can prioritize files so they get downloaded first, or avoid downloading everything in favor of just one or two. That is, unless the torrent is just a single zip or something.

and if everyone does that, noone will offer the particular file except hosts, so the availabilty can be spotty - and the torrent interfaces have no way to tell upfront.

Yes I mentioned that in my comment but it's not an ideal (or even practical) interface when you have 26TB of files. You'd have to build a special torrent interface to deal with that in a useful way. Existing torrent clients are far from capable of handling this meaningfully.

So for your service 26TB * 0.044 USD/Gigabyte = $1144/month to host all the Scihub data

Quite a big fee :p

That seems rediculously cheap and easy to come up with funds for. Storing the entire progress of science for $1k/mo? That's a good deal.

True, a low grade ICO for SciHub, backing an multi-country IPFS store with a decent web interface on top could raise that money easily. You could create a mini forum/social network around it for commenting on papers. Buying a coin to make an account or freemium buying a 'VIP' accounts ala Dribbble...maybe even a DocumentCloud style annotation system, so it has some self-sustaining business model other than raising initial capital.

Although I don't know much about creating 'DApps' or how Ethereum or IPFS works or what SciHub contains beyond a primitive grasp [1], just sayin :p

[1] I watched a video of a talk "Distributed Apps with IPFS (Juan Benet) - Full Stack Fest 2016" https://www.youtube.com/watch?v=jONZtXMu03w

Why does everything have to be an ICO? What's the point of a token here? Just give the appropriate organization some charitable funds.

That's one way to look at it.

Another is 13 2TB drives at $70 a pop. May run into some network congestion sharing however :)

> The IPFS network is ideal for this kind of situation.

I like ipfs, but in this situation specifically it's got problems. It's not private and it's not secret. It means that, by hosting a mirror of the data you're announcing: I'm publicly offering these documents, likely illegally as far as copyright is involved.

(And without a country/region limitation, which may be another issue)

> It's not private and it's not secret

Neither is BitTorrent.

How about Freenet?

the db dump is public too, from the tip of my tongue libgen forums quoted something like 100TB altogether. (this includes more than scihub cache)

Yes, if you have a seedbox or something please host a few random parts! I have three or four hosted pieces and there aren't too many peers.

Is there any way to tell which torrents have fewer seeds, without having to add them all to my client?

I would like to know as well.

See my other post: https://news.ycombinator.com/item?id=14870697

That's brilliant; nice work! Will try to seed some soon.

I wasn't able to figure out a way so I just picked a few at random. Is there an easy way to roll up like a heroku app or something that would check the health of only those torrents? That could be really useful to the libgen guys.

I couldn't find anything, so I ended up making it myself using TorrentSniff.

http://genesis.andreparames.com/

Most of the trackers seem to be down, so it's just getting data from tracker2.wasabii.com, and even that is pretty slow, so it'll take a while to fetch for all 1693 torrents.

I wrote a script that pings the tracker and keeps regenerating a static HTML file, so it'll keep updating itself, though not very fast.

Just saw this, thanks! Super cool.

Reply on news.ycombinator.com