Hacker News Reader

IPFS provides the same privacy guarantees as bittorrent (i.e. none).

Use a VPN.

sixtyfourbitson ZLibrary domains have been seized by the United States Postal Inspection Service

https://gizmodo.com/amazon-secretly-removes-1984-from-the-ki...

sixtyfourbitson ZLibrary domains have been seized by the United States Postal Inspection Service

Fortunately someone had the foresight to make a backup:

https://news.ycombinator.com/item?id=32972923

Z-library were in it for commercial gain (you could access a certain number of books but to get more you had to pay for a subscription). They started out as a fork of library genesis, whose mission has always been strictly non-commercial and about providing free access to everyone without limits.

Hopefully this will encourage more people to go back to the original libgen. I suspect Z-library's popularity was because of a better interface and larger collection, but I think lots of people didn't realize libgen offers all its books/papers for free without limits.

sixtyfourbitson Design and evaluation of IPFS: a storage layer for the decentralized web

It actually doesn't work so well, unfortunately.

Picture this: 3 million books, at least one CID each (in practice it's often multiple, since the libgen collection uses a chunk size of 256kb). Section 3.1 of the paper talks about content publication - for each CID, a provider record is published on up to 20 different peers. Because the CIDs are derived from a high-quality hash function, they are evenly distributed. So this means that a node with a sufficient number of items ends up connecting to every single node on the network. For 3 million CIDs * 20 publication records, this means sending out 60 million publication records, every 12 hours, i.e. an average of 1388 publication records per second (assuming one CID per file, which is conservative). This is just to announce to the network "hi... just wanted to let you know I still have the same content I did yesterday". And every full replica of libgen is doing this.

Another major flaw derives from the way bitswap works as discussed in Section 3.2, which states "before entering the DHT lookup, the requesting peer asks all peers it is already connected to for the desired CID". I'm not sure if that actually means all the machines the node has any type of connection to, or only those connections over which it is using bitswap. Even so, asking every peer you're connected to, even if it's only the subset for which you have a bitswap connection established, is inefficient.

Compare this to bittorrent:

- First there is a much coarser level of granularity; in the libgen case the torrents contain 1,000 books each, so there's much fewer announcements to the bittorrent DHT and trackers. Of course the tradeoff with this is that you can't look up the identifier of an individual book. Instead if you know the magnet link (which can be constructed from the torrent hash) and the name of the file within that torrent.

- Secondly, a node hosting a large number of torrents (and a large number of active connections) will only send out want lists to the peers that it knows are also hosting that torrent. Peers also exchange have lists, and I think one or both can be represented as a bitfield for efficiency (rather than a list of CIDs/hashes). With bitswap you can end up asking every connected peer, just in case they have it.

On a practical note, hosting the torrents is quite practical with adequate hardware, but even with fairly powerful machines IPFS (at least in it's current main implementation, go-ipfs aka Kubo) really struggles and can bring a machine to its knees, even when hosting only a portion of the full collection. In terms of scalability, bittorrent and IPFS are in completely different leagues. Scalability is the main reason the archive of papers from sci-hub (over 80 million) isn't available via IPFS yet, because it's just not going to be able to handle that at all in its current state.

Having said all this, I should state my knowledge of the protocol and go-ipfs is incomplete, as I've only used it but not done any development work on it or dived deeply into the code. I'm happy to be corrected if I've misunderstood anything mentioned above. Also, bittorrent has more than 20 years of implementation experience and i'm sure with further work IPFS can be made to scale better. I don't have the answers as to how to achieve the granularity you get with IPFS vs bittorrent (which is a major point of difference, and something that sets IPFS apart in a significant way). But it's something that definitely has to be fixed to be truly capable of achieving it's stated goals.

sixtyfourbitson Pirate Library Mirror: Preserving 7TB of books (that are not in Libgen)

IPFS is a protocol like bittorrent. The only way to make content inaccessible is to go after the people/computers that are sharing that content.

sixtyfourbitson Pirate Library Mirror: Preserving 7TB of books (that are not in Libgen)

All of library genesis is already available on IPFS (see https://freeread.org/ and https://libgen.fun/dweb.html). Hopefully someone will import this collection into libgen and then these books can be on IPFS too.

Torrents are simpler and more efficient for distribution, but IPFS is better for accessing individual files.

sixtyfourbitson ACM Opens First 50 Years Backfile

This is nice and all, but there's no valid justification for any of the material in the digital library to be locked behind a paywall in the first place. Most of the research published by ACM was paid for by taxpayers, and authors have to either sign over copyright or grant exclusive publishing rights. To be fair, ACM's fees are far more reasonable than the big publishing companies, and there are open access options available (at a cost to authors).

They opened the whole thing up for unlimited access for a brief period during the pandemic, but decided to walk that back after just three months. If you know where to look, there's a 500gb torrent floating round with the 480k+ papers that were accessible as of June 2020.

It's sad that in this day and age, particularly with the widespread acceptance of open source, most academic publications are still behind a paywall. We shouldn't even be having discussions about "open access"; the "open" part should just be implicit.

sixtyfourbitson Sci Hub Injector

The reason why they've stopped temporarily is due to an ongoing court case in India initiated by Elsevier. I'm not sure this is the best article on the case but basically sci-hub agreed not to post any new articles for a period of time (which has been extended) while the case is ongoing:

https://www.hindustantimes.com/india-news/no-new-articles-on...

They did release a bulk issue of 2.7 million articles a few months ago (as part of the torrent collection available from libgen), but nothing new since then.

sixtyfourbitson Indian academics throw weight behind Sci-Hub and LibGen in landmark case

The academic world is too entrenched in the established order, and while many inside it would agree with you, the incentive structures in place often work against individual action like this because the first priority is usually publishing in high impact-factor journals, open access second.

Despite some laudable efforts, academia as a collective has proven themselves incapable of fighting the publishing cartels. As a result, it's turned out that the only viable pathway to true open access is large-scale civil disobedience on the part of sci-hub and its users.

This is a very interesting case to watch though as it could finally provide a legal basis on which sci-hub can operate, even if only in India.

sixtyfourbitson Z Library

Correct. Library genesis itself (see https://libgen.rs/ or https://libgen.fun/) is run by volunteers on a strictly non-commercial basis and does not charge for access or accounts.

sixtyfourbitson Show HN: Fully-searchable Library Genesis on IPFS

The gateways are indeed centralized (though there are several different public gateways). But anyone who installs IPFS on their computer can access the content directly. Also some browsers support IPFS natively, e.g. https://brave.com/ipfs-support/

From what I understand, the notion of sci-hub/libgen "not being welcome" was only about discussion on the official forums. See https://discuss.ipfs.io/t/mirror-of-sci-hub-in-ipfs/1613 and https://news.ycombinator.com/item?id=25209246. But IPFS is a protocol just like bittorrent or HTTP, and the software is open source; it doesn't and can't enforce copyright restrictions.

sixtyfourbitson Show HN: Fully-searchable Library Genesis on IPFS

https://libgen-crypto.ipns.dweb.link/source.tar.gz

It makes use of the sql.js-httpvfs library, previously discussed on HN here: https://news.ycombinator.com/item?id=27016630

sixtyfourbitson Show HN: Fully-searchable Library Genesis on IPFS

Next up is the scimag collection (also maintained by library genesis), which is a backup of all articles on sci-hub.

The torrents are already widely replicated, but whether or not IPFS is going to be able to scale to the level required for 85M articles is still an unknown.

sixtyfourbitson Today Sci-Hub is 10 years old. I'll publish 2M new articles to celebrate

The only one I'm aware of currently is https://github.com/sci-hub-p2p/sci-hub-p2p. Library genesis also hosts database dumps at https://libgen.rs/dbdumps/.

There's really a need though for more developers to get involved with building tools for more easily searching and working with the collection, ideally with a nice UI and integration with things like crossref. This is a massively valuable data set and it would be great to see what people can come up with. Lots of awesome potential for data mining too.

If it weren't for the legal issues (publishers using copyright law to restrict access to literature they got for free since they never pay authors for their work), there's no shortage of projects that could utilize this data and be enormously beneficial for the scientific community and humanity in general. Unfortunately such work can only be done in the shadows right now, which greatly limits the number of people/institutions likely to do so.

sixtyfourbitson Today Sci-Hub is 10 years old. I'll publish 2M new articles to celebrate

Yep! See https://github.com/sci-hub-p2p/artifacts/releases/tag/0

This project is in it's early stages and the documentation has quite some way to go, but the index that's part of the release contains all the necessary information. This tool also contains the code necessary to produce the index files if you have a local copy of the zips.

Each torrent contains 100,000 files, comprised of 100 zip files with 1,000 PDFs each. They are named by DOI. There's a database dump at (http://libgen.rs/dbdumps/) (scimag.sql.gz) which has the id -> DOI mapping and other information. The specific torrent and zip file can be determined based on the id; torrent = id/100000 and zip = id/1000.

sixtyfourbitson Today Sci-Hub is 10 years old. I'll publish 2M new articles to celebrate

It depends on the circumstances and the publisher. In many cases publishers permit authors to host the accepted version of the paper (but not the final version that includes revisions based on reviewer feedback) on their personal/institutional website and to email copies of the paper on an individual basis to people who request them.

For example, see https://www.elsevier.com/about/policies/copyright (under "Author Rights") for what Elsevier permits you to do with your own work.

On the other hand, publishers have sometimes filed lawsuits against sites where authors share their papers, e.g. ResearchGate: https://www.nature.com/articles/d41586-018-06945-6

Elsevier have also sent takedown notices to universities where academics have made the final version available on their institutional websites:

https://www.washingtonpost.com/news/the-switch/wp/2013/12/19...

https://osc.universityofcalifornia.edu/2013/12/elsevier-take...

https://news.harvard.edu/gazette/story/newsplus/elsevier-tak...

https://blogs.library.duke.edu/scholcomm/2014/01/28/setting-...

sixtyfourbitson Today Sci-Hub is 10 years old. I'll publish 2M new articles to celebrate

It's all PDF files, which have their own compression, so it's unlikely there would be substantial gain from additional compression. Each torrent has 100 zip files, and each zip file has 1000 PDFs, but the files are stored uncompressed within the zips (i.e. using the STORE method).

sixtyfourbitson Today Sci-Hub is 10 years old. I'll publish 2M new articles to celebrate

Probably; use a VPN.

Yes they're copyrighted - albeit not by the authors who actually wrote them, but by the publishers who require copyright assignment for the privilege of having your work hosted on their website.

sixtyfourbitson Today Sci-Hub is 10 years old. I'll publish 2M new articles to celebrate

Torrent seeding effort: https://www.reddit.com/r/DataHoarder/comments/nc27fv/rescue_...

All papers on sci-hub are available as torrents from library genesis. The full collection contains 85 million articles (before this announcement), and is about 80TB. If anything ever happens to sci-hub or library genesis, there's enough people out there with backups that a replacement can be set up fairly quickly, albeit without the proxy functionality to obtain new papers.

However, the more the merrier, so if you've got some spare hardware and bandwidth to share, I'd encourage you to contribute to the seeding effort if you're able. At current market prices of ~$30/TB, it costs ~$2400 to have a copy of the full collection sitting on your desk.

sixtyfourbitson Police warn students to avoid Sci-Hub

There's been an ongoing effort for a bit over a year now to make sure the torrents from sci-hub's archive are sufficiently well seeded (specifically, the scimag collection from library genesis, available at libgen.rs). It's about 80TB total, so with current hard drive prices you can maintain your own local copy of all 85 million articles for under $2k. Unfortunately I'm not aware of any tools to usefully browse the collection like what you suggested but it would be great to see something like that.

The main library genesis book collection is already available via IPFS (see freeread.org), so I wouldn't be surprised to see all the sci-hub articles available similarly at some point.

sixtyfourbitson The MIT Press launches Direct to Open

The ACM did this with their digital library to support people during working/studying remotely during COVID but seemingly had a change of heart and put the paywall back up a short while after receiving all the positive publicity despite the pandemic still interfering with education and work habits.

Fortunately someone managed to scrape the entire thing and there's a 500Gb torrent out there with the full content of all the journals and conference proceedings.

sixtyfourbitson Publishers worry as ebooks fly off libraries' virtual shelves

Wait till they find out about library genesis...

sixtyfourbitson Blackballed by PayPal, Sci-Hub switches to Bitcoin

http://gen.lib.rus.ec/scimag/repository_torrent/

http://gen.lib.rus.ec/dbdumps/

It's something like 75TB last time I checked, over 82 million articles in total. Each torrent contains 100,000 articles, named by DOI.

If sci-hub ever gets taken offline, there are enough full backups out there that it can be re-instated fairly quickly (at least the archive of existing articles; the credentials necessary to obtain new ones are not included in these torrents or database dumps).

sixtyfourbitson FreeDB Is Shutting Down

I made a torrent:

  magnet:?xt=urn:btih:96208ba12ca7ffdac5b5adccf1a41e484de94f4a&dn=freedb&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=http%3A%2F%2Ftracker.torrentyorg.pl%2Fannounce&tr=http%3A%2F%2Fexplodie.org%3A6969%2Fannounce

This includes all the files from your mirror except for the older database backups and the "latest" directory (which is just the extracted version of cddbd-1.5.2.tar.gz).