Pirate Library Mirror: Preserving 7TB of books (that are not in Libgen)

A web server can run just fine on e.g. Raspberry Pi Zero W ($10), exposing any such content to any smartphone etc able to connect to it via WiFi (Kiwix sells preconfigured SD cards for their content, even). So, assuming that most people already have a phone or a laptop or even something like a Kindle, the only non-negligible cost here is storage. And a 1 Tb SD card can be had for under $150 right now.

So if anything, I think OP is overly conservative, given that their estimate was "$200 per person". Unless that counts the devices used to consume the content, and not just storage and the server.

From the perspective of how much you individually could read in a lifetime: that's about 4,000 weeks (to roughly age 80-ish, and presuming you're not already reading at birth). Multiply that by the number of books you plan on reading a week. You'll probably read fewer than 4,000 books over your entire life. Many people read few if any books after graduating secondary school, and even a highly-motivated reader might be challenged to crack, say, 40,000 (ten per week for life).

The US Library of Congress has the world's largest book collection at about 40 million distinct titles.[1] There are another 130 million total catalogued items, including photographs, films, audio recordings, maps, pamphlets, and other items. Not all are textual. I'll stick to books.

At 4,000 per lifetime, you'd need 10,000 people simply to read all of the LoC's collection at a reasonable pace.

(By comparison, at the turn of the 20th century, the LoC's annual report to Congress noted that its cataloguing department could handle about 3,000 books per cataloguer per year, or a pace of about 60/week or 15/day. Over a 40-year career, a cataloguer might handle, if not read completely, 120,000 books.)

As a rough rule of thumb, a digitised ebook in PDF format runs about 5 MB.

Every single book in the Library of Congress in digital format would occupy about 200 TB of storage.

Current disk prices are running about $5 -- $20 / TB.

200 TB in raw disk would set you back $1,000 -- $4,000. Figure 2-4x multiplier for a disk storage system all told, and it's still roughly $4k -- $16k to have as local storage every last book in the US Library of Congress. That's well within scope of a moderately wealthy US household budget, and would be reasonable to consider for a small-town city library.

That's the technical storage cost, obviously not the rights or aquisition costs for the materials.

The Library of Congress's budget is about $800 million/yr.

Those 4,000 books you might read in a lifetime? They'd fit on about 20 GB worth of disk. If you're a 10x reader, 200 GB, and a truly dedicated 100x reader, about 2 TB. Those are well within the range of present desktop / laptop disk allocations, and represent a few tens of dollars of storage expense.

The Internet Archive computes $2/GB for storage in perpetuity. That's roughly 400 books worth of storage.

But a small household NAS and a very modest server platform (most routers can effectively operate as media servers) could rival a mid-sized city library for about $100 or less in actual hardware outlay. Those prices are falling by half about every 18 to 36 months, as they have been for decades.

Books are not all published content, and all published content is not all human knowledge. But standard published books are a good proxy for total cultural knowledge, and in all likelihood, then some.

________________________________

Notes:

1. See: https://www.loc.gov/about/general-information/ That includes about 25 million books in the main collection, and 15 million in "nonclassified print collections, including books in large type and raised characters, incunabula (books printed before 1501), monographs and serials, music, bound newspapers, pamphlets, technical reports and other printed material". I suspect you could roughly halve my estimates above based on 25 million vs. 40 million volumes, as many of the nonclassified works may duplicate the main collection.

The idea that we do not do a world library of free digital copies of every book ever written really highlights the problem with the thinking your comment has demonstrated. The idea that individual pursuits must make a profit to be justified leads to us doing terrible things like: not making a free world library of every book.

Though in this particular case this also demonstrates a major problem with intellectual property concepts. Actually hosting the library isn't very expensive. But we have made doing so illegal. Of course, authors deserve to live a decent life just like everyone else. We currently do that by restricting all access to duplications of information they have produced so that they can charge a fee for access, and that fee provides for their survival.

But we suffer an incalculable loss by making all this information restricted. In my view we would be MUCH better off as a society with respect to creativity, innovation, and other popular metrics for progress, if we actually made sure as a society that every person's survival was provided for with no need for them to pay for it. Then authors wouldn't need to get paid, engineers could do what they love to do and post all their work as open source, and we could have a free library for everyone. This extreme openness would in my mind lead to more rapid innovation, and markets would still function as first movers would maintain an advantage for new product releases, though they would have to keep moving as anything they've done that is worthwhile would be copied. But since no one's livelihood would be at stake, this is not a real issue.

This can all be done in a voluntary, libertarian society as long as we have community ownership of the means of production, and promote these ideals of community support in this society. And I think we would be way better off. Doing this would allow us to offer every book ever recorded for free to every person on Earth. A big change, but one with obviously a very big benefit to humanity.

One note though: people who want to own a lot for themselves really mess this up. So people would need to dissuade those people from acting that way. My preferred method of doing so is by starving them of workers and customers, though when it comes to control of land matters get more serious.

The point you’re trying to make (I think) is that information isn’t particularly useful on its own. True. It has to be relevant. With good, useful information you still need community support (whether that’s your household or your country), funding, time, etc. But that’s going to be the case whether you’re using a book or learning from a forum, or a video. Internet videos are worthless if you have no internet. Tribal knowledge is useless if no one can get the information from the previous generation. The medium is not the problem, and suggesting that relevancy is a fault of the medium is a misrepresentation.

By copyright expiry, public domain in the US begins in 1927. Later works may be in the public domain, but all works published prior to 1927 are in the public domain in the United States.

(This may not be the case in other countries.)

There were not many published books prior to the invention of the printing press, and many of those didn't survive. The total number of books (not individual titles but actualy bound volumes entirely) in Western Europe as of 1400 may have been as few as 50,000.

By 1800, about 1 million titles had been printed.

Over the course of the 18th century, presses became vastly faster, as they evolved from hand-operated wooden screw-press to iron-frames to steam and electric-powered rotary and ultimately web presses. Paper became much cheaper (and less durable --- a factor commented on at length in the Librarian of Congress's annual reports to Congress in the late 19th century). Literacy exploded from ~25% to 95%+ over the 19th century (and probably accounted for numerous revolutions and political upheavals).

Through much of the 20th century, certainly by 1950, US publishers were issuing about 300,000 new titles per year, a rate which state remarkably constant through the early 21st century. By the aughts, "nontraditional" self-publishing (a/k/a "vanity press") was nearing or exceeding 1 million titles per year more than had been published through all time to 1800.

Reports that all recorded data was doubling every few years date to at least the 1960s. That would mean that in any two year period ... half of all recorded information was less than two years old.

The catch is that not all recorded data is published. So I'm not sure what the time-distribution of all publishing looks like. But I'm pretty confident it's skewed far more recently than 1927. And would thus tend to be copyrighted rather than uncopyrighted.

If you want to measure works by significance, you might make a different argument --- there are many great works of literature, philosophy, history, and religion which were first published before 1927. But ranking and tabulating these is more challenging than a simple enumeration.

http://pilimi.org/faq.html

This thinking is an artifact of an economic system so dependent on scarcity for its motivation that it is now generating most of the scarcity in the world.

We now have the technology for creative implementations of "From each according to its ability, to each according to its needs". Just keep track of how much each thing is used, and reward creators from a corporate-tax-funded pool. Every for-profit entity contributes proportionally to its profit, and can use any idea for free.

Note that if you don't defend it in court, our justice system thinks it is less valueable to protect. Which in itself is kind of ridiculous.

Also, right to commercialization has nothing to do with intellectual property.

Of course I like to be credited for my work — but some fan adding my work to a pirate page would not be a concern but rather a bit flattering. What would anger me would be someone claiming credit for themselves, some rich company taking the material without paying me, things of that sort.

No one said anything about disallowing you from doing anything.

>my first question would be, other than for the love of doing it, why sink so much effort into a thing only to get nothing back.

Is it wise to sink your time into something you don't really like doing?

I believe you are mis-phrasing the question. What you're actually asking is:

> Should the state criminalize and punish people who make copies of my work, to facilitate my commercial activity with it?

And our answer is "No".

You can go ahead and engage in whatever commercial activity you like, based on open access to your work.

ofc not

but since noone can ever prove that his was the first incarnation of an idea, nobody can be criminalized for also doing things in a certain way.

the concept of protecting invention for some time to facilitate reward is not without merit, but the implementation of IP law and practise has gone so far astray that it's overdue to rethink the whole thing.

https://news.ycombinator.com/item?id=23073126

With grammar like that, it's probably just as well.

Writing is hard work. Writing books and getting them technically correct is expensive. This is very short-sighted.

There is no way to protect video, audio or text from being copied. DRM just prevents low effort consumer copying.

You may see lower revenues, but how much cost is currently poured into resolving licensing / investing in DRM / etc, all for works to be pirated anyway?

God no. Right now, we'd be getting remakes from every piece of pop-culture that was semi-popular in the 80s-2002 time frame. Not just movies, but TV series, books, theatre, musicals, ...

Sure, copyright should be shortened, but I don't begrudge (eg.) a one hit winner making money off their hit decades layer. Life of artist is reasonable, I think. They take a gamble on a profession with risky pay-out; if it works out at least once for them, let them reap the benefits.

If a regular citizen can get his hands on tools/materials to make things like that, you are screwed sooner or later.

It's not very convenient for archival, whereas the system I'm talking about would be (in my opinion, anyway).

this already exists with addons for Kodi

Only those entities would be able to push data to you, nobody else, so if you trust the providers you specified, you should be good.

The same was true of the original GOOG 411, which provided a free service, but was really put in place to train up their voice recognition projects.

This is a long running strategy of Google, and it's a shrewd one. The main thing is not to mistake it for a public good. It is an act of privatization.

I don't search for websites based on their titles...

It's a figure of speech, not a comment on what they do.

That being said, 10 downloads/day feels a bit restrictive to me. I'd get if it was 100, or 50, heck, maybe even 20. I mean, I don't appreciate that it's not mirrorable in the first place, but maybe they cannot afford it, I don't know... But 10 feels less than somebody researching a new topic might need to access in a day, even if he won't read them all immediately.

…That being said as well, it has some really nice UI. I wish somebody did it for Libgen.

Well they are providing a great public service and their system takes money to maintain. It's just a small fee.

I realize that neither source will satisfy many of the people on HN, simply because there is a need for current technical books.

What I also found was that many of the images in the epubs were already unuseable and nothing like their counter parts in phsyical books.

Kiwix .ZIM file format is a good example. The entire Gutenberg Library is a single ~65 Gb file, and you can read any book from it without unpacking anything.

Since transient, ethereal meme culture is also basically emergent culture now, it's difficult not to also foresee a greater cultural divide in such a case. This is saying nothing of live data tools as well, even weather data...

I wonder if there's a kind of memetic effect happening online, where people who lack confidence in their English spelling ability see somebody make this mistake and somehow think that it's correct, so they switch how they write it.

I notice it more and more too.