Hacker News Reader

Comment by krrrh | Hacker News Reader

If this is real and can be commercialized it could lead to some pretty devastating fallout for the entertainment industry. You've got to know that there are a few obsessive archivists out there with pretty comprehensive collections of films spread across rooms of RAID arrays. When they can start backing those collections up for friends onto a single disc, ie a single disc that has more films than you could possibly watch in your lifetime the entertainment industry is going to pine for the days when torrents eve the big problem.

The same goes for rogue librarians, or Google Books employees dumping entire libraries onto discs and leaking them out.

View on HN

> more films than you could possibly watch in your lifetime

I was curious if this statement was true...

So assuming, 4.4 GB her HD film & 2 hrs long each, you can fit roughly 232 films per TB or 232,000 films total (for 1000 TB), that's 464,000 hours of HD content or 53 years (!) back to back.

So, yes, that statement is true. Now the next logical question: Has the human race produced 464,000 hours of video content?

Let's assume there are around 100 cable channels showing content in HD video 24/7 (with no overlap, there are more but a lot of duplication). That's 2,400 hours of content a day, 876,000 hours a year or 101 years back-to-back worth of watchable content shown each year.

But realistically that will be at least 70% duplication (particularly looking at it over several years). So even if we just look at 24/7 news and shopping channels which always produce original content, you're still talking about easily 200,000+ hours/year.

http://www.imdb.com/stats

Seems to be rather likely.

https://www.youtube.com/yt/press/statistics.html claims "100 hours of video are uploaded to YouTube every minute"

So, that would be 4,640 minutes or less than a week. Lower quality, and probably with (near) duplicates, but I guess it is safe to say we have that amount of video.

Another way to look at it: I expect we produce more content in unique wedding videos every year (at an hour each, that takes half a million weddings or a million people marrying. At an average of one marriage per life and a life expectancy of 75 year, that takes a population of 75 million people)

> Has the human race produced 464,000 hours of video content?

Far more than that... YouTube alone claims "100 hours of video are uploaded to YouTube every minute":

https://www.youtube.com/yt/press/statistics.html

...which comes out to 144kh/day. In other words, one of these discs would be enough to store ~3 days worth of contributions to YouTube. So a box of these discs, let's say 1000 of them - which isn't all that big (e.g. http://www.amazon.co.uk/Neo-Aluminium-Storage-1000-sleeves/d... ), could hold an archive of every single video that has ever been uploaded to YouTube.

I am now reminded of this: http://www.youtube.com/watch?v=Y_UmWdcTrrc

Edit: got beaten to mentioning YouTube, but there's many other video sharing sites on the Internet, and of course there are probably countless hours of one of the types of video the Internet is well known for: porn.

The majority of that is junk anyways. But I suppose some people obsessed with archiving everything wouldn't filter anything. If I were to grab "everything" from YouTube I would at least limit it to subscriptions and only subscribe to channels that don't upload bogus junk.

Then the valuable commodity would become directories of worthwhile content. You would have people doing deep dives into the crap to find the locations of a few good pieces to share with their buddies or trade.

With the internet and Video On Demand that's already the case.

>Let's assume there are around 100 cable channels showing content in HD video 24/7 (with no overlap, there are more but a lot of duplication). That's 2,400 hours of content a day, 876,000 hours a year or 101 years back-to-back worth of watchable content shown each year.

I'm thinking that 100 channels provide exactly 100 years worth of content per year instead of 101. Unless you rewind the good parts :-p

And there are of course vastly more than a 100 cable channels. BSkyB alone broadcasts more than 600 channels for the UK market (more than a thousand if you include radio). In 2010 Ofcom passed 1000 licensed UK channels.

The US is supposedly the only other country with similarly high number of channels, but I don't think it's unreasonable to assume that there are more than 10,000 channels worldwide.

> more films than you could possibly watch in your lifetime

>> Let's assume there are around 100 cable channels showing content in HD video 24/7

Note that this is comparing HD content to films, though I still think you are fine in saying we easily have enough content. :)

> So assuming, 4.4 GB her HD film & 2 hrs long each

People bitch constantly, but a typical feature film encoded by YIFY at 720p will clock in at somewhere between 650-900 megs, leaning towards the lower end of that range.

Not that it matters, the rest of your math makes it look ridiculous. Without too many tradeoffs, you could certainly put every American (Hollywood and indie both) film on such a disc.

> Now the next logical question: Has the human race produced 464,000 hours of video content?

Somewhere Google has a gauge that shows how many hours of videos they store... but I can't find it.

> So even if we just look at 24/7 news and shopping channels which always produce original content, you're still talking about easily 200,000+ hours/year.

Agreed. But I don't think that they have archival policies in place where everything is kept permanently. Even the news channels may have been in the habit of dumping everything but stuff deemed important, well into the 1990s.

Books are small enough that this has pretty much already happened (search for "library genesis").

Thank you for that.

Everything I find related to that seems to have been shut down, are you getting better luck?

First result on uncensored search engines.

Some libgen sites may be blocking IP addresses from certain countries. If so, try another mirror, or use a proxy.

gen.lib.rus.ec and bookfi.org both work for me, some time ago I investigated the origin of those collections and apparently they began as a set of 30+ dvds that have been distributed presumably primarily in russia for quite some time.

Especially in math and physics they are every poor graduate students dream. Even large libraries have only one copy of some of those books stuffed away in some underground vault.

And engineering and CS, both undergrad and postgrad. Fantastic website for when the library was out of stock of that book published 40 years ago that there's only 3 copies of in the country...

Library genesis is the best thing that happened to research since Google (and a huge number of researchers use it, secretly). Being able to access any obscure technical book in the world in a few seconds is literally orders of magnitude more efficient than waiting a week for an interlibrary loan. I dread the day it inevitably gets shut down due to pressure from the copyright mafia. It's ridiculous that such a service isn't 1) legal, and 2) publicly funded (with due compensation to the authors).

Could you explain more exactly what "library genesis" is? Is it a collection of basically every book ever published or only modern things? Did someone scan everything, like Google books?

It's basically a huge collection of documents, mostly scientific with a FOSS codebase for serving. It comes with metadata. It is mirrored a lot thanks to bring completely open. It originated in Russia and has had quite the history. It is a great step into the direction how documents (especially scientific) should be available to anyone in bulk without restrictions.

Holy shit... Just tried some obscure stuff I'd been trying to find for ages. This is amazing

You are forgetting that the speed at which optical drives can be written is bound by the rotation speed. If you have a single laser of this type and mount it in a traditional drive as we know, it would take over a year to write a single disk :) Still lots to do :)

is anyone else really bothered that the article says CD but it's actually about DVD?

That seems overly nitpicky. It's about a new technique for beam lithography and achieving a smaller feature size than previously possible.

The "1000 TB" is just PopSci extrapolation at this point. If it does happen, it'll be neither a CD nor DVD.

It's actually about optical discs. But "CD" is the vernacular for optical discs. Kind of like "hard drive." Modern laptops don't have hard drives--they have solid-state drives, but people still refer to their storage as a "hard drive."

Really? Assuming that the rotation speed is the same for a normal disk and this denser format, wouldn't said laser be moving over 250,000 times as much data in the same time period?

At 1X speed, a DVD's data rate is about 11Mbit/s - so at 250,000 times the density, isn't it safe to assume a theoretical data rate of 2.75Tbit/s? Of course, pushing that much data at that rate is another problem...

If my assumptions are incorrect, please let me know - I'm just trying to understand.

52x speed has been the limit for a long time because the outside edge of the CD gets to speeds where it could damage itself. I guess it would need a cushion, like a nearly applied (car/bike) disc brake.

Okay - but I'm not talking about changing the rotational speed.

You're forgetting the square-cube law. Or rather, in this case the linear-square law. Think of each bit as a square, with side length 1/n. Oversimplifying, but the basic principle is there.

The total amount of data the laser goes over in a second is proportional to n - because it's related to how long the laser takes to get from one edge of the square the the other edge. But the total amount of information stored goes up as n^2 - it's related to the area of the square.

Or: to put it another way, the number of tracks also goes up when data density increases.

(This also happens with hard drives over time. Recent hard drives take a lot longer to read or write the entire drive than older ones. See http://tylermuth.wordpress.com/2011/11/02/a-little-hard-driv... for example.)

Hmmm. Maybe you could use a prism/mirror array (like in digital projectors) to use a single laser to write in multiple places at once? (Or rather, in sequence but moving faster than the disc rotates?)

I'm wondering if the entertainment industry is the reason for which Hyper-CDROM[0] is not commercially available. If so, something is very wrong with the way society incentivises development and commercialization of new technologies.

[0]- http://en.wikipedia.org/wiki/Hyper_CD-ROM

Looking forward to being able to carry the Internet Archive around in my backpack.

Will this really be a larger "problem" for the entertainment industry than online file sharing? Is there many people out there that would want such a disc that already doesn't have fast speed Internet and can download any movie they want within minutes?

As of a couple of years ago, the entire content of The Pirate Bay[0] was under 100MB[1].

Think about that for a moment - it's the index for one of the largest collections of publicly available content in the history of humanity, and it fits in my wallet 80 times over[2].

[0] Not the P2P content that people download - just the main site itself, containing all of the magnet links, etc. TPB's 'content' is literally just an index (it's not even a tracker).

[1] I don't want to link to TPB from HN, but just search for "pirate bay archive". There's a large archive that contains all of the site data (db backups), but the magnet links are all you really need content-wise.

[2] http://www.amazon.com/Compact-Credit-Style-Flash-Drive/dp/B0...

Why would you want that? The magnet links alone are rather useless. The whole site can be crawled within 2-3 days easily. There are ~1.2 million torrents indexed, lets just assume that a single magnet-link plus some metadata is 0,5-1kB... so that would be an total of ~1GB data.

> Why would you want that? The magnet links alone are rather useless. The whole site can be crawled within 2-3 days easily.

But the entire site archive is already available as a torrent indexed on TPB - why bother crawling? That's what I'm referring to - search "pirate bay archive" on TPB and you'll find it.

The magnet links are not useless, because they are able to query the rest of the metadata from a tracker and/or DHT.

I'm thinking about this from the perspective of 'minimal amount of information needed in order to reproduce TPB', not 'how much information does TPB index (which is obviously much larger).

(Remember that TPB no longer even operates its own tracker, so if TPB itself were taken offline, the magnet links would still work fine).

> the entire content of The Pirate Bay > Not the P2P content that people download

I think you're confusing different definitions of 'content' here. Also, there are many more sites that have publicly available indexes of other links much greater than the pirate bay, google.com for example.

Furthermore, 8GB is relatively small for a flash drive nowadays, e.g. here is an 128GB micro sd card: http://www.amazon.com/dp/B00M562LF4

Sandisk just released a 512GB SD card a week or two ago:

http://www.sandisk.com/products/memory-cards/sd/extremepro-s...

Reply on news.ycombinator.com