Storing 25 petabytes of Megaupload data costs us $9,000 a day
arstechnica.com
arstechnica.com
With a 25 petabyte discovery, I'm not surprised that everyone's scratching their heads on what to do next. This isn't just an MPAA/Megaupload problem. Even a smaller dataset like a 10-20TB discovery has numerous problems. Hosting/indexing/classifying/reviewing millions of documents is an open issue for the legal field. What do you do when there are multiple parties who all need to see "everything"? If everyone does their own thing how do you reference materials in a consistent manner across the interested parties? If you all agree to host the data in a neutral place who pays for it? What if the technology of that host benefits one party at the expense of another?
For years the legal field has had a "print it all out and have a team of paralegals go over it" viewpoint. Clients don't pay for computers, but they do pay for paralegal hours. Only recently has that become untenable. Discovery sizes are growing exponentially per year. It's common to have a new discovery set come in larger that every previous set combined, and the legal industry doesn't really know what to do about it.
I'm sure most of these drives were arrayed in such a fashion where they're unreadable unless in the proper equipment. It's not like you can just buy a pile of off-the-shelf external drives and start copying, either, as the contents might be unreadable unless the proper software is installed and configured correctly.
I think its clear that this would be a problem in an industry where the most used tool is a bookshelf of various common interpretations of law. It just doesn't scale.
The idea that law firms don't do IT is born of ignorance. Even the idea that the most used tool is a bookshelf of common interpretations of law is wrong, unsurprisingly all that stuff is online these days. Why on earth would anyone with two brain cells to rub together search for case histories on paper when there's Lexus Nexus which will do it in fractions of a second and is updated on a regular basis?
Yes these people are pretty conservative they're not stupid and they have money (a lot of money). Large law firms will typically have IT departments of hundreds and there's a significant market for companies specialising in supporting the legal sector.
But this is a tough problem because not only are there huge amounts of data involved but because people are actively trying to hinder your search. The law says that you have to to disclose everything relevant but it's pretty much common practice to also disclose a bunch of things that might be and a shed load of stuff you know full well isn't. This is a double win for the defending firm - not only can no-one accuse you of not providing everything they might want to see, but you get to do so in a way that makes very very difficult for them to find the stuff that matters.
Think of it this way - the problem is basically the same one Google have (already a pretty tough problem) only instead of the people providing the pages being keen that their information be discovered, they're actively trying to hide it.
I would guess that MegaUpload's lawyers will make the claim that the data on those servers is critical to their defense and must be maintained. That is probably an accurate claim, DotCom will want to present evidence of compliance with DMCA notices, counter the claim that a "majority" of the content was under copyright, etc. Best case for DotCom would probably be that his lawyers argue for retaining the data and the judge lets Carpathia destroy it anyway. That would give DotCom reasonable grounds for appeal.
The lawyers basically try to make things more and more onerous in order to encourage a settlement. It's amusing, as long as you aren't accountable for the data!
1. Purchase uber-expensive mansion and set up an astronomic mortgage payment/month.
2. Get said domicile seized by US gov't.
3. Stall legal proceedings until Marshals have paid off the entire house.
4. ???
5. Profit!To go with your analogy, sure building owners aren't allowed to rent the space back out, but they most certainly not asked to pay for usual water, gas, or electricity bills (because they aren't/shouldn't be being used).
Third parties get f'd in criminal cases all the time. One of the asset seizure stories the WSJ recently covered involved a guy whose cash happened to be in transit with an armored car company. That was, until the companies assets were seized by the feds along with his cash.
Lucky for Kim Dotcom, New Zealand let him get a hold of some of his cash for living expenses. I'm not sure he'd be so lucky in the US. Carpathia probably should have had an insurance policy for this.
Some people have mentioned insurance. Would any policy cover you for a customer being investigated for possibly criminal activity?
The $3.2MM per year is actually sounding a lot more reasonable to keep it up and running.
- if you buy HDDs in these quantities you will get a big discount (50%+)
- they most likely get the HDDs back after the case is finished, but not the electricity bill.
- you have to factor in opportunity costs from not being able to rent out that rackspace.
- I don't even understand how this can be proper evidence if it just hangs around at a private company.
* It's the chain of custody that is important not who keeps it. Most LEOs outsource to various contractors for all types of investigations.
That said, IANAL but I think the interesting thing here is that just making copies of the hard drives is likely not enough from a evidence standpoint. With such a data set there are likely layers of data abstraction software that would be need to be replicated to make any sense of it.
It sounds to me like they are looking for a way to keep their income flowing with servers that they will have a hard time renting out in the near future. In other words, they just got hit with a huge excess of server capacity that they are going to have a hard time renting out short term (and probably longer term as well).
Back-of-the-envelope calculation: Just did a search for "1080" on some unnamed site and it appears a bluray rip of a movie encodes to roughly 10GB. So that would be 2.5 million movies in 1080p quality. I don't think we've made that many, have we? Especially if you consider that movies that came out before the "high-definition era" are encoded to about a 10th of that size (700MB-2GB roughly, afaik).
Maybe I'm missing something obvious.
Not counting TV series for instance (are they also intellectual property represented by the MPAA? I'm not in the USA so I never really dug into that).
Movies duplicated in different quality formats are usually a 10th or less of the size of a 1080p Bluray rip as well, as an upper limit I could add a factor of x1.5 for that.
But then, the "long tail" of movie rips are 700-800MB and do not have duplicates.
Unless ... is the MPAA also representing porn? Because then all bets are off and I can easily accept that this 25 petabyte consists mostly of MPAA protected intellectual properties.
But otherwise, what percentage of these 25 petabytes would you estimate actually represents illegitimate data owned/represented by the MPAA? 2% ? 10% ?
Is that fair to the owners of the other 90% of the data? Even if it's probably mostly porn? (I'm fairly sure most of the data has to be porn)
I'm just wondering. Also because it's interesting to speculate what could be in these 25 petabytes. If you have a better guess I'd love to hear it :)
I've got 4TB of storage on my media server, last time I checked at ~60% used.At _best_, possibly 0.5% of that is stuff that I've personally created and have copyright over. Hell, all the email I've sent _or recieved_ that wasnt spam filtered since mid 1995 only comes to a few hundred meg - including attachments! Smething less than half a TB of it is music which I have some kind of right to have as digital files (some of it purchased as files, some of it ripped from cd and vinyl - which is somewhat less clear legally with respect to my rights to have a "copy" as a file on my hard drives). Realistically, outside of academia and industry (who presumably aren't significant users of MegaUpload) chances are so close to 100% that any 1GB+ file is copyright encumbered in a way that gives the MPAA an interest that it doesn't matter. The nearest I cold come to justifying the rest is that some of it it "time shifted" TV (from the PS3 TV tuner/PVR), some of it is DVD backups for discs I own, some migh euphemistically be referred to as "timeshifted DVD rentals or loans", but a _lot_ is copyrited content found on "channel BitTorrent" or downloaded from YouTube. If the copyright police confiscated _my_ hard disks and catalogued them in front of a judge, I'd have a very hard time looking him in the eye and saying "I didn't think I was doing anything wrong!"
I fear that line of argument bodes badly for dotcom…
!!!
I drown in approximately 2GB/year and that's after deleting some stuff and many attachments. Do you make a lot of phone calls or hand-write letters? Or not run a business maybe?
There's also people that make a lot more video data than you apparently do, same for sound recordings. Do you believe that the majority of recorded video data in the world is MPAA's? That would go against everything we know about the "long tail". You do realize that of all text (or books) written in the world, less than 1% actually gets published? Why would video or audio be any different? Just because you don't produce it, doesn't mean the MPAA industries are the only ones that do.
Also as you say, there's academia, PHD students I know use equipment that generates gigabytes per second. Or without equipment there's computer programs that do it from calculations. Now I agree it doesn't seem likely they'd use megaupload for that.
And that's just a few things I can come up with right now. Who knows what sort of computer stuff other people do that generates retarded amounts of data they need to share?
I'm also not saying that all that copyrighted stuff isn't there, it's just that if it's 25 petabytes worth of data, it just doesn't add up, the MPAA-represented copyrighted part of those 25 petabytes can only be a tiny fraction of that amount.
Interesting. My completely uninformed assumption is diametrically opposite to that. I'm guessing there's not much sophisticated de-dup going on, and _way_ more of the diskspace on MegaUpload is probably various bit-wise different rips of the same smallish set of Hollywood blockbusters. And while I agree the long tail suggests there's almost certainly lots of people out there with lots on non-mpaa-copyright encumbered files - I'd be quite surprised to find the area under the "long tail" was withing 2 or 3 orders of magnitude of the "fat head" occupied by all the copies of all the dvd rips and broadcast tv recordings.
I wonder if there's any believable data anywhere to see whether I'm wrong?
(Note: I've got a non-US-centric view of this too, here in Australia internet connection plans lag behind the US in terms of speed and bandwidth caps, so even though I've got friends who generate lots of GoPro footage for example, but they'll in general be storing them on locally attached harddrives, not trying to push gigabytes of raw data out into "the cloud". That might explain why I make tghe possibly-incorrect assumptions that I do…)
If I start heading about individuals being charged for content they have on their own machines (as opposed to content they're sharing is p2p), I'll invest in a few OpenWRT capable wifi/adsl routers, plug in some large disks, distribute them amongst nearby friends/neighbors (or out-of-wifi-range friends with adequate bandwidth), and run Tahoe on them all in a configuration that means its _provable_ that my sections of the encrypted files do not contain enough to decrypt into any identifiable copyrighted work.
That's still not foolproof, but adding the requirement that "they" identify the existence of a network of storage devices all storing encrypted segments of files, and then having them need to confiscate enough of them as well as my disks should put me that one step ahead of the "lower hanging fruit".
Save your children; MPAA cares mostly to keep servicing them porn; end them now!
So yes, PORN.
(Unrelated, your username is awfully similar to mine..)
> Each petabyte is equal to approximately 1,000,000 (one million) gigabytes, or the storage capacity required to store “…about 13.3 years of HD-TV video. About 50 Libraries of Congress.” (http://www.nasa.gov/centers/langley/news/researchernews/rn_d...)
I was guessing games and software make up some data. I just had a look at a torrent site. They have 400 pages, with 25 items per page. At roughly 5 GB per item that's still only about 50 terabytes.
I have watched some very rare international films on Mega and it never failed me on a search.
If you multiply out all the movies, all the multiple copies, all the TV show espisodes, all the different quality levels, you would get a few petabytes.
> I have watched some very rare international films on Mega and it never failed me on a search.
> If you multiply out all the movies, all the multiple copies, all the TV show espisodes, all the different quality levels, you would get a few petabytes.
I implore you to do a back-of-the-envelope estimate and get to 25 petabytes.
Please take into account that I purposefully started with the highest quality level currently in use, which is ~10GB per movie. The assumption is that lower quality levels will only be a fraction of that size therefore all versions of a single movie won't be more than 20-50GB total. Additionally, many older movies are not available in the high quality formats and take up only a tenth of this space.
Also I'd like to point out that the MPAA is the Motion Picture Association of America, which is an American trade association that represents the six big Hollywood studios. So they don't represent the "rare international films".
Still, I'm just making rough estimates myself as well. I'd love it if you could come up with a possible way to explain that, say, at least half of this data could consist of MPAA represented intellectual property.
I just don't think they produced nearly enough content to warrant shutting down all the data stored in MegaUpload, even given that MegaUpload probably does store all their IP illegally, in multiple formats.
Also, the government could have probably seized everything anyway as evidence. The problem with that is setting up that much rack space and network infrastructure isn't cheap.
That's Carpathia's basis for compensation. They are providing a service to the government. Seems like a no-brainer.
1 terabyte costs them $128.41 per year, right?
Amazon S3 would cost them roughly $444 per year, if they were using the Reduced Redundancy Storage.
The cheapest HD that I see on pcpartpicker (in terms of Price/GB) is the Western Digital Caviar Green 2.5 TB (5400 RPM) for $135.43, which is $0.054/GB. That's $54.17 per TB.
If you want a single backup, that's $108.34 per TB. Two backups (3 copies of each file), is $162.51 per TB.
So, if I'm doing this right, as long as their HDs last at least 15 months, on average, they have triple-redundancy, and the cheapest price ratio for consumer hardware. And I'm not even counting their power, network, cooling, or puny humans to maintain it all. That means their HDs, if they were made out of the cheapest parts I could find, would have to last significantly longer than 15 months, on average.
They're actually doing really good on price, if you ask me.
Or am I missing something obvious, or doing the math horribly wrong?
it is bullshit that they have to foot such a large cost for a government investigation.
Christ, I am paying Dropbox charges me, at $10/MONTH for 50GB the equivalent of $6.67 PER DAY in storage costs.
For my 50GB, I am charged $.33 per day.
I think whomever downvoted that thought I was referring to the per GB cost.
That would satisfy the needs of the government if they need access to the data, preserve it if in the future people are allowed to download it, and prevent the MPAA from complaining that it was given back to Megaupload.
I'm sure the cost of storage would not be minimal, but they could still use the rest of the hardware and not have to keep the servers powered up.
Possible problems:
- Maybe the servers cant be shutdown and brought back up without certain passwords or encryption keys
- Labor cost of shutting down and catalogging all those disks ( if done progressively would probably work )
- Others?
The compatibility problems you have trying to get data off a disks in a hardware raid make it impractical (do you have the EXACT SAME version hardware revision & firmware; without this you can't guarantee you can read it back)? Its either that or you have to pay data recovery guys to rebuild it.
Not to mention hard drive costs are still high, post thai floods. For enterprise gear we are getting most quotes ~ $300 AUD a disk, for consumer gear its ~ $130. Most servers are running at least 2 disks... that $286K worth of disks alone if your talking cheap - low capacity disks, not including labour to change the disks and test the hardware before you deploy a workload to it.
Forgive me for for sounding like a member of Anonymous but..
So? That's the government's problem. I don't see why a private company should be in any position where they're required (at wallet or gun point) to help in an investigation at their own expense. Pull the drives, warehouse them, and let the FBI do what they have to do. They have IT to rebuild the RAIDs.
Also $286K / $9k / day ~= 32 days, so in 32 days it costs them more to keep it all running that simply replace the disks.
Not to mention they could now lease those servers to another client and make money off of them.
I just wonder if they would be in a better financial position declaring it a loss and claiming the tax break / insurance then actually doing the work?
As long as they can support the hit to the cashflow, they are likely to be able to claim the damages back from either one of the parties of the case once its decided and / or insurance.
Just - put them behind bars. I said so.
Carpathia has to pay $65k to move the servers, then $37k per month to keep them in a climate controlled facility while powered down. Lost profits are still a relevant consideration. This is a doozy of a damages calculation. What's depreciation on assets that are rendered obselete by (something like) Moore's law?
I'd say Carpathia deletes the data and then supports the petitioners (those with lost data) in the takings clause case against the government. Carpathia claims indemnity against claims by pointing at MegaUpload and the Feds, but probably gets joined in a bunch of messy lawsuits. Real roll of the dice.
And is there any redundancy included in your figures? (25 petabytes / 135 terabytes == about 190 pods)
This figure doesn't include redundancy. Backblaze uses raid6, so the usable capacity is actually 117TB per pod[1]. With this configuration the final cost should be closer to $1.6 million.
[1] http://hardware.slashdot.org/comments.pl?sid=2341206&cid=36834390So even if you throw out the rest of the hardware and have zero redundancy, $40/TB * 25000 terabytes = $1,000,000.
(Of course, somebody has to ship and store over 8,000 HDDs, too.)
I would have assumed that the FBI would have actually seized the servers, or at the very least pulled the network cables out.
Have any hashes been taken of the drives?
I did some back of an envelope calculations... and its absolutely crazy. Tape would require over 17,000 Ultrium tapes. Now you could De-dupe... but the hardware to process and dedupe that much data.... not really an option. Not to mention the time to write that many tapes...
Something like thumpers (48 disk sun x86 boxes) would be expensive, last time I looked they were around say $30k for a large order... 160tb usable assuming 4tb disks are the thumper is split into 4 Raid 6 arrays... thats 160 thumpers... 4.8 Million
Even backblaze pods would likely be well over a Million...
This doesn't even cover hosting costs, transfer and such. Not to mention to be usable in court there are going to have to be processes in place to document compliance and validity of the copy....
All in all not a great place for Carpathia to be in.
FYI, the data is already comprehensively de-duped.
To build out web scale systems you generally use commodity gear and accept the overhead of duplication, heavy deduping requires massive IO, and there is no way i can see you can be dealing with that much data have that level of IOPS and be profitable charging what they charge.
Can Carpathia sue the Federal Government for NOT seizing assets. It's the data, not hardware. Data is transferable. They want it, take it.
Can Carpathia sue? This kind of injustice just makes me boil.
Alternatively, in a setup of this size, I imagine there'd be no end of redundancy configurations - RAID for individual disk sets, DRBD (/a SAN equivalent) across servers - turning them off would turn all that HA tech off. Meaning that, when you try to bring the system back up, the redundancy implementation might say "oh no, I've lost x peers from my set of n" and fail itself completely.
Shrug. I've no doubt explained it badly, but there are good reasons to keep them running. It's not just a case of "pop the hard drive out and use it elsewhere"; the logic associated with keeping 25 petabytes of data would also have to be restored to its current state.
Sure, a bunch of pissed off people will certainly be upset - but it's not the company's fault - they shouldn't have to bear this burden. I can't see how they could be sued by users for this, they didn't enter into any kind of agreement with the users, only with the customer.
Users don't care about that. Yes, some users are totally legitimate. Dropping their data because someone, somewhere thinks that the original service in general was 'evil' is not a solution that is as straight-cut as you'd like to present it.
If the hardware is left running, and not in official custody, how do any authorities know that the data isn't being tampered with?
They could of course migrate all of the contested data onto new storage. But its large; who would pay for that?
Pull the power and maybe all the evidence goes away...
So either the government needs to pay up, store the drives themselves or dismiss these thousands of harddrives from the witness bench.
Also, I cant see how the EFFs claim has any legal merit. Theres no obligation for a site to enable you to access data you sent them.
Of course, I think the whole category of forfeiture law it blatantly contrary to the Fourth Ammendment, so what do I know.
Taxes?
They wouldn't be held responsible for a breakin, would they?
Backing up your data on Megaupload and then not keeping a local copy is active stupidity of such vast scale that I refuse to believe it has ever happened in the real world.
To borrow a famous quote.. nobody ever went broke overestimating the intelligence of the public.
That said, having your backup service raided by LEO's toadying about for entertainment conglomerates was probably not even on the list of people's possible downtime concerns..
The purpose of having a backup is so that you can restore your data after the original is lost, because there is a good chance the original will be lost. Explicitly retaining it on your local system is not enough to keep it safe. If it was, there would be no reason to backup in the first place.
If a million people used the service for backups, it wouldn't be unreasonable to expect several of them to have a drive failure each and every day.
Do you think they were zipping up the contents of their own systems and then uploading them daily?
Regardless, why is the cost so high if the server is down? Does this $9000/day reflect the loss that Carpathia suffers from not re-allocating this storage to other customers? It would seem to me that given Megaupload's current state, it would be sufficient to leave the servers powered down and unplugged until the legal dispute is resolved... surely the cost of leaving a server idle is not $9000. I don't really know though...
Fucking exactly!! Have fucking MPAA pick up the tab.
EDIT: its going to be amazing (and will take years for sure) to see if this won't bite MPAA in the ass if the judge will rule that yes they do have to pay. Would looove to see that. This should be actually a rule of thumb -- if MPAA believes someone is infringing, court suit is entirely fine, but you guys (MPAA) will pay to keep the light on in the meanwhile.
Edit: The opinion above has NO legal basis whatsoever. As many have pointed out, it's not even legally possible. I made this comment solely from a "In a perfect world..." standpoint.
If they have been ordered to hold the data/servers as evidence, then the Government, as the requesting party, is solely responsible for the costs. If and when any party is found guilty at trial, the government can ask that the guilty defendant(s) be held financially responsible for all costs of prosecution, which would include reimbursement of these expenses.
I seriously hope that was sarcasm.
Expanding on this, if a company wants to claim that they always own a particular combination of bits should they pay rent for storing it on my system? The problem is people are generally short sighted and focused on the immediate future. Other considerations are rarely thought about until other people insist that something will affect their immediate future. There are costs associated with data that are rarely considered at the moment. No one to talk about this, it may change the way they do things and cost them money.
"historically and mind-bogglingly large amount of data" - you could say that again.