Google Has Spent $21 Billion on Data Centers | Hacker News Reader

Google Has Spent $21 Billion on Data Centers

datacenterknowledge.com

111 pointsajaxguy69 comments

Headline is wrong. 2 data points that the article is based on: 1) Google has spent 21 billion dollars on Capital Expenditures since 2006 (presumably gathered from SEC filings) and 2) "Google says the majority of its capital investments are for IT infrastructure, including data enters, servers, and networking equipment." (source unclear)

So, Google has spent at least 10.5 billion on "data centers" since 2006. There are notable disclosed expenditures that are clearly not "data centers" mentioned in the article, including 2 billion spent to purchase NYC office space. Still a huge number, but off by many, many billions.

Didn't they spend 2 billion buying the 111 Eighth Avenue office in Manhatten at the end of 2010? I believe that is data center realestate.

Why would they put a data center in Manhattan? Don't these things usually go where the space is relatively cheap?

111 8th Ave was already a major carrier hotel (there are several on the Island). And it once _was_ cheap real-estate, before Chelsea became trendy and before it was renovated into one of the most high-tech office facilities on the island. As is, it has more usable square footage than the empire state building.

Latency. Putting things far away from your customers means response time goes up.

True, but populous though Manhattan is, it hardly warrants its own data center. If latency was their prime concern they would build the data center in the middle of the population density. For North East US, a location between NY and Washington DC would make more sense.

It could be for transatlantic latency, but this would just be between the US and European data centers anyways.

In short, no, I don't think latency would be a big deal here. Correct me if I'm wrong.

When you're Google, 10ms makes a difference. In my own personal experience, a decrease of 100ms in my response time results in a 20% increase of traffic.

Latency matters that much.

Yes 10ms does matter, but building a data center in the middle of NY vs 20 miles outside is not going to create anywhere near 10ms latency, its not even going to create 1ms, on the Google network at least. Your cable provider is a far higher source of latency.

Depends on where the Meet-Me centre of the city is - it might even be that building.

111 8th Avenue is one of the main data center hubs in New York, but Google is interested in it primarily for office space rather than connectivity.

It's certainly used as a real google office, it has thousands of googlers in it.

There's a very steep depreciation on this capital spending too. Every year technology gets cheaper, and prior data center CapEx loses value.

Well, building a secure, well-ventilated, extensible, vault with reliable power sources, internet backbones, etc, is mainly an infrastructure cost that won't change year-to-year.

But servers are definitely a loss, that's for sure.

True, and Google's New York building had a very high retail cost too.

My impression of accounting (by no means my strong suit!) is the land doesn't actually get depreciated - it stays on the books at cost forever. I think the buildings and repairs do stay on the balance sheets for a while. The servers themselves, probably less so. If only I had the patience to dig through Google's financial reports. :-)

My question is if google spent 21 billion, why did microsoft spend > 15 billion on its datacenters. How much datacenter resources is each microsoft customer using? It can't be from just office 365, Azure, and Bing. So is the majority from just software updates/downloads?

Don't forget about Xbox Live and Skype data centers. I'm guessing that those might be contributing a nontrivial percentage.

Ahh forgot about Xbox Live. I know they recently announced having 300,000 servers: http://www.engadget.com/2013/05/21/microsoft-updates-xbox-li...

Mind you, 300,000 * 3k per server is still only 900 million.

Yeah... but a datacenter is more than just servers right? You need a building, infrastructure (cooling, electric bill, efficient designing and engineering etc), development. These costs get big. fast.

I would guess they are trying to build up their cloud business. They have lots of cash, and they want a successful business. It's easy to justify a large buildout in that case, with the belief that the revenue will come. This is a common trap for innovators: you have to spend a lot just to stay in the game, while also preparing for the future. MSFT missed a lot of opportunities over the past few years due to their focus on Windows and Office (as Ballmer admitted recently) and now they have to play catchup.

Yeah Microsoft went all in on the "living room computer" with the Xbox and Xbox 360 which made sense at the time of the original Xbox however the vision they had wasn't what came to fruition. What did we get instead? Well we are still in the middle of the transition but it is most certainly away from super powerful set to boxes and more towards things like the Apple TV and Chromecast with powerful tablet computers doing most of the work and just pushing the output over wireless to the little receiver.

Now obviously MS saw this which is why we have things like Surface now but I am really surprised that they decided to kill off the Media Center Extender platform they had. They could have easily taken that and built a device similar to the Chomecast/Apple TV for dirt cheap and had a decent head start.

Their plans for Xbox One are interesting with all of the TV features and HDMI pass through but the Xbox One is seriously expensive so it still only really appeals to pretty committed console gamers.

With Ballmer leaving in the next year I hope Microsoft bring in a decent tech-oriented CEO although I have a feeling they will go for another business guy who will do similar to what Ballmer did. Not to say Ballmer did a bad job. Sure he had some duds but he wasn't an awful CEO. I am going pretty left field here but I would love to see J Allard come back but I don't know if he is really MS CEO material. He certainly has a good image though and is respected by most of the tech community and I could see him being a Steve Jobs type figure head for MS if they played it right.

Then again I am just some random guy on the internet who hacks code for fun so I am probably totally wrong about it all but you have to agree seeing Allard back would be pretty interesting :)

Technet, Windows Update, Xbox Live, Hotmail, Bing, Skydrive and so on.

I often wonder what the bandwidth for Windows Update (and MS Download Center for OS patches) is. I would love to see some details on it if it has ever been discussed by MS engineers?

http://technet.microsoft.com/en-us/library/cc627316.aspx gives 500 Gbps for Windows Update in 2008. (2008 announcement: http://blogs.technet.com/b/mscom/archive/2008/04/11/announci...) I can't find recent update.

Thanks!

Don't forget Sharepoint services for large companies too.

Skydrive and Hotmail are some of the services much larger than people realize. Hotmail gives you very large inboxes, like Gmail, and Skydrive backs cloud storage and backup for most Windows devices.

There's also Dynamics CRM online (Salesforce competitor), Sql DB as a service and some windows phone services

I wonder if they included their own software license cost... One division billing another division.

Microsoft has been incentivizing its internal divisions to shift from legacy data center space to Azure. So some of that spending is for new servers and data center space to in-source services hosted by third-party providers. Or, in some cases, older Microsoft data centers. More here: http://www.datacenterknowledge.com/archives/2011/06/01/at-mi...

Does any one know the relative costs of hardware vs building and cooling? If I had a cool billion to spend, how much would I spend on building/cooling/power/generators vs servers and networking hardware?

Power failover and HVAC systems are on-going costs that usually exceed server hardware costs (both initial and replacement) by a long margin. Over a three-year period you'll probably get to spend 1% on actual server/networking hardware.

I'm just pulling that number out of fat air, but you'll be spending far more on other things.

>Power failover and HVAC systems are on-going costs that usually exceed server hardware costs (both initial and replacement) by a long margin. Over a three-year period you'll probably get to spend 1% on actual server/networking hardware.

This is true... if you are running on 10 year old servers, and brand-new power/hvac equipment. On a more reasonable replacement schedule, if you pay what I pay for power/cooling/rack and what I pay for hardware, you are closer to half and half.

Now, I pay a lot more than google does for both power and harware- but I suspect that the google discount on datacenter space is vastly greater than the google discount on hardware. If nothing else, california power is like 2x power anywhere else. Also, i'm strictly retail when it comes to datacenter space. I rent 23Kw usable from Coresite... meaning I'm renting like 28Kw capacity... so I'm not a single U customer, but I'm still firmly 'retail'

You can alter this quite a bit just by going from X series to L series CPUs or by replacing your hardware more or less often. (I'm on L and E series xeons.)

But, anyhow, from that? I'd think that google probably spends more on hardware than on power/cooling/etc...

Have a look at this [0][1], Chapter 6 should give you a some idea.

0: http://research.google.com/pubs/pub35290.html

1: http://www.morganclaypool.com/doi/pdfplus/10.2200/S00193ED1V... [PDF]

Yes, and they aren't particularly relative :-)

But to answer your question you can get pretty close by translating into units of watts and remembering a couple of simple rules.

Rule one is you only use 80% of the provisioned power (this is a safety rule with respect to things like circuit breakers and infrastructure, it keeps random surge events from destroying everything).

Rule two is that your power use efficiency (PUE) ratio determines how much cooling you need.

So lets say your PUE was 1.3 (that means total power / computer power = 1.3, the extra going into cooling) and you were building a 10 Megawatt data center. Well you have 8 Megawatts of actual power (80%) and 8/1.3 or 6.15 Megawatts for machines. Going by 750 watts per server (the PSU label [1]) you can put 6.15/.00075 = 8,200 servers into that data center. At 500 watts/machine you are 12,300 machines. So to make the math easy lets say it is 10,000 machines.

If you servers cost you $5,000 each then you're at $50M in server hardware for a 10MW install. Network switches are $100/port (assuming you're not doing 10G ports) and cable is way more expensive than it should be adding another $20 per port to connect it to the switch, so another $1.2M there. You need to hold them in something and rack solutions go from the less expensive (4 post) to the more expensive (fully enclosed) with a range of $500 - $2500. How many you can put in a rack will change your per machine rack cost up or down, but its somewhere between $25 - $125 per machine so another $1.2M so lets be generous and say our $55M for your 10,000 machines which are going to consume 8MW of your 10MW provisioned power.

Building the data center though, especially if you include land acquisition, that is going to be pricey (of course you can write it off over 27.5 years but still) If we start with the fact that a data center is really just a big warehouse with a cold box inside, we can look at building a warehouse [2] and some folks have some priced out [3] which put it about $50 - $75M range, and that is before you add cooling, back up generators, and pay the local utility to run a 10MW power line into your building. I got some estimates from Crane for a 'pocket' data center project I was looking at (about 3MW) and those extra costs are probably going to exceed $100M (split 33/33/33 into cooling, back up power, and secondary construction).

So back of the envelope 3:1 construction costs over equipment cost. And maybe $20M per constructed MW of compute.

[1] This isn't what you would do since machines don't consume that much, they can only not consume more than that, but whatever.

[2] http://www.reedconstructiondata.com/rsmeans/models/warehouse...

[3] http://www.mohavedevelopment.org/media/docs/One%20Sheets/Moh...

Thanks for the informative post. Are you aware of any non-proprietary tools for doing procedural generation of data-centers?

I'm imagining a generate-and-test cycle given estimates for the various component prices.

I don't know of any off the shelf ones, but there are "data centers" like Google and Microsoft and Amazon build for their "gear" which they resell as "services", and there are "data centers" which are built by Real Estate Investment Trusts which are designed for 'unknown' gear and 'unknown' services.

It is more expensive to build the latter since you cannot know ahead of time what gear to support. Whereas if you're building a box full of Open Compute servers you're in a much better space.

The "services" data centers are really much closer to building sized servers than they are hosting facilities for rack mount servers.

I expect like other specialized facilities part of the 'value add' as a construction agent is both procedural generation tools and experience with various cost trade offs which really only comes from building several of them.

Yes, see the blog post titled "Overall Data Center Costs", from James Hamilton (Vice President and Distinguished Engineer on the Amazon Web Services team):

http://perspectives.mvdirona.com/2010/09/18/OverallDataCente...

So they should use Hadoop?

I'm not sure if you're posting sarcastically, but Hadoop was built based on whitepapers Google released about MapReduce and the Google File System. I think they are covered in this department!

He's definitely being sarcastic.

They're posting in reference to [1] I believe. It's supposed to be a funny comment. They are pretty damn well covered in this department, I agree.

[1] https://news.ycombinator.com/item?id=6398650

What if you were trying to start Google today (assuming it didn't exist), and you had to bootstrap with maybe only $500 available to spent per month? How would you guys do it?

Well this would be a place to start... http://commoncrawl.org. It would be interesting to see a solution to the data center requirement that uses some adhoc bit-torrent like system.

I presume you're talking about hardware budget. You probably wouldn't get very far with such a low budget, but I think your question is how to get the most bang for your buck. The answer is with dedicated hardware, especially for computationally expensive apps. $500/month dedicated hardware will beat the pants off of $500/month of EC2 instances for example, but it comes with the cost of being less flexible.

Amortized over 3/5 years or non-amortized? Different answers :)

Read up on the history of DuckDuckGo; AFAIK it was one person in the beginning.

Hold out with a handful of servers until Google bought you.

is that all? that's only $10k per machine if they have 2M machines (which I'd guess is a conservative estimate)

NOTE: I'm not being snippy - that seems very efficient given the building, HVAC, networking, and the fact that they have probably had a lot of machines fail.

How much of it was subsidized by NSA?

http://www.theguardian.com/world/2013/aug/23/nsa-prism-costs...

Nearly nothing considering your article mentions "millions" for the whole program and Google has spent 21k million just on its own. Google makes more on interest from its cash on hand than it has received from the NSA.

It's really disappointing that you're responding to what is the #1 comment. The NSA noise just overshadows much of the meaningful discussion on HN.

It is quite disappointing. I'm personally very tired of how widespread and acceptable the conspiracy theories have become, everyone's wearing their tinfoil hats and running around screaming like the government's taking away their first born child. I can't say I care anymore. But that's like, my opinion man.

And I have spent tens of thousands on rent.

Has anyone dug in and figured out how these #s are impacted by MMI?

They do have 100% uptime, right?

Google had a 5 minute outage this August, where global internet traffic dropped by 40%.

http://www.fastcompany.com/3015942/most-innovative-companies...

I wonder how much money Google has spent per user. $10? $50?

...and did they improve the fortunes of their advertising client base, or merely suck the wind out the prevailing sail without offering anything in return?

You're thinking of Groupon.

It is a valid question. As a 'brand building' advertiser, I never felt like most of the google products delivered as much value as they claimed; they all seemed to be targeted at products people will buy after the first impression (which is... not, generally, how products in my sector are sold. People buy because they have heard of me. The 'hearing about me' is important, not the last click.) and the tracking mechanisms all seemed... naive. (now, that last part might just be the effect of them dumbing it down to the point where a small business owner, like me, can understand whats going on, but man... in that case, we advertisers trust google a lot.)

Of course, I'm small, cynical, stupid, and don't spend much effort on advertising, so it's possible my impression is wrong.

according to here: http://www.internetworldstats.com/stats.htm

there are 2.5B internet users. since google is more than a search engine(android, gmail, adwords and so on) we can assume that most of these users somehow interact with google services, so just adding other expenses and deviding the number to 2.5 billion would give a quite accurate result.

Google hit a billion unique per month in May of 2011 and they've spent $21 billion in the last 7 years. Assuming no growth (which is absolutely not the case) they've spent an average of $3 per user per year.

google doesn't throw her money away !! I'm positively sure that they will get it double a triple :) they don't do charity

Google actually does a lot of charity

http://www.google.com/giving/

Publicly traded companies even if they do "charity", they do it expecting good PR in return. Any other kind of charity (i.e. done without publicity, only to make the world a better place) would be against their duty to the shareholders.

Due to dual-class structure Google realistically has only two shareholders. The only legally mandated duty is quarterly report of the numbers.

You're just bringing up the tired thought experiment about whether "true altruism" exists. Is it really corporate "charity" if the company expects good PR and eventually profit as a result? Is it really individual "charity" if the person gets a really good feeling as a result?

If real kids are getting real flu shots or whatever, who cares about the PR? Getting personal satisfaction from charitable giving doesn't make your donation less valuable.

"You're only saving me from a life-threatening illness because you have a big ego! I don't want your charity," said no one ever.

Is your point that we shouldn't be supportive of corporations doing charity?

There are corporations which have rationally (long-term profit-wise, or whatever) decided to do charity work. There are corporations which have decided not to. I support the former.

google.org is their charitable extension.

Reply on news.ycombinator.com