Photo tour of Facebook’s new datacenter | Hacker News Reader

Photo tour of Facebook’s new datacenter

scobleizer.com

Very cool, but why in the world would he use Instagram? All it did was crop the photo down to a square, so we couldn't see parts of the datacenter that weren't within that square.

Some of them look like polaroids now!

I use Instagram because I have about 10,000 followers on that service. I go where my friends are.

It seems silly for the quality of the photos in your article to be impacted by the way you share them on a social network.

Rackspace are building a datacenter based on the opencompute plans, very cool. Exactly what I had thought about based on the announcement of open compute. Hosting companies, large companies and all sorts will be leveraging this information to increase the scale of computing in datacenters as a whole. I hope more companies will contribute to opencompute so we continue to see improvements like this. The facebook datacenter looks awesome, as you can see they are serious about being a long term company rivalling the likes of google.

This is absolutely fantastic. I've been following the Google datacenter efficiency project for a while, and an opensource competitor will only make things better for the world. I can't wait for the day when more large companies either embrace this concept for their own datacenters or outsource to datacenters built on these technologies.

How can this be more energy efficient than systems using cold ocean water to dump the heat surplus? E.g. The datacenters in iceland, where Opera has its datacenters. Would love some more details.

The problem is not what to do with the heat, but how to transfer the heat more efficiently than via air (which is very inefficient so you need lots of energy to blow lots of air around).

There's a cute promo video about half way down this page:

http://www.datacenterknowledge.com/archives/2011/04/12/green...

which shows one company's oil cooling solution.

Discharging heat into nature is considered pollution by many countries by now, and in particular discharging it into bodies of water. You might simply not be allowed to do that, or you might have to pay hefty pollution taxes.

"Discharging heat into nature is considered pollution by many countries by now" - Citation please. I'm not doubting you, this is just the first I've heard that discharging heat into nature is considered "pollution.", and I'd be interested to hear the context. I suspect it's in rare scenarios where the water it's being discharged into has a limited heat sink (small lake/river?) Regardless, I find it highly dubious that discharging heat _from a data center into an ocean_ is considered pollution by any country in the world. And I'm willing to wager no country in the world charges hefty pollution taxes for doing so. In fact, I'd be surprised if anyone, ever, has been charged pollution taxes for discharging heat into the ocean. The ocean is pretty big. The United States, in an entire year, uses enough energy to only equal 1% of what the Sun does every day in terms of heat loading.

From: http://en.wikipedia.org/wiki/Solar_energy#Energy_from_the_Su...

"The total solar energy absorbed by Earth's atmosphere, oceans and land masses is approximately 3,850,000 exajoules (EJ) per year."

1/365 of 3,850,000 exajoules = approx 10,000 exajoules.

From: http://en.wikipedia.org/wiki/Joule, " Energy in the United States used per year is roughly 105 EJ."

I'd be surprised if places with active geothermal vents would even notice the human induced heating. In that case Iceland seems like an excellent choice. The main problem is when it comes to active fisheries being located near the effluence.

Another thing you can do is to simply use the slightly warmer water like you would use cold water. Half of downtown Toronto's office towers (including 151 Front) are cooled with water from Lake Ontario that then enters the city's water supply.

Just as Cornell U uses the water from Cayuga Lake. Pretty cool :)

Not a link, but I do know that discharging a bunch of hot water into a stream that is typically cold can wreak havoc with the local fishies that might be adapted to a certain environmental regime.

Like anything, though, a little bit of dumping is just fine, it's when we scale to the size of human civilization that it becomes a problem.

In the US, the Clean Water Act covers thermal pollution (well, "thermal discharges"), and based on the language it looks like the provisions have been present since 1972. http://www.law.cornell.edu/uscode/33/1326.html

Actually I'm a bit surprised this isn't that well known with the recent focus on nuclear power, since thermal pollution and cooling system intakes that kill fish have been a minor source of controversy for some plants, including Indian Point and Vermont Yankee. Recent news on the latter: http://vermont4evolution.wordpress.com/2011/04/07/anr-to-rev...

And just for convenience here are the first few results off Google for thermal pollution:

http://en.wikipedia.org/wiki/Thermal_pollution

http://www.pollutionissues.com/Te-Un/Thermal-Pollution.html

http://www.rpi.edu/dept/chem-eng/Biotech-Environ/Environment...

Thanks - I think it's important to note that the citations you mentioned don't treat the discharge as "Pollution", per se, but instead discuss what the impacts of the discharge is. That is, there's no limit on what you can discharge (as there is with pollution), but instead a limitation on the damage you can do to the local ecosystem.

I'd be interested in knowing if any country has adapted statutes with regards to treating thermal discharges as pollutants that need to be regulated. I'm guessing, based mostly on the pre-existing scale of heat loading from the sun, that thermal discharges are not considered to be a pollutant by any country, and you are free to discharge as much of it as you want, just as long as you don't damage the fish/shellfish/wildlife in the area. This also suggests if there are no fish/shellfish/wildlife in your discharge area, you are free to discharge as much as you want.

The EPA defines "pollutant" for the purposes of the Clean Water Act as:

Pollutant means dredged spoil, solid waste, incinerator residue, filter backwash, sewage, garbage, sewage sludge, munitions, chemical wastes, biological materials, radioactive materials... heat... 14 C.F.R. § 122.2 (definition of "pollutant").

EPA discharge permits, which are required to discharge any pollutant in any amount into US waters, certainly do regulate thermal discharges.

not treated as a pollutant but NZ runs a 'consenting' regime and one of them is to do with the power station (huntley) on one of the main rivers (the waikato) - it is only allowed to operate so as to maintain a temperature window on the river. this is complicated by the fact that the river flowrate (hence volume) is bounded by a consent with a different company that operates the hydroelectric system...

Or how about data centers that use the excess heat to provide heating for homes? http://www.treehugger.com/files/2009/12/data-center-housed-u...

Which brought up the question: why Prineville. ... low tax rates and friendly climate to business, etc.

In other words, whatever labor is needed is considered unskilled. You found a tech company in the Valley, NYC or Boston because, even though the costs are high, that's where you can attract the talent. When the quality of employees is a non-issue, you may as well place it in a place that's economically depressed, i.e. desperate.

I would imagine they will be importing most of the talent -- either from Portland, (inter)nationally, or at least from Bend or Eugene or Corvallis. My guess is that it really is the taxes + land costs + electricity costs + environment (cool high desert is great for evap cooling). Also Oregon has good business taxes, if high income tax. Finally, there is good skiing nearby at Bachelor, nearby Redmond has a great airport, etc.

Just a note: In the high desert, "evaporative cooling" (all those water jets, fans, and filters) are refered to as "swamp coolers". Much more poetic.

Y'all should know that we good ol southern folk calls em that too.

Note the dual axis solar trackers in the field adjacent the facility. They are from Bend, OR solar startup PV Trackers http://www.pvtrackers.com/

Being from Oregon, I always get a kick out of seeing this... Facebook, in Prineville. It's a small town that's not much of anywhere. Previously, it was known as the HQ of Les Schwab (free beef!) Tires, but even they moved over to Bend.

I knew someone who worked at the Les Schwab HQ. She said that as soon as it was seemly after he passed away, they moved the operation to Bend. I can imagine an interview -- "Look, we want to hire you from Stanford Business School to run this multimillion section of our company, but Les is 85 and as soon as he goes we PROMISE we are moving to Bend."

Note -- this may seem sort of horrible, but I am under the impression that Les was the sort of guy to be completely face forward, and could say "Look -- I know you all want to move to a town with a Starbucks, but you just have to wait until I die. And then please keep donating money to the local football team when I go."

Looks like he was a pretty good guy: http://en.wikipedia.org/wiki/Les_Schwab

I wonder why there are at least two ridiculously big (like, 4096 pixels wide?) embedded images that are forced to width=500 in the HTML? Ever heard of thumbnails?

So this is fun to look at, but considering how datacenters are supposed to be secure, isn't this counterproductive to them?

Generally datacenters are weary of photos due to housing hundreds or thousands of different customers, many of which who do not want photographs taken of their equipment or even to have the geographic location of their equipment exposed (governments, financial institutes, etc). At Facebooks datacenter the only occupant is Facebook, so it's nice that they aren't as paranoid and will show these photos.

Well, except the "no photos of the older equipment" rule that Scoble apparently ran into :-)

There were various things we weren't allowed to take pictures of but mostly they were very open. You can't even get to the bathroom without a keycard. I don't think they have a reason to be paranoid.

isnt this: "You can't even get to the bathroom without a keycard" contradicts this: "I don't think they have a reason to be paranoid"

Hah. You got me there.

Two facts:

#1. Nobody attacks data centers. No, really, they don't. Have some solid concrete walls, a man trap for people entry. a secure receiving area, and make sure your external doors are locked and you now have more security than is required for any data center short of critical infrastructure (nuclear/air-safety) - and probably enough for that as well.

#2. Even if people attacked data centers (which, honestly, they don't) - there was nothing in these tours that would increase their odds of successfully doing so. Much easier to just get a job there if you really had some desire to cause damage.

Indeed. Facebook's operational risk is probably geared towards making sure employees don't leak people's private data, rather than making sure someone doesn't blow a wall in the data center and cart off their disks. Because the first one happens, but the second one doesn't. Unless it's a movie.

I know of two datacenter "attacks".

However they were inside jobs by disgruntled former employees which I know disqualifies my original concern.

Oh, I've been involved in several incidents in Data Centers -but they all were a result of people who had been legitimately been let in to the data center. Contractors, Electricians, Customers, Telcos.

In the entire history of data centers, I'd be surprised if there's been more than a dozen external penetrations on a data center that had the following:

  o Solid Concrete walls.
  o Man Traps w/security that vets all entry.
  o External Locking Doors that are always Locked.
  o Secure Shipping and Receiving (Basically Rolling Doors 
    that lead into a "Package Trap" for gear where it is 
    deboxed)
  o And, I should have also mentioned, customer cages.

On the flip side, I'm betting we could find thousands, and more likely tens of thousands, of incidents related to people who had been let in through proper channels. If you want to invest effort in securing your data center, that's where you'll get the biggest payoff.

> Nobody attacks data centers. No really they don't.

I suggest that these "facts" are too general: nobody currently physically attacks this sort of datacenter in the US.

There are clearly many examples of physical attacks on other sorts of datacenter, eg http://en.wikipedia.org/wiki/James_McDade.

Correction: Nobody has attacked a datacenter.

I think that's a fact because the folks who blow things up aren't cognizant of their importance, or aren't using weapons appropriate to the task.

When the US attacks a country, what do they blow up first? A: Airfields, SAM sites, and Command/Control/Coordination centers. Datacenters and big telco facilities fall into that category.

It doesn't happen often, but it does happen. That said, yeah, it doesn't increase odds of a robbery to give press tours.

http://www.datacenterknowledge.com/archives/2007/12/08/ocean...

http://www.datacenterknowledge.com/archives/2007/11/04/armed...

http://www.zdnet.co.uk/news/security-management/2006/11/02/l...

http://www.pcmag.com/article2/0,2817,2381108,00.asp

http://www.theregister.co.uk/2006/10/20/easynet_brick_lane_r...

would be even more interesting if a tech person could do a tour/photoshoot instead of a non-technical buzzhound.

do all/many queries to facebook go to Prineville and how physically? shouldn't datacenters be in the middle of some thick cable hub?

The article explains that Prineville is a railroad town.

Some time back most railroads switched to using fibre optics for signalling purposes, not least because copper requires more repeaters and is more likely to be dug up by dickheads.

The amount of bandwidth needed for railroad signalling is pretty minimal, so many railroad companies now moonlight as telcos.

Also, back in the 1990s, a lot of firms laid additional fibre along the railroads because of the existing rights-of-way. Laying alongside a railroad is much faster because the legalities are settled -- you only need to negotiate with one organisation, not every little county you happen to cross.

Consequently there is a lot of spare capacity running along railroads.

By the way, why did Rackspace send me there? For those who don’t know, I’m a full-time employee of Rackspace which is the world’s biggest web hosting company.

Scoble has a day job at Rackspace?

Yes. Here's what I do at Rackspace: http://www.quora.com/What-does-Robert-Scoble-do-at-Rackspace...

Nice query-spam suffix. Nothing personal, but it sounds less like a day job and more like a retainer for you to give RS presentations/info based on your normal travels.

Why is a corporate guy wearing a jacket showing the datacenter ? Doesn't represent the startup spirit IMHO.

Because that's how Tom dresses. He's not a "corporate guy" but manages the team who runs our data centers and was directly involved in building this one.

Having "Startup spirit" doesn't mean imposing a new uniform of tshirts and khakis.

Reply on news.ycombinator.com