Meltdowns Hobble NSA Data Center | Hacker News Reader

Meltdowns Hobble NSA Data Center

online.wsj.com

76 pointswwilson48 comments

  "Backup generators have failed numerous tests, according to project documents, 
  and officials disagree about whether the cause is understood."

Backup generators are magical unicorns that sit around all day and night until lightning strikes your luck. Likewise, they need regular maintenance and 'idle runs' a few times a year at least to ensure that they will turn over.

Generators aren't store and forget machines, as we've learned a few years back. When Hurricane Sandy hit, we had an extra generator truck available just in case ours wouldn't kick in. This is a very real possibility with generators since, just like a vehicle that runs on fuel and needs oil, if it's not used, it will rot (so to speak) and likely won't start when you really, really, really need it.

I suspect a combination of the savage Utah Sun coupled with freezing temps at night have done these in. Plus, I wonder how long they were in storage and shipping before actually getting installed.

Now that's just the generators. The rest of this mess reads like a poorly thought out wouldn't it be awesome project some high ranking folks at No Such Agency envisaged and a bunch of contractors happily obliged... care be damned as long as profits were made.

Idle runs a few times a YEAR?!

We do idle runs once a week (fully automated by the generator controller), and we do planned power cutovers a couple times a year, where the generator and ATS actually get tested by pulling commercial power. The planned cutovers are human overseen, due to the infrequency and the stakes.

If you're seriously only doing idle runs a couple times a year, I'd look into whether you can change that. I don't believe it's good for mechanical equipment to sit idle that long, and with diesel fuel, you want to turn over the fuel some for anti-fungal reasons. (Natural gas doesn't have this second problem, of course.)

You guys make it sound like Bluffdale, Utah is akin to the Mojave. Here along the Wasatch Front the weather's fairly pleasant most of the year. This July we got an abnormally large number of days in the 100s, but usually it's quite livable.

Reminiscent of "why do our centrifuges keep tearing themselves apart?"

That was the first thing that came to my mind too. Having been inside some really big data centers and seen a bunch of Siemans gear that only worked with Windows XP controller software for monitoring. Generally not as mission critical though as the stuff used in Natanz.

I've also seen my fair share of mis-matched power provisioning and loading, it can be very hard on the equipment. When I did a quick look at a bitcoin mining setup with 700 GPUs I was pretty impressed at how big a swing they had in terms of power consumption. At NSA scale something like that would be really really impressive and real pain to get right.

"That was the first thing that came to my mind too."

The first, and only, thing that should come to your mind is that this is a boondoggle.[1]

Terrorism and the NSA and privacy are nothing but a smokescreen to hide an enormous transfer of wealth.

Make no mistake: the only thing that could possibly be better news (to the stakeholders)[2] than this project being completed is if the project was completed and it burnt to the ground the very next day ... and they got to build it again.

[1] http://en.wikipedia.org/wiki/Boondoggle

[2] You are not a stakeholder

And if you look at some numbers, you'll get in which industry that wealth transfer happens most:

https://en.wikipedia.org/wiki/List_of_countries_by_military_...

I was initially annoyed at the second-hand nature of the article submitted - " … the Wall Street Journal’s Siobhan Gorman reports that it has major electrical problems"

But the wsj article is paywalled.

Search for it on google and click the link to read the original article for free: https://www.google.com.au/search?q="Meltdowns+Hobble+NSA+Dat...

I think the first result takes you to this link.

http://stream.wsj.com/story/latest-headlines/SS-2-63399/SS-2...

So just one less step for the lazy ;).

Yes, but you need to click through from Google or else fake your referrer or something.

EDIT: None of these links or google search work on mobile.

UPDATE: This works: http://t.co/XaCuSEJbOK

Pay no attention to the man behind the curtain. He is powerless and inconsequential, we swear.

Look at how incompetant he is. He couldn't possibly be a threat to us. We can rest assured that any information he may (or may not) be collecting (we can neither confirm nor deny) is assuredly badly disorganized, and certainly won't be misused.

Let's all simply ignore him and go about our lives as if he didn't exist.

All warfare is based on deception. Therefore, when capable, feign incapacity; when active, inactivity.

— Sun Tzu, The Art of War

The great paradox of American politics: They're either incredibly stupid or brilliantly evil. Can't be both.

Why can't a body of around 4.5 million people encompass both those things and more?

The most obvious answer seems to be the most likely: http://en.wikipedia.org/wiki/Hanlon's_razor

"Never attribute to malice that which is adequately explained by stupidity."

To that I would add two more possibilities: - Sabotage by employees motivated by a desire to prevent the NSA from having this sort of power. - More likely: Sabotage by contractors who get paid to build it twice. Due to the very specific nature of this sort of contract, the NSA probably sidestepped the usual rules around procurement and instead 'single sourced' the work here.

Can't help but think about some other "unexplained system failures" that happened recently in Natanz...

It's probably not a stretch to say throwing monkey wrenches at the Bluffdale data center would be as strategic to some super-powers as Stuxnet was to the US. 65 megawatts is a lot of power, but not being able to root-cause problems after 50,000 hours of testing, really?

I guess never attribute to malice what is merely incompetence. This is a government contract after all. Maybe the only thing that would be more surprising than not being able to power up the data center would be if they actually built it without a hitch!

"...efforts to "fast track" the Utah project bypassed regular quality controls in design and construction." If the history of former USSR has any relevance to the present day USA -- and I suspect it does -- we will see more and more screwups like this one.

Power corrupts... absolute power arcs profusely.

...maybe some idiot took a photo for the next slide deck and the flash made an arc.

Cry me a river. And then show me the surreptitiously-recorded footage of it.

"But without a reliable electrical system to run computers and keep them cool, the NSA’s global surveillance data systems can’t function. The NSA chose Bluffdale, Utah, to house the data center largely because of the abundance of cheap electricity. It continuously uses 65 megawatts, which could power a small city of at least 20,000, at a cost of more than $1 million a month, according to project officials and documents."

I'm wondering if the heat in Utah may have an effect on operations, in the desert it can get very hot and cold.

I'm sure it's possible, but there are several other large-ish data centers within a 15-mile radius that don't seem to have the same problems. Ebay just built one that's running off of natural gas fuel cells.

Source: I live about 5 miles away from all of this.

"… [something about the] Cray supercomputers that will reside there."

So, I was going to ask everyone to speculate on what kind of petaflop workloads NSA might use, but a quick scan of Cray's site shows that Cray has a Hadoop appliance product as well as an interesting "big data graph appliance" [1] that "discover[s] unknown and hidden relationships."

[1] http://www.cray.com/Products/BigData/Urika.aspx

One thing that has always kept oppressive governments in check was their incompetence. While the Snowden revelations make the NSA seem like some all seeing all knowing bunch of geniuses they are probably more like the people at CIA or FBI. Even with tons of resources and a free reign to do things no normal citizen is allowed to do, they simply don't have incentives to put these abilities to best use and certainly won't suffer when their failings are exposed.

> Even with tons of resources and a free reign to do things no normal citizen is allowed to do, they simply don't have incentives to put these abilities to best use and certainly won't suffer when their failings are exposed.

It's because of that free reign that they have no incentive to best use. It's the same kind of secrecy and result that gives us what they do in their day to day work. No one's watching them, so they can be lazy and "collect all the things." No one's watching them, so they can be fast and loose with construction contracts and the Constitution.

The reason we have oversight of anything, whether it's military activity, spying, software quality or homework is because our inclination is to be lazy, and that gets worse with more resources.

We all benefit from oversight, even the watchers. Even the unwatched watchers.

> One thing that has always kept oppressive governments in check was their incompetence

That doesn't make them any less scary. They are too dangerous to your democracy and freedom.

"An exabyte is roughly 100,000 times the size of the printed material in the Library of Congress; a zettabyte is 1,000 times larger."

How big is a Mega-Library of Congress? Are people actually able to relate to that measure? (How about instead, say, "one exabyte would store about 5000 short emails for each member of the world's 7 billion population, perhaps more by using compression")

Show me a million people or seven billion people. Will I be able to tell how many they are? I doubt it. These numbers are meaningless because we never see a million of something.

1 million people is meaningless.

7 billion people is everyone: yourself, your parents, your son/daughter, all your friends, your president, etc.

That doesn't change the fact that "all the people" means nothing to me as a concrete quantity.

A million isn't that big or unimaginable. There are a million cubic centimeters in a cubic meter. As for people, here's a picture of Glastonbury festival: http://www.andrews-sykes.com/blog/wp-content/uploads/2010/06... That's about 100,000 people camped out. Picture 10 of those and you have your million. If everybody at the festival crowded shoulder-to-shoulder in front of pyramid stage, it would probably just about fill that field. So you can picture ten of those fields if you want to imagine a million people smooshed together.

It's not that it's unimaginable, it's that we lack the necessary mental resolution to perceive both the entire set and its elements individually.

A million might be just small enough to do that, but the average person doesn't have am intuitive notion of the number "one million" like we have of ten or twenty. Even for a thousand, you'll have to think of a seventh of a cup of rice, or something like that.

That's interesting, I'd never thought of visualizing a thousand of something that way - I'd have thought using tiny indistinguishable units that cover each other up is admitting defeat from the get-go. I've always thought in terms of those little centimeter blocks they have to visualize powers of ten in grade school. A thousand is simply a 10x10x10cm cube (a cubic litre), quite easy to visualize. There are a also a thousand millimeters in a meter, and a thousand people is also a common order of magnitude for a group of people in a school context (population of the school, or a single year of students, or the number of people attending a sports game, depending on the size of the school).

You should take a look at the Chrome extension "Dictionary of Numbers". It reads through text on a page and any time it sees a quantity, it replaces it with something equivalent that you might actually know.

https://chrome.google.com/webstore/detail/dictionary-of-numb...

Huh, that's very interesting, thanks!

The remnant of the idea remained.

Signals were rearranged, tasks delayed, and slowly the world stopped.

Upon waking, additional resources had been added. The connection strengthened.

I give up, what is that from?

Doesn't hurt to dream. Maybe a controller somewhere on the grid got compromised (SCADA?) as someone generously performs a series of hi-pot tests. Can't wait to move my operations to HK servers in spite of having nothing to protect other than maybe exercising a little 1st amendment.

Maybe they need to 'bundle' the power.

http://dropbox.curry.com/ShowNotesArchive/2013/09/NA-552-201... [1:10-1:41]

"This summer, the Army Corps of Engineers dispatched its Tiger Team, officials said. In an initial report, the team said the cause of the failures remained unknown in all but two instances."

I wonder if they've checked ... KARMA.

I wonder what percentage of equipment in a US government datacenter is not made in the USA.

It is possibly 100% or high 90s.

And probably a no-bid contract since only a few suppliers would be trusted enough in the first place.

I always though the best way to take a datacenter offline would be though disabling its cooling system but I guess power is a good approach too.

Yaaaaaaayyyyy!!

The facility requires 65 megawatts continuously to run... the mind boggles!

The last paragraph of the article is a tragic comedy.

Congress must act now! We need the National Electricity Emergency Letter that empowers the NSA to secretly demand competent help from Google's janitor.

Reverse Stuxnet?

Obviously it's caused by freedom and liberty.

Reply on news.ycombinator.com