Hacker News Reader

Comment by cperciva | Hacker News Reader

I think you're overstating the dangers a bit. A hot aisle might be uncomfortable and not a place you'd want to spend many hours, but it isn't going to kill you quickly. If by some series of amazing coincidences you managed to get stuck in one with no way to call for help, you could just start unplugging servers until someone came to investigate and rescue you.

View on HN

Years ago a data center outside DC in Silicon Alley had an A/C failure. Our equipment was reporting temps of 140F on intake. We were allowed 10 minutes in, one person doing work, another observer, both required to have squirt water bottles and a two-way radio. We were instructed not to carry anything and to use a cart. We were told not to remove equipment that weighed more than a few pounds.

I pulled 10 drives so we could run them to another data center as a 'just in case'. The guy I was with passed out about 60' from the door. The air temp outside was ~72F and they had portable hurricane fans blowing air into the data center.

In that ten minutes, I drank 32 ounces of water. I remember that because I went to pick him up, kicked my water bottle by accident and it fell over because it was empty.

He spent 30 minutes in an ambulance, about three hours in a hospital from 10 minutes exposure. It took me about two hours to recover to the point where I felt I could eat.

Don't lose respect for heat. :)

You guys deserve hazard pay. :)

Saunas have 70C - 100C temperature.

http://en.wikipedia.org/wiki/Sauna

You had only 60C (140F).

Fainting after just 10 minutes in 60C is too fast.

BTW, most likely reason your friend spent time in ambulance and hospital is because your employer wanted to be on the safe side, not because your friend really needed it.

I'm reasonably sure the air temp wasn't 140F in the data center, but, he had Heat Stroke or Hyperthermia. He did collapse and I remember calling for help on the radio when I saw him go down. In the space of about 10 seconds when he said he was feeling weak, he went from looking flushed to pale as a ghost and bam, he was on the ground. He was wheeled off in a stretcher and I remember sitting in the back of the ambulance with him. After 30 minutes, he was still not feeling well and reluctantly accepted the hospital visit.

He and I were business partners and this was one of our management contracts. It was ultimately his decision with my strong recommendation to go to the hospital since he was still weak. The paramedics offered ten more minutes on site, but, he opted for the hospital.

We were young and dumb, now I'm not so young. Faced with the same situation today, I doubt I would do it again and I certainly wouldn't force an employee to do it.

I'm glad you replied to the comment. Heat exhaustion (leading to heatstroke) is serious stuff.

It really happens - even to young people at their peak condition. In dry climates, with mildly hot weather around 90-93 F.

Perhaps I'm stating the obvious, but awareness makes a big difference in these situations. It doesn't hurt to have a little EMT / Paramedic training too.

In fairness, that's not the whole story. When I sauna, I'm not doing anything remotely strenuous. I'm sitting around naked and not feeling the slightest stress. These guys were working in an emergency situation - clothed - and doing physical labor.

I agree: stress from handling emergency situation + physical labor definitely contributed.

But just 10 minutes of it still shouldn't have such serious effect.

Yet it did have that effect. Unless you're claiming the parent is lying? Or are you suggesting it's merely an unlikely event? The fact is people have all kinds of physical issues they might not even be aware of and you can't take that chance with people - not for legal reasons, not for ethical reasons. The best you can do is prepare in advance by training people in the situation to detect if they have any issues with the environment.

The air in a sauna is very very humid, right? And the air in a server room will typically be very very dry.

In a hot dry climate, you do lose water much faster.

I don't think that's right.

When overheated, you sweat. Sweat leads to evaporation, which leads to your skin cooling.

In humid environments, our sweat does not evaporate as well. This leads to less cooling, which leads to more sweating and thus more water loss.

Anyone that's been in a sauna can verify this. High humidity humidity feels hotter. You can demonstrate this by throwing water onto the sauna heater - the sauna will instantly feel a lot hotter.

Hmm... but what happens when you get dehydrated, which happens faster in a dry environment?

Yes: in dry climate you lose water much faster, so you cool off. So you should not faint that fast.

But may be it was mostly due to stress...

Are you actually claiming that the GP is lying? He's an eye witness, was instrumental in getting his buddy out. I'm pretty sure that if he felt his friend didn't need to be in the ambulance he would have omitted the bit about him passing out.

Not lying. Most likely some relevant details are unintentionally omitted. It could be that his buddy had poor health or was too overweight. Or may be was quite stressed out even before going into that hot facility. Or stayed there longer than 10 minutes. Or the temperature was higher. Or the air was slightly toxic (that's actually my best guess). Or all of the above.

But 10 minutes exposure to 60C dry air is not nearly enough for fainting.

Several hundred thousand people from the US have recent experience working in 120-140F weather wearing 50-100 pounds of equipment and heavy clothing, often doing labor a bit more strenuous than pulling drives or racking servers.

Disneyworld in Orlando Florida with an average summer temperature of 91F responds to dozens of heat stroke cases with people wearing shorts and shirts, merely walking around and waiting in lines to enjoy rides.

It is extremely humid in Orlando. Evaporative cooling through sweating doesn't work very well in those conditions.

Yeah, there's a wide variation in training, condition, and the level of care/equipment/hydration, and the results (and there are lots of heat injuries in the military, too). The OSHA limits are clearly not the limits of human performance (either low or high). One of the things is you acclimate to it over a period of a week or two fairly successfully, so if there were a datacenter which constantly operated in that range, you should be able to get staff who can handle it. (I'm sure steel mills and other industrial facilities have high temperatures, too).

If I read your comment correctly, it sounds like you're comparing healthy young people who have undergone extensive physical training and testing to technicians of a highly variable age who have gone through no such selection and training? Or are you really suggesting that any old technician can run around in a desert carrying military equipment with zero training or physical aptitude, or vice versa?

I thought he was suggesting hiring Iraq veterans to manage data centers.

The heat in Iraq goes up to 120F, and some soldiers have been hospitalized, or even killed by the heat. In the extreme case of 140F heat, I'd imagine soldiers would only be allowed out in if there's a very good reason. >120F heat should be treated like a sandstorm - an environmental hazard which you try to avoid.

Fairly certain he is talking about fire fighters folks.

>>> weather

I'm not.

I... don't see you participating in this conversation in the first place? So ok...

Well regardless, firefighters have plenty of experience with running into hot as balls buildings and doing physically exerting things. If he wasn't referring to firefighters, then he should have been.

I was definitely thinking of Iraq because I'd say national guard are a pretty broad cross section of regular Americans, but firefighters, steel mill or industrial workers, etc are all highly relevant too.

Relatively extreme temps are an issue, but can be overcome. If it requires a couple weeks of practice for your staff to get used to working in 100-120f low rh environments but saves huge amounts of energy, it is worth considering.

In a cloud hosting environment, you could eventually have non safe temperatures in active operation, lots of redundancy, and just do all your human maintenance during reduced power/temp windows all at once. Sort of like treating the data center as the inside of a computer. Either run multiple facilities or partition by thermal suite or hae the equipment power management be able to throttle cores for reducing heat.

When Ramadan is during the summer, observant Muslims in hot Middle Eastern countries still have to work, oftentimes in the heat, and fasting includes no water either. Maybe it's because they're used to it, but the point is, 140 degrees is certainly doable with water, even if it's not a lot of fun.

The hottest recorded temperature on earth was 136. Average highs in the Middle East aren't anywhere near 140.

He may not have been very eloquent, but he wasn't saying that Middle East temps reach 140.

(He was saying ME Muslims do x degrees without water, so 140 with water should be tolerable. X is anywhere from 90 to 120+ in certain ME countries).

fasting includes no water either

One of the strongest principles of Islamic law is that necessity overrides prohibition. If you're working outside in severe heat and becoming dehydrated, you're allowed to -- and arguably required to -- drink water, Ramadan or no Ramadan.

Of course, in "hot" data centers today the environment is no more dangerous than a nice sauna, but this development makes me imagine what it might progress to in 10 or 20 years. This is the first small step across that threshold between "comfortable for humans" and "comfortable for machines" in data centers, and the first such step is usually the hardest one. If it's more efficient to run the servers in an even hotter environment, or in one that's less-habitable in some other way, you can probably expect it to happen, now that the baseline technology and ideas are here.

Not that I'm trying to seriously forecast anything, or have any special knowledge of data center design; this is just a pleasant exercise in near-future sci-fi.

http://www.youtube.com/watch?v=xeJ0LLsr3sM

Chris Evans in Sunshine dies as he gets stuck under a computer tower which is being lowered back into its freezing cold fluid.

speaking as someone who has been inside a "hot" datacenter of a few thousand servers (after a failed ac system which didn't trip the alarms for a few hours), uncomfortable does not really begin to cover it. The extremely dry, hot air was quiet difficult to breathe, and being inside for more then a couple of minutes at a time was pretty much impossible. Definitely not the same experience as sitting in a hot aisle. It has been a number of years, but I believe the thermometers read close to 100F at the time.

If I'm not mistaken, humans can live in 100F dry air non-stop (with enough water to drink, of course). 100F is about human body temperature.

Could it be that hot datacenters have some poisonous gases evaporated into the air?

I wouldnt be worried about more than a few minutes. Millions of people live in climates where it gets above 100F for a significant part of the year. I used to go for hour long runs over lunch in Phoenix during the summer. Typically 100-107F. Dry/arid is actually preferable to humid ("yeah, but it's a dry heat").

I'm not saying that 100+F degree air is not livable, just relaying non-medical personal experience. I'm not doubting there were other factors, other then the heat, but it was more then just unpleasant. I've been to arizona in the summer (though I'll admit, I didn't go for long runs), but being inside that datacenter was considerably worse. Could have been the shock/drastic temperature difference though, I suppose.

Reply on news.ycombinator.com