Something is burning in the server room; how can I quickly identify what it is?

In most datacenters I've seen, I'd probably be willing to do a run through with IR cam/temp probe, or just visual inspection, with a handheld 1211, especially if I had a respirator, if it were just "smell of smoke". Clear view and path to two exits, someone at the EPO switch, etc.

The "big scary things" are battery plant and generator plant, and any kind of subfloor or ductwork. As long as the fire isn't in any of those, it's far less of a big deal. I probably wouldn't EPO a room for a server on fire, either -- just kill the rack, which takes slightly longer.

I've been in places where "smell of smoke" was a fucknozzle smoking a cigar or burning leaves outdoors outside an air intake, and another where it was a smoker's coat being put on an air handler.

Always plan for the single event because no amount of money will keep a single site running.

"Not being prepared for them is unforgivable" - would mean that 99% of business do not deserve forgiveness.

It just doesn't make sense to have that kind of redundancy for such a rare event for all but a very, very small minority of businesses. (Telecoms, Google, Stock Exchange, 911, etc...)

The fact is, there aren't that many flammable things in a data center. Nobody is going to die because they wandered up and down the aisles after they "thought they smelled something burning," in the absence of any visible smoke.

The guidelines in the highest-voted answer on the SO page make sense to me. 1: If you actually see smoke or fire in any significant amount, evacuate. 2: Make someone else aware of what's going on before doing anything else. 3: Keep your escape options open. 4: Think about how much time you can safely spend "guessing", and don't exceed it. 5: Don't second-guess your own common sense. You aren't paid to be a fireman or a hero.

The article says "smell", not "smoke", but I don't think inhaling it was a good idea in any event.

To re-iterate, a lot (most?) of the people who die in fires actually die from smoke inhalation than from getting burned by the fire/flames.

You do not try to debug the fire. There are people who are good at not dying while trying to do that. You are not one of them.

You do not try to avoid Big Red Buttoning because your bosses are idiots and they might come down on your hard for it: while your bosses are probably idiots, the first thing they'll tell you about Big Red Buttons is that nobody has ever gotten fired for pressing the Big Red Button, because everyone agrees that Big Red Buttons exist to get pushed and you never want to not have it pushed because someone was worried about getting fired. Big Red Buttons are costly affairs, sure. That's why we have redundant systems, insurance, and other various things that suggest we're responsible professionals.

The problem with that is that you only get one shot with the agent (although everywhere I've seen has a gas/clean agent with dry-pipe water as the true final protective measure). If there's electricity still going in, and you use your one shot with the facility clean agent, you might not actually put out the fire, or it could re-ignite, and then you either have no fire protection or only water. The cost of filling a datacenter with water, especially one important to have gas and water backup in the first place, is huge, even relative to an EPO pull.

If it's an "OK situation" they will be able to talk you through what needs to be checked, and they'll likely know it better than you if you don't have a proper in-case-of checklist prepared before hand.

If the smell turns into something worse, then you'll get them faster on-site.

In a "real" datacenter, you should have smoke sensors which would map where heat/smoke is coming from (since you have controlled airflow, it should be obvious which rack or small group of racks was the source -- it doesn't just exhaust into the whole room). But it's pretty clear this wasn't a "real" datacenter by their lack of protocols for handling fire, it was some office server thing.

You can get the same guts for $15 (http://www.amazon.com/Accuracy-Non-Contact-Infared-Temperatu...)

It tells you the surface temperature of whatever you point it at, and has a convenient laser for aiming. When cooking, I use it to see if a pain is hot enough yet (e.g. to sear meat), rather than relying on the "smoke point of various oils" test. You can also use it to see how close water is to a boil, although I'm not sure if it is measuring surface temperature, some slight penetration into the water, or the pan bottom (although, arguably, these should be fairly close in water).

It's also useful in something like a fusebox to find hot/overloaded circuits. It's essentially a 1x1 pixel themal imager, while a 100x100 thermal imager costs much more.

You still want a probe thermometer (for measuring meat internal temperature) such as http://amzn.com/B0000CF5MT, and if you have kids/sick people/etc., probably the internal-temperature kind (the ear/IR kind are the least gross).

Dropping a whole server room without seeing any smoke or fire is silly. Do you pull a fire alarm if someone smokes a cigarette indoors?

I made my 12 year-old read the story (along with pictures) of a girl his age who was trapped in a burning house after he set off the smoke alarms at midnight by melting bits of plastic in his room.

Hopefully he learned something that night.