If there was a comedy tour for IT/Programmer types, I'd pay to see you guys in it.
Best thing about your stuff is that it's literally all funny precisely because it's all true.
Well I was in the middle of that. When the Director decided to show off the new security doors. So he closed the room up. Then found out that new security doors didn’t work. I find out as I’m coming back to turn AC back on. Room will get hot really fast.
We get office Security to unlock door. He says he doesn’t have authority. His supervisor will be by later in the day.
Completely deadpan, and in front of several VPs of a forth one 50.
I turn to guy to my right who lived nearby. “Go home and get your chainsaw”
We were quickly let in. Also got fast approval to install proper cooling.
Fire extinguisher nearby, smart temp sensors, but still...
I have extinguishers all over the house, but hadn't considered a temperature sensor set to send alerts.
Do you have any recommendations?
Hell, even in DC you can look at temperatures and see in front of which server technican was standing just by those sensors.
Second cheapest would be USB-to-1wire module + some DS18B20 1-sire sensors. Easy hobby job to make. They also come with unique ID which means if you put it in TSDB by that ID it doesn't matter where you plug those sensors.
Hell, some weeks ago my oven decided the lower heater's line is now connected to ground and blows fuses...
Our 8 racks in DC only had single event of something blowing (power supply) and aside of smell and fuse blowing nothing really happened
Servers are essentially metal boxes with a bit of glass-reinforced epoxy and some plastic inside so there is limited amount of stuff that can burn. UPS is probably bigger problem
For example the OVH datacenter fire was more of "stuff around servers was flammable" (they had wooden ceilings for some reason...) rather than "just" servers.
I deal with Scientists that think AWS is some sort of a massively expensive enterprise thing. I can be, but not for the use case they're going to be embarking on. Our budget is $7M spanning 4 years.
Compared to using dedicated instances with way cheaper bandwidth, storage and compute power, it might as well be.
Cloud makes sense when you have to scale up/down very quickly, or you'd be losing money fast. But most don't suffer from this problem.
My PhD involved quite a lot of high performance computing simulations. We kept running into problems where on warm days, all our jobs would get killed pretty consistently. Our IT guy noticed a pattern where the temps would be perfectly normal, and then suddenly out of nowhere go through the roof, triggering a thermal shutdown of the racks.
In the end our IT guy camped out in the server room on a hot day to watch what was happening. The Astrophysics' cabinet were directly infront of ours, and they had jerry rigged their cabinet door so that when it got too hot, it would swing open and their hot air would be blown all over the neighbouring cabinets...
Love how you’ve fuzzed the root cause to make it seem like the “dollar store strip” is the problem and not that it was plugged into an overloaded outlet or run inside a closet at significantly elevated temperatures, leading to plastic melting and wires shorting.
Always helps to keep the “magic” a secret so the rubes have to keep us wizards employed, right?