https://perspectives.mvdirona.com/2013/02/the-power-failure-...
https://perspectives.mvdirona.com/2013/02/the-power-failure-...
Wow!
Funnily enough I believe the SR-71 has the opposite problem, where the engines would merrily go above Mach 3.5 with no issue but the airframe and everything attached to it would get torn apart by the intense heat if the pilot tried to go any faster
Additionally, there's the Mach Cone. The shock wave off the nose forms a cone shape, with the angle determined by the speed of the aircraft. From the nose to the wingtip forms an angle of about 17.5 degrees, which corresponds well with the max speed from the CIT of Mach 3.3.
That both methods of determining max speed agree shouldn't be surprising. Skunk Works was filled with good engineers.
The airframe, however, will get bent from temperature and you better hope the missile cooling loops don't fail.
The power isn't dialed down at lower speeds in order to provide thrust for maneuvering and climb.
The boost controller has a "scramble" button which overrides the preset boost targets and allows you to run a higher (and unsafe) boost level as long as the button is pressed.
This is useful when you're in the middle of a $10,000 race and you're losing by a few feet.
The "scramble" button I'm talking about is for "boost", which is the amount of positive air pressure inside the intake manifold.
Both "boost" and nitrous can dramatically increase power by essentially getting more oxygen into the combustion chamber, but nitrous is a much more dramatic effect.
This mode is pretty much reserved for go around or post V1 takeoff when one engine on twin engined airliner had failed.
Looks like there's a drop in replacement for Halon.
Older systems, before Halon was invented, are typically CO2 or Argon. They require a lot more agent to accomplish the job, but they're very simple.
I worked briefly in a CO2 datacenter. It dated from the mainframe era, and had some gorgeous mementos on the walls. There were SCBA packs in the hallway outside, and everyone got a brief safety lecture before being allowed into the facility. The point I remember was basically "The alarm will sound intermittently for twenty seconds, then go solid for five, before discharge. If it goes off, do not stop to save your files, just run for the door. If you're not outside yet when the solid tone sounds, hold your breath and run on what's in your lungs."
Apparently the CO2 is considerably nastier than Halon to get a lung-full of...
- The Stratus supposedly had fault tolerant duplexing of boards - pull one out and the other keeps working. A famous demo where local consultant pulls out the only board which wasn't duplicated!
- The machine room was also fire proofed with Halon gas (if I recall correctly), but due to security concerns the door was locked from inside by operators when present (luckily as programmers we weren't expected to work inside with doors locked!). Luckily also, it was my first exposure to "paired programming"/"paired operating" as in there were invariably 2 of them.
- A couple of the operators became too interested in how to work things, and wrote a quite comprehensive manual. They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves".
- the whole headquarters building was in a guarded compound, where ultimate protection against "red brigade terrorists" (https://en.wikipedia.org/wiki/Red_Brigades) were the armed guards. We used to meet them in the ground floor cafe every mid-morning, with a revolver on their hip, while they drank an espresso with brandy chaser. Really inspired confidence that they would be there to protect us should the need arise!
And this is why we cannot have nice IT security practices.
In a datacenter though your bets are usually much lower. Nobody is going to die, or lose many millions, if some of your equipment shuts down to prevent it from being damaged, catching a fire, etc.
There are though high-stake situations where you want exactly that: spend the entire amount of the resources of certain hardware to prevent a loss of life, or of untold millions, when you are powering a surgery chamber, or a large stock trading operation.
People who realize that do over-provision and pay top dollar for that when they can afford that. I remember that when a major fire occurred in one of the skyscrapers in the financial district of NYC, traders of a particular financial company were evacuated with their laptops into helicopters on the roof, and ferried to a spare office across Hudson river, in NJ. To minimize the impact of that, they connected to the corporate network via their phones as hotspots, and kept trading while airborne.
Of course one cannot hope to pull such an operation off without extensive preparation and likely regular drills.
Not preparing to a black swan event during a high-stakes event, like translation of a Superbowl match, sounds like either not having enough paranoia which is professionally required, or, more likely, as a cost-cutting after a wrong assessment of risks.
(2) Think about hiring twice as much senior programmes, just in case. Good day traders are even more expensive.
Day traders are expensive? I thought a lot of those folks keep what they kill, and when they screw up they don't get to eat. "Coffee for closers," etc.
If they trade for a large corporation, with the leverage provided by that corporation, it's a whole another ballgame.
> According to Allied Ordnance Publication AOP-38-3,[1] a NATO publication, a battleshort is "The capability to bypass certain safety features in a system to ensure completion of the mission without interruption due to the safety feature." It also says, "Examples of bypassed safety features are circuit overload protection, and protection against overheating".
> For example, the electrical drives to elevate and traverse the guns of a combat warship may have "battleshort" fuses, which are simply copper bars of the correct size to fit the fuse holders, as failure to return fire in a combat situation is a greater threat to the ship and crew than damaging or overheating the electrical motors.
> Battleshorts have been used in some non-combat situations as well, including the Firing Room/Mission Control spaces at NASA during the manned Apollo missions — specifically the Moon landings.
As a side note, with great apology to those who may have lost their lives due to the consequences of real-world battleshorts - this is actually a really interesting analogy for the "war room" scenarios we find ourselves in at startups. Risking damage to morale and productivity can be acceptable if the alternative is irrecoverable loss of the startup's reputation. But this also can't be a sustained state of affairs - these types of procedures are meant for a battle, and risk compounds if you don't return to a steady state. Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.
Thank you for this, I'll be updating our documentation to include these concepts first thing tomorrow.
... And 6 months later marketing people will have commandeered it, without regard to the original meaning, and they’ll be asking people to battleshort so they can really be agile and lean in on providing air cover via the updated sales deck.
We're now sharpening pitchforks to use on the next salestrooper that comes close to the SalesBell™
/s
Doubly so because I used to be in the US military but now live in a different country, and it's crazy to see how those terms and acronyms infect a lot of US business culture.
A decade ago I would have been fine, even glad, that I could wrap myself in verbiage that would jive with poor career choices I made at 18. Now it just feels incongruous or jingoistic.
The owner of the datacenter's aversion to downtime is not going to be a concern to the building inspector or the fire marshal.
At one of my previous workplaces I learned a bit about our local firecode in relation to electrical code. Mostly that firecode supersedes electrical code. The fire alarm is running of a 230V line (standard) without grounding (non-standard) via a special line from the upstream power transformer that has higher amperage tolerance.
All the fire detectors are powered and wired from this at 40-70V, no ground or if there is a ground, it's grounded at the control panel of the fire alarm system.
In the event of a fire, the entire system is rated about 10X over it's standard ratings. The wires that are allowed to handle 1 Amp before are now allowed to go to 10 Amps, just in case there is a short this might burn it out and allow other equipment to operate.
I've seen some systems that are "optionally fused", they have an internal relais that's held by external power and if the fused lines fail, it automatically switches to unfused power, where the alarm system can evaluate the condition and decide to go back to the fused line if power returns. That's on top of having it's own backup batteries. Some of the more modern systems have watchdogs systems built in so that in case the fire alarm computer is on fire, it won't switch to the unfused line once the computer is no longer operational.
At to amperage ratings, don't you just mean that they have a 10x safety factor? Probably using current limiting supplies that can handle a short circuit. Fire circuits (and the feed from the main panel) should use fire rated cabling -- generally mineral insulated copper sheathed cables. Most of the standards (NFPA 75/76) are moving away from being prescriptive to being risk or performance based though, so it is possible they used normal conduit if the overall risk was small.
Incidentally, hospitals also make extensive use of isolation transformers for wet areas (like emergency / ICU), to provide a local point for resetting breakers (electronically), identifying which point is tripping, etc, see IEEE 602-2007. Much better filtering for medical equipment too. You also tend to see positive retention plugs (in the US, green dot UL 817).
The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline).
Test components individually - at the factory and in the field - and test systems end-to-end.
Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope.
When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system?
Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails".
If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?
http://resistive-loadbank.sell.everychina.com/p-109417602-35... is the first example I pulled up, and there are bigger ones.
I don't have actual knowledge of nuclear power plants, that is classified beyond the public. However hospitals do the above, because if thing dont work on a bright sunny day how do you think they will work when a tornado has not only taken out power, but also filled the er with server trauma cases.
Disable the turbine. Kill AC power (turn off all the rectifiers and HVAC) and let the office run on battery for a while. Closely monitor the batteries, and the temperature in critical areas.
After a good while on battery (an hour or two, if I recall), walk leisurely over to the turbine and enable it. Let it start and warm up, then transfer the rectifiers and HVAC over to generator power.
Closely monitor the turbine while the batteries recharge, the HVAC while on unusual power, and the humans who are really glad to have air conditioning back.
When satisfied that all is copacetic, transfer things back to utility AC, shut down the turbine, and call the fuel company to top off the tank.
Most of the batteries in these offices were between 10 and 20 years old, with a couple instances of the beautiful cylindrical "Bell Cells" still in service. They were sized to not break a sweat running the whole building, and impeccably maintained, so they didn't tend to fail prematurely.
Once a quarter we'd do a live load transfer, and watch all the PDUs and UPS systems do their thing. Those could be stressful...
Diesel generators for hospitals and water pumps for fire suppression systems are generally set up with very loose settings as it is clearly worse for a patient to die or a building to burn down than to destroy a machine, and this requirement is in the specifications.
Those systems also require regular testing by qualified personnel. NFPA 25 requires monthly testing of fire pumps and annual flow testing. For low rise buildings with inadequate municipal water pressure, a water tower might be cheaper and easier. NFPA 99 - Standard for health care facilities, requires emergency generators to be tested 12 times per year.
I’ve built and programmed control systems that do all this with multiple generators, multiple utility feeds, and multiple tiers of loads of different priorities.
You can test the system’s ability to detect loss of power or poor quality power by lifting sensing wires or injecting out of frequency or voltage range power with a test set, and load pickup and load testing we used a load bank. The actual hospital load is fed through bypass breakers during these tests so if the utility went out we would have to manually operate breakers to disconnect the load bank and swing the vital loads over to the generators.
If I am setting up electrical protection on a generator I don’t put a short circuit on the generator, I show the protection relay what it would see by injecting current and voltage with a relay test set and make sure the relay gives the signal to open the breaker when it should.
Greenfield hospital generation sites are straightforward to open the utility feeds since the vital loads are still fed through the old power system.
In the end you WILL test what happens when the hospital plunges into darkness. Would you prefer the first test to be a a bright sunny day when all the staff is ready for something (and you can turn the utility power back on quickly if something fails), or when someone with a backhoe/chainsaw has an accident with your power feed. The latter will happen, I don't know how or when, but at some point in your future trees will take down power lines, backhoes will cut something, major weather events will take out power for days, maybe a large blackout...
You should have course test everything separately often. However if you wouldn't let a random electrician (electrician only because they will know how to not kill themselves) turn off your utility connection with only a day of warning, your system isn't trusted to handle real events which are typically much worse than a clean turn off the switch.
FYI where I’ve worked the tiers of loads are metered individually and picked up in order of priority and with a check to verify the combined capacity of the generators which are online is great enough to pick up the load. Likewise loads are shed in order if there isn’t enough generation such as one generator tripping.
I don’t really see that much difference between opening a breaker and bad quality power. It is the same sensing and logic in both cases: is the magnitude and frequency of the phase to phase voltages in the acceptable range or not?
Years later I had a tech support job in that hospital and all the PCs had small UPSs to keep them alive for the 30-60 seconds it took the big generators to come online.
I never did get a chance to see the generator room.
e: Another personal anecdote that is more relevant to the topic.
I was working for a VOIP provider back when Hurricane Sandy hit NYC. Part of our production network was housed in a telecom building in Manhattan. We received daily updates on how many floors the flooding was from our equipment and about the fuel deliveries to the building backup generators. Apparently the generators were on the roof? I'd love to know more about the setup in that building because we had zero downtime.
The blog where they posted regular updates is still up; here's the first post related to the storm: https://interdictor.livejournal.com/2005/08/27/
"Hmm. This could actually be a nasty storm."
Very good read!
Run a combined-cycle natural gas generator 24/7, and use the grid as your backup. Gets good efficiency. Runs loops on the coolant, oil and exhaust for water/building heat, and can switch over instantly in the event of failure.
https://www.solidwastemag.com/feature/talking-bout-cogenerat...
Can't you just flip the circuit breaker or just pull the plug out? Or does a real power outage usually look different to that in some way, like a surge followed by a shutoff?
Initially, there's a fault, say, a tree contacting a wire. This conducts a bunch of current to ground, the voltage in part of the network sags a fair bit, and a protective relay senses the overcurrent and opens the circuit. This happens in a second or two, and your power goes out. You may notice lights dimming or flickering in the instant beforehand.
But, many faults are transient. Perhaps that tree branch finished falling, and is no longer against the wire. So the protective relay performs a "reclose" operation, where it turns the power back on for a moment and measures the current. There's normally a huge starting inrush as motor-driven appliances restart, so it waits a moment before sampling the current, and this is the period when the power comes back on for a few seconds. (It's probably pretty nasty power during this interval, too.)
If things are good, hey, you're good! A brief outage while waiting for the fault to clear itself, and then everything's back. The operation will be reported to the dispatch center, and someone may come out to inspect the area in case there's damage.
But if the fault is still there, and after a moment the current is still way higher than it should be, the recloser opens again, and this time it stays open. Gonna need a crew to move that tree, and then they'll manually try again.
The customer was running the generators right at the maximum capacity, and we'd see really, really odd things on the power lines. Some times a surge like you mention, sometimes changes to the frequency, sometimes changes to the voltage. We discovered all sorts of weird brownout conditions in our devices.
In our small data centre we have a board with five lights and a key. You turn the key and the mains is cut off. The lights give the status. The UPSs beep for a few seconds "on battery", the genny fires up in the boiler room and when it stabilises, it takes the load. We leave it running for 5-15 minutes, go out and check the fuel levels and top up accordingly. It's diesel so safe to hot refuel. When done, turn the key back and then the UPSs take the load again and then hand back to the mains. The genny runs for another 15 minutes and then shuts down. That should avoid flip flopping.
You then fill in the test form on our wiki and job done. We do it weekly to fortnightly. An ESP8266 based thingie reports status back to the central monitoring (GPIOs and ESPHome -> Home Assistant)
You could just, you know, pull the plug and see what happens?