That UPS you bought for your home server may not be as useful as you think
fitzcarraldoblog.wordpress.com
fitzcarraldoblog.wordpress.com
https://perspectives.mvdirona.com/2013/02/the-power-failure-...
Diesel generators for hospitals and water pumps for fire suppression systems are generally set up with very loose settings as it is clearly worse for a patient to die or a building to burn down than to destroy a machine, and this requirement is in the specifications.
Those systems also require regular testing by qualified personnel. NFPA 25 requires monthly testing of fire pumps and annual flow testing. For low rise buildings with inadequate municipal water pressure, a water tower might be cheaper and easier. NFPA 99 - Standard for health care facilities, requires emergency generators to be tested 12 times per year.
I’ve built and programmed control systems that do all this with multiple generators, multiple utility feeds, and multiple tiers of loads of different priorities.
You can test the system’s ability to detect loss of power or poor quality power by lifting sensing wires or injecting out of frequency or voltage range power with a test set, and load pickup and load testing we used a load bank. The actual hospital load is fed through bypass breakers during these tests so if the utility went out we would have to manually operate breakers to disconnect the load bank and swing the vital loads over to the generators.
If I am setting up electrical protection on a generator I don’t put a short circuit on the generator, I show the protection relay what it would see by injecting current and voltage with a relay test set and make sure the relay gives the signal to open the breaker when it should.
Greenfield hospital generation sites are straightforward to open the utility feeds since the vital loads are still fed through the old power system.
In the end you WILL test what happens when the hospital plunges into darkness. Would you prefer the first test to be a a bright sunny day when all the staff is ready for something (and you can turn the utility power back on quickly if something fails), or when someone with a backhoe/chainsaw has an accident with your power feed. The latter will happen, I don't know how or when, but at some point in your future trees will take down power lines, backhoes will cut something, major weather events will take out power for days, maybe a large blackout...
You should have course test everything separately often. However if you wouldn't let a random electrician (electrician only because they will know how to not kill themselves) turn off your utility connection with only a day of warning, your system isn't trusted to handle real events which are typically much worse than a clean turn off the switch.
FYI where I’ve worked the tiers of loads are metered individually and picked up in order of priority and with a check to verify the combined capacity of the generators which are online is great enough to pick up the load. Likewise loads are shed in order if there isn’t enough generation such as one generator tripping.
I don’t really see that much difference between opening a breaker and bad quality power. It is the same sensing and logic in both cases: is the magnitude and frequency of the phase to phase voltages in the acceptable range or not?
Wow!
Funnily enough I believe the SR-71 has the opposite problem, where the engines would merrily go above Mach 3.5 with no issue but the airframe and everything attached to it would get torn apart by the intense heat if the pilot tried to go any faster
Additionally, there's the Mach Cone. The shock wave off the nose forms a cone shape, with the angle determined by the speed of the aircraft. From the nose to the wingtip forms an angle of about 17.5 degrees, which corresponds well with the max speed from the CIT of Mach 3.3.
That both methods of determining max speed agree shouldn't be surprising. Skunk Works was filled with good engineers.
The airframe, however, will get bent from temperature and you better hope the missile cooling loops don't fail.
The power isn't dialed down at lower speeds in order to provide thrust for maneuvering and climb.
The boost controller has a "scramble" button which overrides the preset boost targets and allows you to run a higher (and unsafe) boost level as long as the button is pressed.
This is useful when you're in the middle of a $10,000 race and you're losing by a few feet.
The "scramble" button I'm talking about is for "boost", which is the amount of positive air pressure inside the intake manifold.
Both "boost" and nitrous can dramatically increase power by essentially getting more oxygen into the combustion chamber, but nitrous is a much more dramatic effect.
This mode is pretty much reserved for go around or post V1 takeoff when one engine on twin engined airliner had failed.
Looks like there's a drop in replacement for Halon.
Older systems, before Halon was invented, are typically CO2 or Argon. They require a lot more agent to accomplish the job, but they're very simple.
I worked briefly in a CO2 datacenter. It dated from the mainframe era, and had some gorgeous mementos on the walls. There were SCBA packs in the hallway outside, and everyone got a brief safety lecture before being allowed into the facility. The point I remember was basically "The alarm will sound intermittently for twenty seconds, then go solid for five, before discharge. If it goes off, do not stop to save your files, just run for the door. If you're not outside yet when the solid tone sounds, hold your breath and run on what's in your lungs."
Apparently the CO2 is considerably nastier than Halon to get a lung-full of...
- The Stratus supposedly had fault tolerant duplexing of boards - pull one out and the other keeps working. A famous demo where local consultant pulls out the only board which wasn't duplicated!
- The machine room was also fire proofed with Halon gas (if I recall correctly), but due to security concerns the door was locked from inside by operators when present (luckily as programmers we weren't expected to work inside with doors locked!). Luckily also, it was my first exposure to "paired programming"/"paired operating" as in there were invariably 2 of them.
- A couple of the operators became too interested in how to work things, and wrote a quite comprehensive manual. They were "moved on" within the bank, because for security, management only wanted "people who could follow instructions", not "who can think for themselves".
- the whole headquarters building was in a guarded compound, where ultimate protection against "red brigade terrorists" (https://en.wikipedia.org/wiki/Red_Brigades) were the armed guards. We used to meet them in the ground floor cafe every mid-morning, with a revolver on their hip, while they drank an espresso with brandy chaser. Really inspired confidence that they would be there to protect us should the need arise!
And this is why we cannot have nice IT security practices.
In a datacenter though your bets are usually much lower. Nobody is going to die, or lose many millions, if some of your equipment shuts down to prevent it from being damaged, catching a fire, etc.
There are though high-stake situations where you want exactly that: spend the entire amount of the resources of certain hardware to prevent a loss of life, or of untold millions, when you are powering a surgery chamber, or a large stock trading operation.
People who realize that do over-provision and pay top dollar for that when they can afford that. I remember that when a major fire occurred in one of the skyscrapers in the financial district of NYC, traders of a particular financial company were evacuated with their laptops into helicopters on the roof, and ferried to a spare office across Hudson river, in NJ. To minimize the impact of that, they connected to the corporate network via their phones as hotspots, and kept trading while airborne.
Of course one cannot hope to pull such an operation off without extensive preparation and likely regular drills.
Not preparing to a black swan event during a high-stakes event, like translation of a Superbowl match, sounds like either not having enough paranoia which is professionally required, or, more likely, as a cost-cutting after a wrong assessment of risks.
(2) Think about hiring twice as much senior programmes, just in case. Good day traders are even more expensive.
Day traders are expensive? I thought a lot of those folks keep what they kill, and when they screw up they don't get to eat. "Coffee for closers," etc.
If they trade for a large corporation, with the leverage provided by that corporation, it's a whole another ballgame.
> According to Allied Ordnance Publication AOP-38-3,[1] a NATO publication, a battleshort is "The capability to bypass certain safety features in a system to ensure completion of the mission without interruption due to the safety feature." It also says, "Examples of bypassed safety features are circuit overload protection, and protection against overheating".
> For example, the electrical drives to elevate and traverse the guns of a combat warship may have "battleshort" fuses, which are simply copper bars of the correct size to fit the fuse holders, as failure to return fire in a combat situation is a greater threat to the ship and crew than damaging or overheating the electrical motors.
> Battleshorts have been used in some non-combat situations as well, including the Firing Room/Mission Control spaces at NASA during the manned Apollo missions — specifically the Moon landings.
As a side note, with great apology to those who may have lost their lives due to the consequences of real-world battleshorts - this is actually a really interesting analogy for the "war room" scenarios we find ourselves in at startups. Risking damage to morale and productivity can be acceptable if the alternative is irrecoverable loss of the startup's reputation. But this also can't be a sustained state of affairs - these types of procedures are meant for a battle, and risk compounds if you don't return to a steady state. Perhaps adding "battleshort" to our lexicon would make it clear that "doing things that don't scale" is often necessary but not without its costs.
Thank you for this, I'll be updating our documentation to include these concepts first thing tomorrow.
... And 6 months later marketing people will have commandeered it, without regard to the original meaning, and they’ll be asking people to battleshort so they can really be agile and lean in on providing air cover via the updated sales deck.
We're now sharpening pitchforks to use on the next salestrooper that comes close to the SalesBell™
/s
Doubly so because I used to be in the US military but now live in a different country, and it's crazy to see how those terms and acronyms infect a lot of US business culture.
A decade ago I would have been fine, even glad, that I could wrap myself in verbiage that would jive with poor career choices I made at 18. Now it just feels incongruous or jingoistic.
The owner of the datacenter's aversion to downtime is not going to be a concern to the building inspector or the fire marshal.
At one of my previous workplaces I learned a bit about our local firecode in relation to electrical code. Mostly that firecode supersedes electrical code. The fire alarm is running of a 230V line (standard) without grounding (non-standard) via a special line from the upstream power transformer that has higher amperage tolerance.
All the fire detectors are powered and wired from this at 40-70V, no ground or if there is a ground, it's grounded at the control panel of the fire alarm system.
In the event of a fire, the entire system is rated about 10X over it's standard ratings. The wires that are allowed to handle 1 Amp before are now allowed to go to 10 Amps, just in case there is a short this might burn it out and allow other equipment to operate.
I've seen some systems that are "optionally fused", they have an internal relais that's held by external power and if the fused lines fail, it automatically switches to unfused power, where the alarm system can evaluate the condition and decide to go back to the fused line if power returns. That's on top of having it's own backup batteries. Some of the more modern systems have watchdogs systems built in so that in case the fire alarm computer is on fire, it won't switch to the unfused line once the computer is no longer operational.
At to amperage ratings, don't you just mean that they have a 10x safety factor? Probably using current limiting supplies that can handle a short circuit. Fire circuits (and the feed from the main panel) should use fire rated cabling -- generally mineral insulated copper sheathed cables. Most of the standards (NFPA 75/76) are moving away from being prescriptive to being risk or performance based though, so it is possible they used normal conduit if the overall risk was small.
Incidentally, hospitals also make extensive use of isolation transformers for wet areas (like emergency / ICU), to provide a local point for resetting breakers (electronically), identifying which point is tripping, etc, see IEEE 602-2007. Much better filtering for medical equipment too. You also tend to see positive retention plugs (in the US, green dot UL 817).
The cardinal rule is - test everything. Test it when you install and commission it, test it when part of the system changes, test it periodically during maintenance shutdowns, test it when you have a convenient time to do so without losing production (such as an outage to a different system that necessitates your system being offline).
Test components individually - at the factory and in the field - and test systems end-to-end.
Test with the most adverse possible conditions, not the most optimal. Test beyond your normal operating envelope.
When I participate in design reviews as a maintenance engineer my primary line of inquiry is: how will I test this stuff during operations? Has it been designed in a way that makes testing impossible without an expensive outage? Can workers safely gain access to components that need routine testing without having to de-energize other, unrelated parts of the system?
Now not every industry has the budget available to test to the extent that we do. But even in our industry, I often see a penny-wise, pound-foolish approach of "assume this thing works perfectly from the factory then act surprised years later when an abnormal condition occurs and it fails".
If something was not tested, it may not work. It might be for a very simple reason (like mis-configuration of a single setting during installation) or a very complex reason. But either way, wouldn't the company rather know their equipment doesn't work -before- putting it into service than after?
http://resistive-loadbank.sell.everychina.com/p-109417602-35... is the first example I pulled up, and there are bigger ones.
I don't have actual knowledge of nuclear power plants, that is classified beyond the public. However hospitals do the above, because if thing dont work on a bright sunny day how do you think they will work when a tornado has not only taken out power, but also filled the er with server trauma cases.
Disable the turbine. Kill AC power (turn off all the rectifiers and HVAC) and let the office run on battery for a while. Closely monitor the batteries, and the temperature in critical areas.
After a good while on battery (an hour or two, if I recall), walk leisurely over to the turbine and enable it. Let it start and warm up, then transfer the rectifiers and HVAC over to generator power.
Closely monitor the turbine while the batteries recharge, the HVAC while on unusual power, and the humans who are really glad to have air conditioning back.
When satisfied that all is copacetic, transfer things back to utility AC, shut down the turbine, and call the fuel company to top off the tank.
Most of the batteries in these offices were between 10 and 20 years old, with a couple instances of the beautiful cylindrical "Bell Cells" still in service. They were sized to not break a sweat running the whole building, and impeccably maintained, so they didn't tend to fail prematurely.
Once a quarter we'd do a live load transfer, and watch all the PDUs and UPS systems do their thing. Those could be stressful...
You could just, you know, pull the plug and see what happens?
Years later I had a tech support job in that hospital and all the PCs had small UPSs to keep them alive for the 30-60 seconds it took the big generators to come online.
I never did get a chance to see the generator room.
e: Another personal anecdote that is more relevant to the topic.
I was working for a VOIP provider back when Hurricane Sandy hit NYC. Part of our production network was housed in a telecom building in Manhattan. We received daily updates on how many floors the flooding was from our equipment and about the fuel deliveries to the building backup generators. Apparently the generators were on the roof? I'd love to know more about the setup in that building because we had zero downtime.
The blog where they posted regular updates is still up; here's the first post related to the storm: https://interdictor.livejournal.com/2005/08/27/
"Hmm. This could actually be a nasty storm."
Very good read!
Run a combined-cycle natural gas generator 24/7, and use the grid as your backup. Gets good efficiency. Runs loops on the coolant, oil and exhaust for water/building heat, and can switch over instantly in the event of failure.
https://www.solidwastemag.com/feature/talking-bout-cogenerat...
Can't you just flip the circuit breaker or just pull the plug out? Or does a real power outage usually look different to that in some way, like a surge followed by a shutoff?
Initially, there's a fault, say, a tree contacting a wire. This conducts a bunch of current to ground, the voltage in part of the network sags a fair bit, and a protective relay senses the overcurrent and opens the circuit. This happens in a second or two, and your power goes out. You may notice lights dimming or flickering in the instant beforehand.
But, many faults are transient. Perhaps that tree branch finished falling, and is no longer against the wire. So the protective relay performs a "reclose" operation, where it turns the power back on for a moment and measures the current. There's normally a huge starting inrush as motor-driven appliances restart, so it waits a moment before sampling the current, and this is the period when the power comes back on for a few seconds. (It's probably pretty nasty power during this interval, too.)
If things are good, hey, you're good! A brief outage while waiting for the fault to clear itself, and then everything's back. The operation will be reported to the dispatch center, and someone may come out to inspect the area in case there's damage.
But if the fault is still there, and after a moment the current is still way higher than it should be, the recloser opens again, and this time it stays open. Gonna need a crew to move that tree, and then they'll manually try again.
The customer was running the generators right at the maximum capacity, and we'd see really, really odd things on the power lines. Some times a surge like you mention, sometimes changes to the frequency, sometimes changes to the voltage. We discovered all sorts of weird brownout conditions in our devices.
In our small data centre we have a board with five lights and a key. You turn the key and the mains is cut off. The lights give the status. The UPSs beep for a few seconds "on battery", the genny fires up in the boiler room and when it stabilises, it takes the load. We leave it running for 5-15 minutes, go out and check the fuel levels and top up accordingly. It's diesel so safe to hot refuel. When done, turn the key back and then the UPSs take the load again and then hand back to the mains. The genny runs for another 15 minutes and then shuts down. That should avoid flip flopping.
You then fill in the test form on our wiki and job done. We do it weekly to fortnightly. An ESP8266 based thingie reports status back to the central monitoring (GPIOs and ESPHome -> Home Assistant)
Most tasks are running scrapers or data wrangling so I don't have to keep my desktop running.
I've been doing this for maybe close to two years, and for now it works. I wouldn't recommend for VMs though.
Also, your router needs an UPS too but it will be fine with a cheap one.
Why not? I have a laptop with a 4th gen i5 that has VT-x. I've upgraded it to 16 GB RAM and an SSD. For now, it's a Spotify connect player but I'm contemplating running a couple VMs on it too, like pi-hole and whatnot.
Sure, that’s still reasonable, but it’s not young.
I’m also using similar class hardware for my VM server at home. Works well enough for home workloads, and for everything else there’s the cloud on demand.
I wouldn't run VMs on it, but I do run docker on it, and that works just fine.
Also, RDP. I still hope for something so smooth in Linux, using console meanwhile.
I did some tests on a RPI3 a year or 2 ago and showing some scrolling test in a UWP app was making the whole machine lag and be unresponsive.
I've also imaged a version of that SD card, so it would be easy to just swap out and probably get things running again without much hassle.
I'm sure there are more fancy solutions with overlayfs, making the SD card read-only except for upgrades. Though I couldn't find a nice resource on it at the time.
It has only been down when the power has been out, and so far no issues with the card. It is however an A1 or A2 class card (I forget which), so supposedly design with applications in mind.
It should also be noted that a lot of issues with the Pi is due to poor USB cables / power supplies. I had another Pi that kept crashing every few days, until I measured the USB "charger" cable I was using for it. Turns out it had a resistance of almost exactly 1 Ohm. So if the Pi drew say 1A, that would be 1V loss in the cable... After I swapped cables it has been rock solid. A key point here is that a lot of SD cards do not like losing power while being written to. Avoid that and a quality SD card should have a long life.
Any suggestions? Though I'm a software engineer, I've worked mostly in app dev rather than infrastructure setup. This project is a way for me to learn more about the latter. The rpi is the easiest and cheapest option, but I have wondered if it wouldn't make more sense to use AWS or something like that. I wouldn't mind paying a few bucks a month, but more than that and I'd rather wait until I've learned a bit and know what my specific needs are. I've also considered an actual computer, but figure it's a waste of power when my initial work will probably just be figuring out the server config, HTML pages, etc.
You could host everything on your home network but there are a couple of things you should probably know before you start: 1) Some ISP do not give you static IP, meaning its extra work
2) Security - Ideally your home network has some kind of segregation, you keep your trusted devices on one, IoT or not patchable on another for example, with firewall blocking connections from dirty network to clean one. Having externally facing website potentially having direct access to personal devices and services might be an issue from security point of view.
You can sign up on AWS/Azure or any other cloud provider for free and get free credits, which might be enough for you to learn. Feel free to ask further questions if you need to
I have a running rpi for more than half a year. No issues whatsoever. But again - I have a very stable electricity and I dont remember last time it was cut not due to planned maintenance.
1) I think the pi is not particularly power efficient.
I got this impression from this (old) article:
https://www.bitwizard.nl/wiki/Reducing_power_consumption_of_...
2) I think the pi would benefit from a more robust filesytem layout such as an overlay filesystem to allow continuous writes to /var to go to ramdisk.
Openwrt has a layout like this and it prevents flash from being burned out and prevents most problems if power is abruptly lost.
While there might be efficiency gains available, RPIs use a very small amount of power compared to a laptop. Idling a RPI4 uses 2.8 watts[0], which not only rates favorably compared to most laptops, but is far below what my charging cell phone uses. Maxed out my RPI can only hit 15W, since that's the maximum power that my official USB C adapter can output.
Now that being said, it does depend on what kind of load you're expecting to handle. If you're constantly maxing out a RPI, it is probably more energy efficient to purchase a larger server than to just keep adding RPIs. But if you're staying well below the theoretical max of a RPI, it'll consume far less than a used laptop.
> I think the pi would benefit from a more robust filesytem layout such as an overlay filesystem to allow continuous writes to /var to go to ramdisk.
This is one of the biggest drawbacks of the RPI; microSD cards cannot handle a ton of writes without frying. I've settled on using log2ram to ensure that logs are only periodically flushed from RAM to the SD card, to extend the lifespan of my cards.
Watts per low traffic website it's very efficient. CPU is generally not pegged at 100%.
For watts per CPU cycle not as much, I agree. Say it takes 10W at full power (I've not measured). That makes it 1/10th of a normal CPU. But it's like 30-40 times slower.
They really think their prosumer workstation is going to give them an advantage.
I guess thats one silver lining about Robinhood’s proliferation, many people know its good enough. Gullible traders still misread options settlement UI but seems there is a selective evolution at play to make that less common too.
[1] Quite often, I couldn’t ssh into the pi for all kinds of reasons. With a laptop, I don’t have to worry about that.
Have you any suggestions for increasing the amount of storage one could address? I can't find any Thunderbolt 2 or FireWire enclosures so I'm sadly limited to the two USB3.0 ports. Right know my plan is to resort to putting a high-capacity 2.5" hdd inside the MBP and adding a usb hdd in addition to the one I already have. But that doesn't strike me as ideal for some reason.
USB 3 is fast enough for a hard drive.
Relatively recent smartphones also work great for a lot of stuff.
Then the next level of slipping down the chain is the headless server or timelapse photo taker. :)
Refurbished with a Windows license (if I ever want to use it) was less than a surplus laptop. Power usage seems acceptable but I'll compare with a Kill-A-Watt. I have one on the way and if it works out I'll grab a few more.
I do like the idea of a surplus laptop cluster, mostly because of the built in UPS factor but there's a lot of wasted hardware there, such as the monitor and keyboard. But I suppose that's kind of a feature too.
I have shower thoughts of some kind of custom bladecenter made of old laptops but that's a lot of work for probably zero benefit.
Can you post a link to those 7 year old micro towers, please?
The tiny tower I have on the way is one of these: https://www.lenovo.com/us/en/desktops/thinkcentre/m-series-t...
They’re available in a bunch of configurations and on multiple sites so you can look around for a good deal that fits your needs. I found one with an SSD for additional power savings.
In regards of power efficiency I'd still be drooling over one of the Xeon D-1600 configurations but haven't found one that's not in a rack form factor yet.
https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E...
https://www.supermicro.com/en/products/system/Mini-ITX/SYS-E...
Also available with AMD Epyc Embedded (35W for 8 cores).
Broadwell NUCs are available on eBay and Newegg has a matching Akasa Plato X case for $100. With an SSD, that has no moving parts. But only dual core, single NIC.
In most cases the beefiest air cooler + the lowest tolerable fan curve (meaning no overheating) will do the job just fine, especially if the CPU TDP is between 35-65W. Alternatively you can also limit the CPU power usage by disabling turbo or doing other tricks that forces it to run slower, but more efficiently.
Some fanless cases here for different NUCs: https://www.neweggbusiness.com/product/productlist.aspx?subm...
Plex can use nVidia cards for hardware acceleration for encoding.
These features do require an active Plex Pass.
https://support.plex.tv/articles/115002178853-using-hardware...
Replace the fan with a Noctua-type quiet model bolted on. If the HSF needs that airflow directed, hack a simple shroud (or 3D print a replacement.)
I run a uSFF HP box for my hypervisor and a low power J5005 core in my NAS and am more than happy with the performance let alone minimal power usage. Sure the J5005 system takes ages to update requiring a backup DC/PiHole setup on the HyperVisor but it just works at low cost.
I thought I should replace my 2012 Mac Mini (HTPC) with a modern Hackintosh and 6c/12t or 8c/16t and do my testing workloads on that (or even just use my current 2c/4t for small workloads). But it's just so easy to spin up DO droplets within seconds and do so programmatically. Plus you can snapshot and create droplets from those, and turn on backups if you need to.
Here is how I differentiate the APC brand:
* Back-UPS: inverter offline, expect a 15-20ms power transfer
* Smart-UPS: interactive, inverter is always online 2-5ms trasnfter
* Symmetra and above: inverter online, voltage conditioned, 0ms transfer
Most desktops will work just fine with the Back-UPS line. The processor is practically sleeping. Workstations should be on a Smart-UPS because the PSU capacitor won't have the capacity for a brownout with a Xeon-class CPU and multiple GPUs.
I have these.
https://www.apc.com/shop/us/en/products/Power-Saving-Back-UP...
They are line-interactive. Most of their "tower" UPS models seem to be Line Interactive.
Its those little ones that look like oversized surge protectors that usually are offline.
Also make sure it's rated for the load. My UPS went "lol nope" when I lost power while I was gaming, because I had forgotten about it when I went from single mid-range GPU to SLI top-end GPU...
Some of the "newer" online models will have a green mode, which although double conversion green mode introduces a transfer time (~2ms, basically relay lift time).
Backups are also square wave, or stepped sine Smart are Sinewave output.
This is specifically APC.
Also remember there are tare losses (copper / magnetizing and conversions), so a UPS just sitting there can use up to 30W.
Back-UPS stepped sine wave has about 3 square steps above or below zero, so it's a very rough approximation of a sine wave. Adequate for many kinds of equipment, but ugly on a scope.
There is, or used to be, a pin on the Smart-UPS line that you could tie high (or low?), to tell the device to operate in "Online" mode (inverter always active, line power continuously tending the battery). APC also sold it preconfigured that way as a separate, more expensive, product. :)
There's also a trick to jump-starting an unplugged-but-charged Smart-UPS. I've used this for several hours of remote power in inconvenient locations.
Quality devices, but test them regularly. The switch from idle to high-draw can be violent, and it takes out weak components. Similar to tungsten lightbulb filaments.
I'm most familiar with the APC line because that is what I use.
I'm not sure I buy that. The motherboard/GPU backplane VRMs alone should be able to buffer 20ms on these devices. Would love to see a test of this kind of thing.
Ouch. Does anyone know where the author lives that a couple of times a year is considered "infrequent?"
Decades ago I lived in a rural area of America where we'd have power outages two or three times a year because of storms. But I've lived in more than a dozen cities since then, and the power has only gone out twice. Once about a decade ago during a tornado, and then again last year when the power company replaced a transformer across the street.
I ask because it's my impression that electric service has gotten much more reliable over the years. There are still access issues where I live now (believe it or not, thousands of Americans don't have access to the electric grid at home), but reliability seems improved. Am I living in a bubble when it comes to electricity?
Edit: My UPS is a line-interactive rack-mount APC unit, which as I understand things don't actually contain a power supply, so everything runs from the battery voltage and is supported by the charger in mains operation, but without a battery inserted the charger doesn't work, because its control circuit is powered by the battery.
Storms during the rainy season usually means i experience cuts. Usually 1-2 times a month for 3-8 hours ea from May-Nov
Hurricanes are usually expected to be 3+ days at minimum of outage. Most recent ones were closer to 14 days.
I have a couple gennys, and interlock kits with 30a hookups on my house during these times. I can run 1 genny on the whole house, or even 2 since i have a sub-panel with another interlock kit.
The only thing i CANT run is my AC. But we have 2-3 window units/portables to get by.
FWIW running generators for days at a time is NOT cheap. on my setup its about 20-30 bucks a day in fuel alone.
Even with my roof (which is east/west facing) the cost to just install panels would be 20-30k. And my city has a 1:1 buy back...And even assuming i sell them back power it would take a decade to recoup the costs...and by then the panels would be less efficient. And this is with buying 95% panels.
but then during an outage....
So you need something to store it... Short of rednecking a series of deep cycle batteries the Tesla Powerwalls are really the only big option.
And those arent cheap either. To run my house for 2-3 days would be in another 10-30k
I can do all of that with 2x7kw gennys and some ancillaries. With a 14-17kw i could probably run my AC (which have hard start caps on them) Even at 30 bucks a day in gas im still well south of even 10k.
Next step up would probably be a active standby generator with transfer switch and giant Propane tank to fuel it. That would be about 10-15k once permitting and all is done. And that adds complexity and cost. But it would mean i dont have to manually flip breakers and cutover. (as it is, my computers are just set to talk to a NUT server, and shutdown in the event i dont cutover within 10 minutes.)
And during storms supply chains are strained. Its hard to source diesel and LP and sometimes even petrol (in fact i warned a previous company of this, and we almost had to shutdown due to fuel levels). And you have to maintain a contract with LP providers and usually rent a tank from them.
With petrol and my little standby units i can source it myself (3-4 am is the best time during runs on gas), even siphon from one of our cars.
I may spring for a full kit standby one day. But for me, it costs about 2k in generators. Another 1k to have an electrician install the interlock kits. And i can service/manage the gennys myself. swap them out while i do maintenance during extended runs etc.
https://en.wikipedia.org/wiki/Solar_cell_efficiency#/media/F...
So he sized the number of panels to offset 95% of his usage is my guess.
They used to sell them last time i was looking into it, which was about 3 years ago.
The quality of the residential wiring in the original half of the house was not friendly to my hard disks. I lost four in rapid succession. Got a newer UPS, line interactive, had better luck after that until we moved out of state.
(That month of disk meltdown made me love ZFS. Didn’t lose any data.)
What type/size of generator are you running? Might be worth taking a look at an inverter unit next time; they're much quieter and sip gas. Can't stress the quiet part enough - you can hold a conversation a few feet away easily.
Assuming you need 240v since you mentioned you have an interlock kit; Champion makes a 5,000 watt model for $1K. Will go for 12+ hours on 4 gallons of gas at 25% load. https://www.electricgeneratorsdirect.com/Champion-100519-Por...
If you can get by with a single 5-8K BTU window unit, a WEN 56200i/Harbor Freight Predator 2000 (I own one) will last for 8-10 hours on a single gallon, keeping both the window AC and kitchen fridge running. They're a clone of the Honda EU2000i/Yamaha EF2000is, and I've found it to be just as reliable. https://wenproducts.com/products/wen-2000-watt-inverter-gene... & https://www.harborfreight.com/2000-watt-super-quiet-inverter...
I currently have a 7kw (8750 surge) and a 6kw unit. they are dewalt gennys with Honda engines. I really only would trust a B&S, Honda or Kohler engine i feel like.
I think my next purchase will either be a bigger inverter or just a 14-17kw contractor genny and switch to double conversion Battery Backups.
Not quite sure. But what i have does work too. Just that my current generator is approaching a decent number of hours (400-500).
I've lived in many different states and cities, and "it varies" is about the best you can say. Areas I would expect to be relatively problem free, end up being some of the worst offenders (looking at you, El Paso County, Colorado).
[0] https://en.m.wikipedia.org/wiki/Arc-fault_circuit_interrupte...
I had AFCI trip on my fridge after renovating my kitchen while we were away for a weekend and ruined the brand new floors when the ice box melted. So infuriating.
I’ve since pulled the AFCI’s out of all mission critical circuits and sold them on eBay.
Just an absolutely disgusting example of regulatory capture.
I guess not having earthquakes, floods or hurricanes helps.
Where I lived in Sweden, 30% of the network was already underground, but after a large storm in 2005 caused long-lasting power outages, they invested heavily and are now up to 70% underground.
Your electricity service may not be as reliable as you think - some utility power can be quite "dirty" with high levels of distortion. There's a lot of infrastructure between you and the power company that can cause problems. A good UPS will not only protect against outages, but will filter/condition utility power.
I live in Chicago, in an older apartment building. I haven't had a full on outage in the past few years, but during major storms I have seen some flickering and other issues. I just looked at the readout on the UPS, and so far it's kicked in 5 times this year. I have a pretty sizable NAS (40TB) and a beefy workstation - the $200 I spent on an APC BR1500MS is well worth the protection.
Safety is another consideration. My parents live on their hobby farm a few hours away - they are the only house at the end of a mile long road with older power lines. I put a similar UPS to keep their alarm/phone/network online. If there's a medical alarm, fire, carbon monoxide, water leak, etc the alarm can still alert the monitoring company, their internet/VoIP landline can fail over to cellular, and we can get alerted that something is up. Again, the peace of mind alone is well worth the $200 (plus $100 for a battery every few years).
In Southern California, we rarely get rain, let alone major storms and we still lose power 1-2 times/year.
On the other hand, I lived in a number of areas across Canada and power outages were quite rare despite the harsher climate.
PSEG...has their work cut out for them.
PSEG spammed us with all these boasts about how they were prepared.
They weren't.
In other cities I've lived in power cuts were a once every two or three years thing and you could still drive on the roads when it rained.
I have a fairly expensive line-interactive unit that can be tricked into doing bad things just by power cycling my laser printer which is plugged into the same circuit (but not the actual UPS). One day this caused me more trouble than it was worth WRT lost work. I would have been better off plugged directly into the grid and just taking the 100 millisecond voltage dip at the PC's power supply. This is one area where buying high quality PSU can go a long way. High wattage Seasonic units can ride through some pretty nasty conditions before needing any outside help.
I also have a cheaper passive-standby modified-sinewave unit which has never misbehaved in terms of trigger conditions, but it is hot, noisy and otherwise does not inspire any confidence when it's running on its inverter. I would also never dream of plugging anything into this unit that uses a non-switching power supply.
Again, I am faced with the reality that you get precisely what you pay for. I always assumed double conversion was expensive paranoia never for home use, but over time this has proven out just like everything else when looking at the value equation of technology.
I have a PC tower server and realized off the bat that it would not be feasible to supply it with backup power for any worthwhile period of time. If the power goes out it will crash, but it runs on ext4 and ZFS and odds are very good it will come back when the power does and put itself back together.
I’ve tested all three extensively with unreliable media (raspberry pi kernel dev testing on an sd card) and ext4 was the absolute worst.
What is the best FS for microSD cards in your experience?
SD cards don’t perform wear leveling so you really need a file system that minimizes/distributes block rewrites, and a log-structured filesystem will do that by never rewriting old data (until there is no space left).
As for my server, I have it on a UPS big enough to allow it to shut down (with some slack for battery degradation). That's all I really need.
If we are just talking about cable modem + wifi router + NAS, you cant be looking at more than 30 watts. Buck/boost DC-DC converters are upwards of 98% efficient, so you don't have a bunch of losses with that extraneous inverter/rectifier stage. All of this stuff can be purchased and assembled with minimal fuss.
I want one for work to keep the network and internet devices running, but it needs about 30 Watts continuously for say 24 hours, not hundreds of Watts for 30 minutes like a UPS for a server.
I presume a "trickle" power UPS would need to care about power efficiency?
I wonder that most consumer SSDs are aimed to power loss safe (by implementing like CoW?) even without capacitor but I don't know the truth.
The flash translation layer in a SSD looks a lot like a journalling or log-structured filesystem, and without power loss protection capacitors you get similar data integrity guarantees: unexpected power loss may mean loss of recently written data (still in the drive's write buffers, and maybe stuff in partially-written NAND pages), but shouldn't cause catastrophic loss of data. You generally have more data at risk from being buffered in main system RAM by the OS than you do from the volatile/unprotected buffers in the SSD itself.
The SSD should never be erasing a block as part of a garbage collection process until the live data from that block has been committed to a new block.
So I encountered in a previous IT helpdesk job a laser printer connected to one of the tower APC units, with a wattage display.
Laser printer (it was a Ricoh Aficio SP 4100-something MICR printer for checks IIRC) was connected to the battery backup set of receptacles for some odd reason.
Any time the laser printer would start up, or run, and I presume charge the fuser, it would briefly spike up to 900w and often set off an alarm. It was rated for 550w. I think sudden high loads like that could be damaging to the unit.
I would not attempt to connect a laser printer to a UPS at all, even if not connected to backup battery--IMHO it's in the same category as high-current devices like toasters and hair dryers. Many commercial copiers I've seen have some type of large surge protector but no UPS.
I'm guessing the surge current to the laser was causing the UPS to register a brownout and potentially not behave correctly at that point.
I wonder how that is even possible in the first place.
Total speculation.
Probably is a fire hazard
https://www.cyberpowersystems.com/products/ups/pfc-sinewave/
(Why yes, I live in California, land of multi-day power outages. Why do you ask?)
Generally the rack mount models, such as CyberPower OR1500PFCRT2U PFC Sinewave UPS System, 1500VA/1050W, 8 Outlets, AVR, 2U or CyberPower OR2200PFCRT2U PFC Sinewave UPS System, 2000VA/1540W, 8 Outlets, AVR, 2U.
Very happy, and stopped having to replace sensitive flat screens in bad power zip codes (trees vs. lines every other storm, such as for the last week after Isaias).
But the main point in the article is worthy: UPS manufacturers have been skimping somewhat on functionality, features and quality over the past few years...
I still have a few APC Smart UPS's bought in 1999-2000 running in my house; at the same time I had a few cheapies that broke after a few years, sometimes in spectacular ways (electronic exploded melting part of the battery case); I will not name brands, it is not worth.
Since 2000 I changed cases full of batteries, usually every 2-3 years. The good thing with the APC Smart series is that it works even if it gives an alert that the battery is bad, but you will get a few minutes of power instead of 30-60 minutes (my server & NAS is very low power, 5% of the UPS capacity). I also set the server to shutdown when the battery is reaching 50%, so I never deplete the batteries. At the same time, the UPS on the water pump is usually running on batteries as much as it can, so I need to change these batteries after 2 years. I have both the server and the water pump in the basement, the temperature is around 16-20 degrees Celsius (winter-summer), when I was keeping the server and the UPS in the house the battery life was at half (~18 months). The heat and the deep discharge cycles are effectively killing batteries.
I had the same type of battery on my motorcycles, one lived about 8 years and a second one 6 years, but in very different conditions: it was warmer, but almost no discharges.
I've used UPSes that discharge batteries no lower than 10.5 (no deep discharge) and only once in a couple of years and that charge them to and keep them at 13.6 volts under stable room temperature all year. All batteries lost most of the capacity after 3 years, two almost all of it, one had like 40% left. In five years only that one was still surviving short minutes outages, while initially was able to do a few hours. For comparison, I had one battery just laying around for 6 years in the same room without touching it and it lost only like 30%.
I found this forum posting while searching for how to maximize the runtime on my APC UPS: https://forums.overclockers.com.au/threads/hacking-the-newer...
I had issues with power supply to my apartment once in 2 years or something like that.
This model has "controlled by master" outlets which I did not use because I needed only backup lines connected to battery. That "controlled by master" thing would turn on once in 3 months without reason.
Then one day there was no power loss and UPS flipped and turned off power to my NAS and to my devices. Battery was good power was there but switches flipped and my NAS was turned off without warning. There was no system updates, maybe just normal writes to the disk.
Turned out my HDD WD-RED 1TB died because of that one flip. It would have been better for me not to have a UPS...
It sounds like you used a Back-UPS unit instead of a Smart-UPS, which I’ve heard switches to battery 10x faster than the Back-UPS.
After learning this I won’t use a Back-UPS for anything where meaningful data loss might happen. They’re great for lights, cable modems, WiFi routers, etc., though.
I was doing turn off tests with my NAS attached and it was switching correctly and NAS was not going down, I had couple RPis connected also not going down and notifications from NUT were in logs. Where my NAS was NUT server and RPis had NUT client.
It was that one day when UPS malfunctioned in a bad time. Where my electricity provider was more reliable than UPS I owned. So bad risk management on my side.
I called their customer support, and they asked about the kind of equipment I had plugged into it.
My main NAS motherboard had died, and I hadn’t gotten around to the full rebuild that was going to require. So I had ended up with about five or six cheap USB external drives, with those “wall wart” DC power supplies, strung along a couple of power strips that were plugged into the UPS. That plus a 27-inch iMac.
It was generally not much power - not much surge to spin up, or to power up the Mac, and of course at idle was less than 100 Watts.
APC Customer Support said that the power strips might actually be weirding out the battery.
They sent me a replacement battery at no charge, and suggested that I go with a Smart-UPS, with their Power Distribution Units if I needed more outlets.
I had always thought the rack-mount PDUs were a silly expensive replacement for a no-name power strip. But I had long had a keen interest in this sort of IT equipment fail, after witnessing some spectacular crazy failures in corporate, and losing the manuscript while I was writing a book on small-office IT practice.
So I bought a basic PDU, and a Smart-UPS rated at something like 700 VA.
That has been rock solid. Even better than a larger setup I had with some big CyberPower systems.
I’m willing to go along with the cargo-cult of PDUs now. I would be quite surprised if there were any significant difference among the basic PDUs of well-known brands.
I still have those USB drives. They’ve lasted about as long as the hamsters, not as long as the dogs.
I think it's more likely that you purchased a guerilla-remanufactured unit that was properly sealed and packaged to look brand new (with hologram stickers and anti-tamper seals, etc.) but had an old, already dead battery that was reconditioned just enough to last a few cycles after purchase.
Did you buy this directly from APC ?
I bought the SmartUPS and PDUs from an authorized dealer that APC Customer Support referred me to.
That worked out better.
It works perfectly for me, these APC devices have given me at the very least 1500 hours of continuos internet access where I could work or entertain myself during the outages over the last 4 years, so I think the 180$ per unit it cost, was a pretty good investment in my case
1.) check your battery health annually and replace bi-annually. I don't care about the battery quality, an always-online UPS is very hard on lead-based batteries. they're float charged basically all the time, and deep discharged during outages. if you're in a power failure prone area (i.e. more than one per year), invest in a generator too in order to avoid stressing the batteries with full discharges.
2.) check the battery float voltage with an actual volt meter to make sure the UPS is calibrated correctly. APC UPS's in particular, prior to and post the Schneider Electric buyout, had a lot of really poor cost-savings decisions applied to their enginnering. this included using a carbon film resistor-based voltage divider to measure the battery voltage on much (or all) of their Back-UPS and Smart-UPS range. if you don't know anything about resistor technology, carbon film resistors (especially cheap ones!) change in value due to temperature, humidity, and age, usually drifting up in resistance. this is a bad thing when you need a stable voltage reading...
oh and on APC's cost cutting measures; there's the fun time they stopped using nickel plating on the bus bars in their Symmetra UPS line. they switched to tin plating instead. apparently none of them had heard of tin whiskers. :)
We had quite a few APC units in the 1990s and 2000s, and I had the impression the older ones were easier on batteries. We were on a three year replacement cycle until Y2K, and after that we found it necessary to check the batteries every six months or so in the newer units.
> So I thought I had covered all bases, and, indeed, the UPS proved useful on several occasions. I would quite often be on a work trip and receive an e-mail from the server informing me that mains power to the UPS had been lost, then another e-mail soon after informing me that mains power to the UPS had returned. Only once did the power cut last longer than the battery capacity, and the server was shutdown automatically.
It sounds like the UPS is doing exactly what he wants it to do. This is exactly why I bought a UPS. I would have power transients sometimes and rebooting my entire network (with NAS and servers) was annoying, and I didn't want it to go out when I was gone.
His main complaint is he bought a standby UPS where the battery failed, it turned off the UPS (which is understandably frustrating), and now replaces the battery every three years. Looking at the white paper referenced, it even says to go for Line Interactive for servers. I'm also curious how the person knew that power wasn't cut when he was away?
So other than buying the wrong type of UPS (which at face value looks to be a valid complain based on research)....why isn't a UPS as useful as they'd think?
I assume this would be because the power is always flowing through the battery/inverter? Wouldn't that kill the battery a lot faster? Although it'd make the switch from mains to actual battery power basically a non-event from the output's perspective.
Now that I think about it, going 120/240->low voltage dc->120/240->12/5/3.3 is incredibly silly. IIRC there are 48 (or -48?) DC power supplies out there, mostly for data center use. Does anyone make an ATX version that you could hook up to a UPS that has a DC output?
This is a design choice from APC, money is on the control electronics being powered off the battery meaning a dead or missing battery means the entire unit is non-functional. Making the unit resilient to this probably costs as much as a line-interactive unit, however, so you might as well just tell people to get a better unit than make a middle-ground solution.
This isn't common for servers though which, if equipped for redundant power, have two AC power supplies. I think this just comes down to the higher current draw usually associated with servers, running even 48VDC power can become costly and impractical when you need to supply say a couple thousand watts per rack, so the extra space and thermal load of an AC power supply in each unit becomes preferable.
The downside is that while redundant power supply is usually a "standard" feature on switches and routers (that is, they always have the connection whether or not you buy an RPS is up to you), you need to specify and pay extra for dual AC power supplies up-front in servers, so you don't get as smooth as an upgrade path. That said in datacenter environments you often buy servers, racks, and power systems all at once so it's not as much of a concern.
There's a similar tradeoff that exists around bottom-of-rack UPS and central (building or area) UPS---bottom-of-rack UPS tends to be more expensive, higher maintenance, higher thermal load, etc for large installations, so usually large installations use one (or two for A+B power) large central units which may be hybrid between different technologies like flywheel and battery (and you could view the transfer to generators as a "third stage" of a central UPS system), but it also makes your cabling a little more complicated and you have a bit of an "eggs in one basket" situation. Central UPS generally have internal redundancy so their risk of failure is low, and often a bypass device where if a non-recoverable failure of the UPS is detected a contactor "shunts" the entire UPS removing it from the circuit so you don't lose power due to a UPS failure, but that doesn't necessarily save you when the HVAC pours water directly into the UPS control cabinet causing widespread failure of the control electronics... a situation that I have somehow seen twice. I bet on bigger installs you can get an external bypass with some sort of health monitoring though? If HP/Tandem taught is one thing it's that you can always through more redundancy into your very special basket.
That UPS was well worth it to put a stop to that happening.
As a small business network/systems admin, I've had more failed UPS's bring things down than I can count. It's very common.
After wasting a lot of time remotely trying to diagnose this, it was obvious what was happening once I got on site. It turned out that once this server was shut down, the UPS didn't have enough power draw to power the other equipment attached to it, so a switch that was plugged into the same UPS would shut down as well. For some reason it had failed in such a way that it required a certain level of draw for it to function at all.
Most homes have something with a mains-powered, non-battery-backed clock. You get home and the microwave or the oven is flashing 12:00 and you know there was a power cut.
> So other than buying the wrong type of UPS
This isn't a failure of 'type' of the UPS - it's a design fault deliberately left in by the device manufacturer for market segmentation.
It would be trivial for them to keep the power outputs on if the battery failed but the mains remained on. Leaving it out is a business decision, hobbling their home product so it doesn't steal market share from their small-business product.
The constant humming (no fan, just electrical) so cant sleep in same room
The loud can't turn it off beeps when power is out.
And none of those come in the product description.
I personally like the white noise from electronics, so I can't help you there.
It's enough of a pain to do that rather than configure all 3 UPS devices in my house, I've only ever gotten around to doing it on one.
Can't say I've noticed the electrical hum, except during a blackout. But I don't have mine in a bedroom.
That depends on the UPS, it seems.
I have a pair of Back-UPS 1500s with the external battery packs. They have a button on the front-panel for disabling/enabling the audible alarm (just hold it down for 2s, IIRC).
Now imagine the sound at the APC factory, when the power goes out and thousands of UPSes on the charging racks start beeping in sync...! (this is not a hypothetical)
I have been using the same offline UPS and battery for the last 7 years and it is still working fine - a few days ago it handled a 15 min power outage with 50% of the battery.
Can someone explain why offline UPS can fail if the battery dies? Is it by design or a manufacturer specific issue (e.g. only APC brand ones do this)?
How did he also know power didn't go out too? That confused me as well.
I had a script in a raspberry pooling my UPS each minute asking if the unit was running on AC or battery, and dropping that info into a file when it changed. I know the software/daemon can also do something like that.
I actually had the exact same thing happen to me a few months ago. A power flicker so brief that it didn't even reset any of my clocks, but it caused one of my cheapo standby UPSes to give out.
"online" is generally a double conversion UPS.
These are generally the best for clean power to the electronics. Since its always supplied by the battery. Its also often one of the hardest on the battery.
Line-interactive are probably most common. They are pretty solid, but since there is a delay for the cutover, surges and really dirty spikes can make it through to the equipment. So say a lightning strike or REALLY bad surge on an overloaded generator can get through, whereas on a Double conversion it may just trip the fuse or breaker. Also running a whole home or standby generator on these line interactive can make them trip constantly since the generators often dont
A) run at 60Hz (ie: mine runs closer to 63Hz)
B) run with a pure sine wave
You can get inverter generators to help with this. But thats a cost too. And otherwise the solution is to "de-tune" the UPS sensitivity.
I de-tune this on mine. Its been fine, running on gennys during the rainy months for hours and every know and then...days at a time.
I have had to swap out power supplies more often..But even then, usually thats after like...5+ years of runtime.
I have to de-tune their sensitivity but they work.
But it does stress the CPU power supply caps a little more and shorten their life. But the caps on the board are fine, and havent had an equipment failure outside of a PSU...And ive run literal weeks on genny power.
But that stress is more due to Hz being at the threshold and artificial stepped sine wave i think. Voltage fluctuations when load kicks up certainly doesnt help.
For once there was a high voltage fault in my neighbourhood, destroying few lights, all mobile chargers plugged in and microwave left in standby. Had I not been home, and fault lasting more than 5min, things might have gone really bad.
Now on, Whenever I leave my home for long duration, I just empty the Refrigerator and switch the house mains supply off before loving the house.
First and foremost: fire alarms. Buy them, and use plenty of them.
Secondly: let your neighbors know you’re on vacation. This implies a certain amount of trust with your neighbors, but it’s good to be on good terms with them for more reasons than just this.
Thirdly: I get notifications if a server goes down or a fire alarm is triggered.
With all this in place, I don’t worry at all. I may be somewhat ignorant or oblivious to certain risks, but in general I trust the safety net I just described.
- Sustained overvoltage. Any overvoltage lasting too long to be handled by a surge protector should trip the main breaker.
- Sustained undervoltage. Some devices don’t like this.
- Phase error. A two phase main should detect incorrect phase-to-neutral voltages and incorrect phase-to-phase voltages. A three phase main should also detect incorrect phase sequence. This should also catch some cases of a disconnected neutral.
- Excessive neutral to ground current. In the US, for reasons I personally strongly dislike, every “service” has a connection from ground to neutral, usually in the meter box or the main panel. Various errors can cause substantial current to flow through this connection and can be dangerous. For example, if a house and the utility both have excellent grounds but the house’s neutral feed breaks, all the voltages can be close to correct and the house’s neutral current will return to the utility through ground. This is dangerous and will not be detected by common equipment. (A whole-house residual current detector would satisfy this, too. I think these are used in Europe.)
I used to feel this way, but then I realized that the electronic devices don't know I'm away on vacation, or when I've returned, and they're just as likely to start a fire when I'm down the street getting coffee as on the other side of the planet. And when I return, my presence doesn't magically reset some catastrophe timer and everything's all better because I touched "base."
That said, if I'm gone from home long enough that I turn off (or down) the refrigerator, then I'll unplug everything. So I guess I'm still a bit paranoid.
A lot of devices will smoke before starting a fire, sometimes for quite a while until temps increase sufficiently for ignition. You would smell the smoke and figure out the culprit.
Turning off everything is still worthwhile while you are away. This is just risk management. It is a small risk (maybe you have faulty appliances or house wiring? maybe you live in an earthquake or tornado area? etc) but the mitigation is also not very inconvenient.
That said, if you can have a smoke detector pinging you (or a similar device - Amazon's cameras have a microphone that can alert you to alarms) it would be better. As you point out, stuff could happen while you are temporarily outside the home.
Responding to a water-related disaster is a different story, though, and worth bringing up.
You might want to consider turning off your water at the main if you're going to be away from home for an extended period. I was fortunate in that the only water-related failure I've had personally was a ruptured water heater leaking into a floor drain of a 1st floor utility room. I know others who haven't been so lucky (having clothes washer hoses rupture, fixtures fail, etc) and have had water running inside their home for an extended period (several days, in one case). Until someone notices the local municipality reading excessive usage, a neighbor noticing water running out a patio door, etc) the damage continues and, to add insult to injury, you're also getting charged for the water / sewer usage.
If you believe fires start at random, there's no difference between a 2 week holiday and 9 weeks working 8 hours a day.
It's kind of annoying and I'd love to see a solution that isn't just "get a battery based UPS" -- maybe something capacitor based, enough for say 15-30 seconds?
I guess I could get a flywheel [0], but I don't think it'd fit in my basement.
[0] https://www.vertiv.com/en-us/products/brands/liebert/?id=59
The conditioner provided a constant voltage to the servers and could correct "sags" where the voltage would drop, and it could keep things going for a few seconds if the power went off completely. As you say this took our total outages down from probably 10 a year to maybe 1 or 2 - most outages are quite short. Although looking at the graphs on this thing, there were pretty frequent surges and sags coming from the street, like when you see light bulbs flicker in your house.
One other thing we noticed was that if there was a short interruption, all of the HP servers could survive for a second but the Dells turned off right away. Different power supplies store up some amount of power locally.
https://new.abb.com/ups/power-and-voltage-conditioners/volta...
Looks like it's capacitor-based, removes the need for an inverter, and even has a clever way of communicating to the host, by taking over the power switch connector.
Unfortunately, I can't seem to find a price or where to buy. Would love to see more momentum around products like this, and marketing directed at home systems.
EDIT: Thought of one more system that uses capacitors to limp to a clean shutdown--the Unifi Cloud Key Gen2.
Again though, the power solution isn't packaged for consumer use.
Though, maybe it'd be amusing to try to hack together a NAS out of Cloud Key Gen2 Pluses... there's a hard drive bay in each one ;D
Since I am stuck here til next July I started fixing some of this. More than anything else power conditioning was the single most important thing. Anything with any reasonable switch-mode power supply will be fine on a line interactive UPS, most people don't need to spend $ on an online UPS - esp if your power is fairly clean. We had old locally made generator so I finally replaced that with a nice single phase 25kVA generator. The alternator has voltage regulation. For the mains power I was told most people just put a Servo Stabilizer but I didn't want any moving parts so found an IGBT based static stabilizer instead.
http://anjalipowersystem.com/static-stabilizer.html https://download.schneider-electric.com/files?p_File_Name=JS...
I did a compressor change out on a package system last year because while we apparently sprung for all sorts of other “options” on the machine apparently the line was drawn on a $48 phase monitor. Our cheaper machines had them and rode out our service being single phased, and the package units stage two was thankfully off at the time; but we still had few thousand dollars of compressor needlessly converted to a piece of scrap metal.
For critical loads where I don't want interruption I prefer online or double-conversion UPS's. Basically they have two inverters instead of one. The load runs off of battery constantly on the first inverter, and the second continuously charges the batteries. There is no cut over time when there is mains power interruption since the loads run from batteries.
But I hadn't considered this dead battery scenario so that's another thing to add to the checklist of things to ask about. It seems that there should be a bypass mode if there is mains power to supply power to the load too all the time.
I've found UPS's to be a mixed blessing - when the power does glitch they can save your bacon but the battery maintenance issue can cause their own outages. If I lived in Florida with their legendary lightning I'd just put up with it but I used to have UPSs on everything and these days I have really dropped down where I have them since we get maybe an outage a year and glitches are equally infrequent too.
And this is why professional servers have two, redundant, power supplies.
If you had two power supplies, the UPS would fail, but you'd continue to function off of mains voltage.
UPS can fail. Mains can fail. Two power supplies hooked up two both powering your server redundantly means you'll only fail if both fail simultaneously.
This will probably occur at a different time than when the UPS wears out, so it could be beneficial to keep them separate so you can replace the surge protector only.
It's just that consumer home UPSes are almost scam level products designed to profit from naive people. They are both not improving things for an average user and killing batteries after like a year of operation (lead acid batteries can last for like a decade when not subjected to those UPS chargers).
Although I'm a bit "old school" and generally prefer serial (RS-232) ports, this is one case where the change to USB really was a huge improvement!
Maybe APC really wants to drive users of its lower tier (and cheaper) UPSes mad with the beeps and force them to buy the more expensive Smart-UPS range that comes with a built-in option to turn off audible alarms.
Well, audible alarms do have a place in such systems to inform the person that the connected devices or systems are running on a limited battery capacity. But forcing that at all times is a big annoyance.
I meant to tear it apart and remove the beeper after that but never got around to it before the battery made it useless anyway. TBF it was a few years old at that point so the battery giving up wasn't unexpected.
https://www.amazon.com/gp/product/B01FWAZEIU/
The beeping didn't really bother me but it does frighten one of my dogs.
Agreed. The cyberpower devices do let you "silence" the alarm, but even with the alarm "silenced" the UPS will begin rapid beeping as the battery nears depletion.
So when it came time to move our staging database over to the new wiring, the head of IT unplugged the UPS from the wall, the UPS complains like UPSes are wont to do in such situations, and in the time it took him to untangle the cable and plug it into the new circuit, the UPS dies. Bad batteries that the health checks didn't detect. Unplanned shutdown of our shared database. Whups.
And that is when management was taught about redundant power supplies, and periodic UPS maintenance.
And hopefully about setting up monitoring on your UPS to keep track of battery health ;)
If my batteries die, I'll just replace them with non-battery backup strips. Not worth the trouble for me living in a city where power goes out maybe once per year on average for 10 minutes, and I don't have servers that need to stay online anymore.
- failed battery caused APC to stop operating even with power (just as in the original post)
- stopped charging battery even though battery was still good (blow electrolytic cap. When I into the circuit that capacitor was rated for 14V on a 12V trace, ie no guard-band. Seriously, skimping on maybe $0.01 total parts cost to leave a system at risk of fire??? Who the heck signed on on that decision???)
- couple of data center APC UPSes caught fire. On the plus side the emergency systems kicked in quickly enough that the only damage were the UPSes.
Anyway, that was a few too many stupid failures. APC is on my "never ever buying from this company ever again" list.
Teah, it's just this simple. This is just a byproduct so all those companies can boast their insurance coverage on damaged equipment you connected to their UPS. And guess what, most of the time you won't be able to get this insurance even if your stuff got fried, saw a few stories like these from recent Slickdelas posts of UPS.
You really don’t want line-interactive gear when dealing with larger loads either, as the transformer will just cause extra current draw during a voltage sag. This is why you don’t see (many of) them above 3-5KVA.
EDIT: I found an interesting thread about the powerwall, switchover and using a UPS:
https://teslamotorsclub.com/tmc/threads/powerwall-2-ups-conn...
it appears the powerwall moves the line frequency up from 60hz to as much as 65 hz as a signal to the solar power inverters and that can trip up some UPSs which monitor 60hz as an indicator of "good power"
I live in a fairly rural place (Somerset, UK) In this particular part of the county, outages are approaching "per demi-decade". A recent boiler room explosion in a BT phone exchange caused my office to lose power for about 10 hours but my home, which is about a mile away, was fine. I kept the genny topped up at regular intervals. The many but small UPSs we use keep things running for the 10-20s that the genny needs to fire up and settle.
At home I have several small UPSs to keep things running for about 45-60 minutes. The esxi (Dell T320) in the attic gets shutdown after five mins so that everything else up there gets more time.
Anyway, its all about risk assessment, monitoring and understanding your gear. A Back-UPS is not the same as a Smart-UPS - output waveform and response to load etc. Also, your UPS should be self testing every two weeks. I have several APC Smart UPS 1500VA and others. The 1500 jobbies are something like 15 years old but on their third set of batteries at least. When apcupsd whines that they are getting on a bit we buy new batteries and swap them in on-line.
My golden rule of thumb is do not buy a battery from a firm that you do not recognise. Do not skimp and save pennies over something that can explode or simply be rubbish.
https://www.cyberpowersystems.com/resources/choosing-an-unin...
These low-end APC dinosaurs still use lead acid, and in my experience (with a solid utility over the past decade) outages are just as likely to be caused by battery failure than some kind of actual power outage.
If anyone wants to build something better, happy to chat further.
Edit: Since we're armchair-building for a predictable, high-uptime application, failure modes might have an interesting advantage though. Any battery engineers know if lithium ion has more predictable or measurable failure modes? Yes, we've all seen the exploding hoverboards... I'm assuming theoretically well-built electronics.
Additionally, li-ion has non-neglegable disposal costs, so I don't think it's fair to count that solely again lead acid batteries.
Car batteries are lead acid/gel cell batteries. They last for years and can take a very large discharge while still being rechargable.
Also, by cycle I meant a single vharge-discharge cycle can be more useful when larger amperage or more power is needed between charges.
What I don't understand is why it would be so hard to just scale those power packs up a little bit and slap a normal 110 volt A/C plug on them? Wouldn't that essentially be a super efficient, light weight UPS?
I did a bit more digging and I found a couple of portable power packs that offer this sort of functionality: https://www.nytimes.com/wirecutter/reviews/the-best-portable...
What I don't understand is why they are so expensive, and why they can't seem to scale them up to the same size as a normal lead acid battery UPS?
When we moved offices 3 years ago I pulled our UPSes out, because all the outages the previous 5 years were caused by the UPSes.
In my last job I had around 100 APS SmartUPS's (mostly 3KVA) in various wiring closets and in the ~3 years that I was there, none of them failed unexpectedly, but we had to replace a few batteries and/or entire units when they failed their monthly self check.
UPS's take about 10W to just power the inverter. This gives up a lot of the battery capacity not to mention the losses from converting your power 2 times.
Keeping a whole system running for a while was my real intention, but I quickly learnt there were real shortcomings with my no-name brand purchase.
The most common cause of death is sulfate build-up, which can be prevented (and often reversed) with a homebrew circuit[1].
1. https://batteryuniversity.com/learn/article/sulfation_and_ho....
I have worked on UPS that are in charge of maintaining power to DP systems. When they fail you get things like oil rigs pulling off station costing millions. The company I work for just finished putting batter powered thrusters on an oil rig. First one ever.
Don't have a job right now. Rather sad.
You can purchase replacement batteries from them for roughly half the price!
For these I wonder if instead of a UPS: buy two LiFePo 12V batteries with included BMS, two battery charger (AC to 12V DC), so you get a redudant 12V bus where to plug your devices.
If above 12V is needed add two DC-DC converters eg 12 to 19V to add a 19V bus. Same for 5V.
It shoud work and support failure of any one of the parts?
It's good for brief outages caused by thunderstorms, though, so still better than nothing.
Another thing to be aware of, if your PC has a PFC power supply, is that it might reboot or have undefined behavior on event of the switchover to battery, unless the UPS is designed to avoid this.
APC ships crap batteries and they often do not survive the warranty or die shortly later.
You can choose your own battery and battery maintenance equipment for maximum longevity, and choose your own switched power supply with different tap points for different DC voltages.
This is simpler and more reliable than AC line-interactive UPSes because synchronizing an inverter with wall-socket power is not required. Yes, you need some EE skills to build it, but the end result will be far more reliable than a cheap off-the-shelf UPS.
[0] https://www.rvweb.net/best-solar-charge-controllers-for-rv/
I've come to the conclusion that if you are going to use a UPS, your equipment should always be plugged in to at least 2 UPSs, either via redundant power supplies or an ATS PDU.
I got personal experience with this failure. Maybe the thinking is that in case of a power outage you don't want to find out that all your UPS devices fail at the same time. But in this case a 24 hour warning would have been much better.