California power outage triggers chaos in science labs
nature.com
nature.com
One day some years ago, the university administration thought, "why are we wasting money keeping power on, running lights and aircon and God knows what, when the buildings are vacant?". They soon turned that thought into action, and when a long holiday break came, they just shut off the power to the buildings as soon as everyone left for the day. About a week later, the PhD I mentioned and his co-workers came back to the lab, saw some droplets on the glass, and almost had a heart attack.
Turns out, after a week of being without aircon to dry and warm it, the air was just about to cross the dew point threshold, and if it did, it would kill the optics on a million-dollar research laser they recently got installed.
I hear that since then, they're a little bit less pound-foolish.
He was newly appointed as the supreme leader at this automotive plant. Uncertainty was high, this being 2007, so people were let go, and costs reduced with any means possible. So he went into production and one day decided that it is too cold in the storage area. He orders his people to increase the temperature with a few degrees since that would save tons of money. People protest, he hears only "blaa-blaa", and his request is implemented.
A few days later, it turns out that barrels upon barrels of chemicals in the storage area needed to be kept under a certain temperature. So the stuff solidified and it was now only good for the dump site. Out to garbage the chemicals went, new stuff was ordered, production was stopped until the new stuff arrived and temperature was restored back to normal after that.
https://www.chesterton.org/taking-a-fence-down/
Often the most important comments in a codebase are signs on such fences, preserving hard-won knowledge of pitfalls in software and human processes.
Chesterton's fence is sort of about understanding something that isn't necessarily obvious, changing storage conditions in an industrial environment without checking requirements is neglect rather than ignorance.
Reasonably avoidable ignorance of essential knowledge is a manner in which neglect creates disaster; ignorance and neglect are not opposed concepts.
I suppose I am also arguing that there is a mechanistic requirement of the role 'plant manager', where they shouldn't make changes without understanding the impacts. That's somewhat different than ignorance, because it's pretty much their job to do what it takes to act correctly, avoiding assumption and the like.
http://quichemoraine.com/2009/06/the-hurricane-lantern-effec...
My manager used it quite a bit.
Why can't we do a help page as good as company X?
Well, we can but it might take 3 FTE for 3 months
Why can't we can't we have feature W?
We can but we have features X,Y and Z to deliver to customers and nobody is asking for feature W
and so on
Edit: If a Uni has a multi-million dollar laser, it should have a few thousand dollar backup generator!
This. If you've got a multi-million dollar experiment, gee I dunno make sure you're in a building with a generator?
"hey Mr. Government, I know you gave us that multi-million dollar DARPA reserach grant, but uh, someone hit a telephone pole drink driving on the weekend and, uh, power was out for several hours and uh yeah, that 2 years of research, uh, we gotta start all over can we have some more funds please?"
What exactly is the usefulness of a felony as applied to a corporation?
As applied to an individual, it is capable of ruining your life through privation of constitutional rights (firearm ownership, right to vote, permanent mark on background checks make it increasingly unlikely you'll ever hold any sensitive position, etc...)
What does it mean for a corporation to be a felon? It sure seems like the concept is discontinuous in terms of scale. Obviously, there's an extra layer of court oversight that's been mandated, so some autonomy is lost. Still doesn't seem though like the the punishment for a corporation is concommittant with the general treatment of an individual having committed the same relative severity of infraction.
In fact, I'm surprised that anyone even wants to do business with anyone involved with the company during the period the felony was handed dowm.
The more I think about it, it seems like all the "felon" label does is act as a signpost for discrimination being okay.
Anyway. Brain dropping. My apologies for the tangent.
> Also, how important can these experiments be (in CA) if they don't have sufficient backup power at a university??
How important can freedom from fires be if they don't dig trenches for power lines in critical areas?
They have backup power which can get them through any expected power outage. They don't have unlimited backup power that would be able to get them through a "apocalypse" or major terrorism scenario where the world is ending and power will be out for more than a week. Preparing to survive an apocalypse or major terrorism incident is possible. Build all facilities a mile underground with a self enclosed ecosystem. Might cost trillions, but can be done.
PG&E has intentionally deployed, at short and inadequate notice, what is equivalent to a terrorist attack on west coast infrastructure. No one prepared for or expected that scenario. Past events have shown that PG&E management are sociopaths who intentionally engaged in mass criminal negligence which resulted in dozens of deaths and billions of dollars of property damage. Even under the oversight of a captured court, they continue with their games.
If there is any sanity left in California, the USDOJ would bring the executives who deployed this plan and the judge who signed off on it under terrorism charges. If any deaths occur as a result of these antics, it should be capital charges.
You make a good point about distributed distribution. Would have to be very local to avoid this wildfire problem as PG&E is doing (shut off power instead of deal with actual issue). The wildfire problem is one that is decades in the making and coupled with poor open areas management by government policy in general and not just the power company, in particular not allowing periodic fires to burn out underbrush. I had this huge pile of brush in my backyard. When it burned it was very dangerous. Now I make sure to do more regular smaller burns as giant burns are not easily manageable.
I like the solution of small Hitachi style safe nuclear reactors that service neighborhoods of around 2000-10000 people, can't meltdown or breech, are useless to terrorists, and require no maintenance. Doubt we'll go that way though.
However when you have a laser system it's not as simple as having a backup generator. When the entire building power is affected, the AC system stops working, and all your nicely aligned optics become unaligned. If you fix them at that time, then when the AC restores, you have to realign them.
There are hundreds of externalities that affect productive state of the art research.
Berkely has a few generators though so I think these particular complaints are due to cost tradeoff decisions made decades ago and a general lack of foresight to predict the utility power completely shutting down.
At some places like MSSL we even had an onsite diesel generator becuase there were flight critical experiments going on all the time.
Many other faculties don't need that level of resilience so they don't bother. I'm not sure what the situation is for wider engineering or bio/chem. I imagine there are serious health and safety concerns for things like fume hoods that must be able to evacuate gases in the event of a power failure.
Reminds me of something I saw shared here on HN https://cybersquirrel1.com/
>This map lists all unclassified Cyber Squirrel Operations that have been released to the public that we have been able to confirm. There are many more executed ops than displayed on this map however, those ops remain classified.
>Confirmation for all ops has been preserved by the Internet Archive's WayBack Machine whenever possible.
>"I don't think paralysis [of the electrical grid] is more likely by cyberattack than by natural disaster. And frankly the number-one threat experienced to date by the US electrical grid is squirrels." - John C. Inglis, Former Deputy Director, National Security Agency 2015.07.09
Silicon Valley Power serves many homes in addition to large companies including Intel and Nvidia, and they do so with lower prices and better quality of service than PG&E. I will be very unhappy going back to PG&E if I move to another city.
Electrical utilities don't need to be a government-granted monopoly like PG&E, which combines the worst aspects of government-run and private-sector business. Silicon Valley Power proves there are better ways to run electrical utilities.
Is Silicon Valley Power not a government-granted monopoly? How does their business model differ?
Statewide utilities are bloated and inefficient, and they abuse their monopoly power. PG&E acts like the only choices are to cause fires or turn off power when it gets windy -- it faces no pressure to find a better solution because there is no competition.
Small-to-medium municipal utilities that serve smaller areas can serve their customers much better, as Silicon Valley Power demonstrates -- if people don't like it they can move to a neighboring city. In contrast, PG&E is almost impossible to avoid.
California should give more cities and counties the ability to experiment with this model. Competition between municipal utilities is better than granting a single company a monopoly on a utility millions of people depend on.
A government entity in terms of public services tends to have "getting their job done right" at the top of the list, albeit at a lower cost when possible.
not my experience. Most government entities try to raise as much money as possible to protect their jobs and raise their salaries. gettingthe job done right is not remotely near top of the list. See SFMTA or MTA (new your public transportaion) as 2 examples. Or watch IKIRU to see the same attitude in Japan. Yes it's a movie but it rings true to people's experience that goverment never tries to "getting their job done right"
I'm not defending evil and corrupt PG&E. I'm only responding to the idea that government services are magically incentivized at "getting their job done right". Their incentives rarely align to "getting their job done right".
Other examples off the top of my head, public schools in the USA who have more money per student than pretty much every other country but have worse outcomes. I don't have a solution but clearly they are not "getting their job done right"
https://nces.ed.gov/programs/coe/indicator_cmd.asp
Another example might be the USA's militarized police force killing its own citizens. Some people have arguied they're incentivized to buy all the military equipment because it raises their budgets. Again, they are not targetting "getting their job done right". And again I don't know the solution. Just pointing out making something a government serivce does not magically remove all bad incentives.
you can read I guess a right leaning POV
https://www.charleskochinstitute.org/issue-areas/criminal-ju...
or a left
https://www.vox.com/2015/5/31/17937892/ferguson-protests-pol...
Perhaps, but you didn't say "Some government services are poorly run". You said:
"not my experience. Most government entities try to raise as much money as possible to protect their jobs and raise their salaries. gettingthe job done right is not remotely near top of the list."
You have not yet given an example of this. You're just pointing to random organizations' failings and saying that they're caused by incentives specific to government, without any evidence whatsoever.
Are MTA and SFMTA's failings because they are "trying to protect their jobs" or because they have to maintain aging infrastructures in a culture that idolizes cars?
Are public schools' failings because they are "raising as much money as possible" to "raise their salaries", or are they because of mandatory religion-based curriculum, mandatory standardized testing determining funding, lack of healthcare/childcare/nutrition support for parents, and parents being incarcerated?
It's clever of you to bring up police, because you can look through my previous comments and see my political bent, but keep in mind that the discussion at hand is whether government or corporate incentives lead to better results. Looking at where the money goes when overly-militarized police purchase military equipment, I think it's hard to attribute this to government incentives as compared to corporate incentives. You can't point to corporate meddling in government causing problems, and use that as justification to put the corporation in charge.
Saying "not my experience" at the beginning is pretty disingenuous, because it's become increasingly clear that you don't have any relevant experience.
Of most universal health care systems vs. the US "mandated private" insurance - see how much of society is covered and how well.
It doesn't mean they are setup that way or that they (or the electorate) don't become captured by vested interests but it is the most effective way of keeping natural monopolies in our economy free from corruption (heavy regulation can also work).
Or they can be shielded from accountability because government bureaucracy. Imagine if the MTA were a private entity. Their contract would not be getting renewed (to put it mildly).
<wikipedia>
SVP receives generation produced outside Santa Clara via transmission facilities owned and operated by PG&E under the direction of the California Independent System Operator (CAISO).
</wikipedia>
About 2/3rd of SVP power comes from outside. If PG&E cuts certain transmission lines the good folks of Santa Clara will feel the bite, a bit lesser perhaps, but a brown-out can be willfully imposed by PG&E
The Berkeley campus is up against the East Bay hills and so there are trees all over near the campus.
This seems overly general. There's plenty of state-wide utilities in the rest the world which are fantastic in every regard including cost.
that was my interpretation (from the context) anyways... maybe i’m wrong ^^;
This changed in the 80s and 90s with deregulation. Some of them even turned into growth stocks or did other financial fuckery that made investors money at the cost of rotting out the business.
End of the day, local municipal utilities or co-ops almost always deliver a better level of service at a lower cost. Smaller scale makes field service more efficient and reduces overheads like executive compensation. One municipal utility I’m familiar with has a civil service scale for all staff — the head guy makes $140k.
No-one wants a job, or to keep their current job?
Compare how cops act (who are damn near impossible to fire and have a local monopoly) with how private security acts (who can be fired on a moments notice and who's customers are not locked to a particular provider).
And the biggest, of course, is Los Angeles Department of Water and Power https://en.wikipedia.org/wiki/Los_Angeles_Department_of_Wate...
PG&E seems to be particularly bad in terms of upkeep and frequent power failures. That is, the unintentional kind, not like the shutoffs this week.
Maybe a natural monopoly would not have developed statewide. Maybe the government makes a bad situation worse by picking the wrong winner before the market decides. Maybe the government should allow competition if cities and/or private businesses are willing to give it a shot -- it works pretty well where I live.
This overt dereliction can only come from an entity that is "too big too fail". If this were one or two cities' local power companies, it would not be national news. Residents of those communities would be showing up at city meetings going "wtf", and could easily point at adjacent communities not doing such things as a reference.
This is really a situation where bad incentives were created (because as far as I can tell the California state government is pretty bad at doing good incentives in general) and then people are surprised when they lead to bad behavior...
I’ve since left the Bay Area and now have SMUD. They come out every year to check brush and trees. This past summer they cut down a dying tree on the property at no cost. The arborist they sent out was extremely knowledgeable and didn’t mind that I took his time to ask about native replacements, etc.
In the 2000 blackouts the state forced us to have rolling blackouts out of “solidarity” even though we had access to plenty of power.
Unfortunately the phone system was sold off in the 60s, the cable plant in the 90s, and the animal shelter a few years ago. Short sighted.
I remember some time ago when the cogeneration plant failed and took out power in one building where many biological samples were stored at -80.
If the system wasn’t designed for the switchover from the beginning, I can see how it would be considered impossible/not practical to do. Especially when this type of “blackout” was thought to be rare.
I’m sure the prior assumptions would be reconsidered today.
Not many? As in, I don't think many buildings would have two separate power feeds. I'd expect a small number of very large connections from the cogen plant and the outside word into local distribution network(s), and then everything is single-path from there on.
In both cases, they would flip over their home (or a sub panel) to the alternate power source (and disconnect it from the upstream grid). I looked into these systems and the natural gas ones could do this automatically using a kit available at Home Depot, and the gas generators required a manual switch, but were still very simple.
I looked into Solar in California and when I found out that most people can't use their systems during blackouts I asked about these generator systems that switch over and they basically said they had not seen equipment that could it and they didn't believe it was allowed anyway.
> Most anybody would spend $500 extra on a 30k system for guaranteed power during blackouts.
To be clear, to get "guaranteed power during blackouts" out of solar, you need both a more expensive inverter ($1k or so last I checked) and batteries ($5-15k last I checked). If you're OK power only when it's sunny _and_ a blackoutm you might be able to do just the more expensive inverter, maybe. I'm not actually sure whether that work without a battery sink for your unused current.
What happens to unused current now when the panels are disconnected during a blackout?
But if you don't disconnect them, now you are maintaining a voltage across whatever your load is, but that voltage depends, for solar panels, on the current being pulled from them, as I understand. And you need to maintain the voltage in some nominal range. Maybe it all works out, but I'd have to do a lot of digging to be sure...
As to battery requirement, I can't see how attaching a wire to a panel would cause it to burn up. The wire would charge and then it would be the same thing as if the panel was disconnected with the same amount of latent charge in the panel.
Basically, it can work, but the whole system is going to need to be engineered for it from the outset.
For a blackout, the diesel generators kicked in (and could completely power the platform).
Emergency power would be expected to work independently of mains power.
The most likely limitation of a typical power plant operating disconnected from the grid would be if it could “black start” or not. Cogen plants would normally have this capability. Essentially the problem is whether you have enough power available from backup generation to bring all of the plant auxiliary systems online and get the generators initially started and producing power. Once started you set the governor to maintain 60.0Hz (if in North America) and begin slowly bringing on load so you can ramp the generation to match it.
The easiest solution to a future blackout would be to match the load in your island area to the output of your plant and then disconnect from the rest of the grid before or at the time the blackout occurs. This way you have avoided the black start situation completely. The blackout detection, disconnection, and load balancing steps can be automated to occur nearly instantly if the system is designed well.
The main issue for Berkeley is that the cogeneration plant doesn’t seem to be sized to accommodate their islanded load. This is especially problematic if you lose a generator while islanded because then you will have instant frequency collapse if you were already on the edge of your plant’s capabilities.
Cogeneration is designed to produce heat (in chillier climates) and the process makes electricity as well while doing so, because why not?
* https://en.wikipedia.org/wiki/Cogeneration
Emergency / standby generators do nothing most of the time, and only kick in when needed:
Washington University uses about 376,394 kWh/year. [1] That's about 7,841.54 kWh/week. Washington pays about 13.1 cents per kWh for power [2]. Running on the electric grid alone this would cost $598.59 per week. This isn't what they pay though, because they offset much of this usage with renewable energy like solar panels.
That would take a 50kw generator to power. Assuming they don't power all systems during a blackout they could probably get away with a 25kw generator. For argument, let's assume a 50kw generator full load would probably use about 4.0 gallons of diesel fuel per hour. [3]
At $3.64/gallon for diesel [4], at 4.0gallons/day, for one week the university's fuel cost would be $2,446.08.
To recap, the university would spend $2,446.08 to generate electricity they could purchase for $598.59 from the electric company during a 7 day blackout at full capacity.
[1] https://www.electricchoice.com/blog/25-of-the-most-energy-ef...
[2] https://www.bls.gov/regions/mid-atlantic/news-release/averag...
[3] https://www.dieselserviceandsupply.com/Diesel_Fuel_Consumpti...
UCB has a 26MW gas turbine.
You're welcome to continue retracing my steps to correct me though.
Otherwise I routinely see single racks pulling 15-25kWh albeit vastly overloaded. But that’s the norm now.
> the school has completely replaced their lights with more energy efficient, low wattage bulbs that save an average of 376,394 kW hours per year in energy
[2013] https://www.powerengineeringint.com/2013/10/14/university-of...
WHAT?
Here in France I pay 0.138c (€, thus 0.15c in $) for 100% renewable elec. That's like twice as much in CA... for what I guess is not environment-friendly by a long shot?
I'd really have thought with the USA fracking their soil like it's apocalypse tomorrow, and pulling out of any carbon commitment on top of that, at the very least it would yield considerable short-term domestic advantages, i.e. cheap elec. I'd have thought you guys paid like 5 or 10c kWh. Next thing you're gonna tell me you pay more than $1.2/L ($4.6/gal) of diesel fuel.
Actually just checked, diesel fuel is $3/gal, regular petrol slightly less in general (~$2.5) but more in CA.
Still baffled at the cost of electricity.
You'd be wrong. CA has one of the cleanest energy mixes in the country, arguably among the cleanest in the world for it's size:
Way to go, CA. Really. I imagine it's not exactly easy given the 'context'.
Lots of valid reasons though, I don't know that we've been particularly bad at this. Nuclear fission is just not the insanely cheap producer economic theory would suggest on paper, at least not in all configs/countries.
It could be 0.05c/kWh less and still be an economic solution.
Probably the disparity is in part because of how likely it is in CA for power lines to burn down entire towns, thanks to a dry climate with many large forests adapted for fires; CA is so naturally fire-prone that some native Californian tree species, like redwoods, require large forest fires as part of their reproductive cycle. So more line maintenance is required (although PG&E has probably not done enough...).
Thanks for explaining!
There are plenty of native plants that need fire to reproduce, but they do better with smaller fires that produce heat to open seeds but don't damage topsoils.
The large fires we have now are the result of climate change, long-term fire suppression, and unsustainable forest management. It's not normal for California, even though there are fire-dependent habitats here.
I agree that CA is much more at risk of fire now than it naturally is, thanks to short-sighted human endeavor; I was only meaning to point out that it isn't correct to compare CA electrical rates to France as a measure of U.S. vs French rates, since CA's are so much higher than the U.S. average due to fire risk.
I’m guessing the problem is in the capital cost of having a generator system installed. Getting procurement, university real estate/capital projects, a general contractor, and any deans/admin involved to build the system.
I have just assumed that all science buildings had backup generators by default. Movies may have tricked me. I think all hospitals are required to though, which makes sense why the researcher moved his metabolic stuff to UCSF
> Throughout the rest of the campus, the school has completely replaced their lights with more energy efficient, low wattage bulbs that save an average of 376,394 kW hours per year in energy. Their overall lighting plan has saved more than 20.6 million kilowatt hours in total.
GIGO
Can't delete it now though!
The 376,394 kWh/year is the savings they got from going to low wattage bulbs.
"Throughout the rest of the campus, the school has completely replaced their lights with more energy efficient, low wattage bulbs that save an average of 376,394 kW hours per year in energy" [1]
And your prices are from the Washington-Arlington-Alexandria municipal area, not the St. Louis area in which Washington University St. Louis is located.
[1] https://www.electricchoice.com/blog/25-of-the-most-energy-ef... #7 on the list
[2] https://www.bls.gov/regions/mid-atlantic/news-release/averag...
Seems odd though, since one of the things I thought was always touted about our municipal power plant growing up was that they could power the town if external power got cut off (But didn't normally because our generators were more expensive to run)
This whole situation represents a loss of "social technology". Just as material technology is a bag of techniques for organizing matter into useful configurations, social technology is a bag of techniques for organizing people for a useful purpose. When a society loses a technology, be it material or social, it loses a capability. It becomes unable to do what used to come easily to it. For 120 years, California was able to keep the lights on. Now it can't. How is anyone supposed to not see this situation as a kind of foreboding regression?
This new power grid unreliability is far from the only example of our struggling to do something that came easily to us 20, 30, or 40 years ago. We've definitely lost something, although it's hard to pin down exactly what.
The wind that shut grid blew at 10MPH. (Granted, it was forecast at up to 45MPH.) Did we not see winds of this strength during the 120 years* that we didn't shut down the grid over human coordination failures? If PG&E had to shut down the grid to avoid fire in light to moderate winds, it was because the people operating the grid were unable to coordinate the resources needed to operate safely under these conditions. What we've seen here is a breakdown of human systems that used to work reliably. We used to be able to operate an electricity grid reliably in moderately windy conditions. Now we can't. That should worry everyone.
* Yes, I know about Enron. That we've had two grid failures in 20 years due to human-coordination breakdown is a huge red flag.
Edit: phrasing, note that the winds were forecast to be stronger than what we ended up seeing
> NWS forecasts call for winds to be slightly weaker Thursday throughout the Sacramento Valley than they were Wednesday, but a wind advisory remains in effect through 6 p.m. as gusts between 40 and 55 mph remain possible.
[1] https://www.sacbee.com/news/california/fires/article23595593...
The article goes on to mention forecasts of 35-45mph gusts. I'll edit my original post. My point stands: whether we're talking about 10mph winds or 45mph winds, we're talking about weather within the normal range of variation and conditions that electricity grids elsewhere handle admirably.
California has very different conditions than most other places in the US. It gets no rain for 9 months a year followed by a wet winter which results in a lot of vegetation growth. And that pattern is getting more extreme. You can't compare that to a wetter place, or even to a much dryer place with little vegetation.
How can you be so sure that many major fires in PGE territory over the last 120 years weren't caused by utility equipment? And why do you assume that conditions haven't changed over the last 120 years in a way that makes fires more likely or the damage to humans more extensive and expensive?
I'm not sure if public outcry was a factor in their decision to not follow through with the shutdown before, but you can bet that it isn't going to be a factor now.
The current outages are being triggered by similar weather conditions, https://www.spc.noaa.gov/products/fire_wx/fwdy1.html
A society is a complex machine. There are thousands of interacting incentive structures, contractual obligations, and soft expectations that combine to allow us to accomplish more than we can do alone. For a long time, this machine operated smoothly enough to provide the power grid with the maintenance it needed to operate safely in 10MPH winds. The social machine broke down, so the maintenance stopped, and because the maintenance stopped, the grid became unsafe to operate.
Why did the social machine break down? Why did nobody fix it? Why are we having so much trouble operating this machine smoothly when our ancestors were able to keep it going for over a century?
If anything the social machine that appears to be broken is how we're apportioning liability. Electrical grids, even well maintained ones, sometimes throw a bit of sparks or start small fires. No amount of tree trimming is going to prevent fires if strong winds blow dry flammable brush across high tension lines.
You can find zillions of videos on the internet of power lines going down and sparking small fires... but in California if a line goes down in the wrong place and in the wrong weather it causes tens of billions of dollars in damages.
It's not clear to me that its economically possible operate an electrical utility under the risk that the smallest spark could result in being responsible for billions in damages.
The notion of collective good. The government may not have been drowned in the bathtub yet, but the idea that there's any higher purpose to humanity than maximized profit extraction for the rentier class appears to have won.
This seems to be the hindrance for most progressive changes in California. Housing is quite similar. I almost feel like some people are intentionally making the situation worse so as to provoke a far more radical change rather than allowing problems to be solved.
I'm finding it hard to put this idea into words in relation to a kind of 'civilization-scale', but the analogy is the disappearance of the 'killer instinct' after a certain time dominating some field of competition. Maybe not even that, just the blind spot of being the one sitting on top of the mountain.
Looking at how the winners are doing things in order to best them. Once you are the winner, the whole methodology flips on its head at the point of realization of 'we're now the best'. There's an amount of legacy baggage that comes along with it that's very hard to dispose of.
X got us to the top, so let's keep doing X forever since we've got our whole structure aligned to the methodology behind X. Inevitably, the things that made X the right answer at the time will slowly change, and that change won't be noticed, or accepted, until it's too late, and the entity doing Y ends up besting the entities doing X.
I'd say 70-90 years is starting to look like the 'world superpower' transition period.
Once infrastructure starts failing (bridges, apartment buildings, power grid) in ways that would get politicians fired and the responsible business directors given jail-time a decade or two ago, and yet none of these punishments is even considered likely in the current state of the world, then we're getting to the end-game. It's a failure on so many levels. It's the spiraling failure of 'X'.
I really wish the PUC wasn't captured, otherwise we might be able to fix this problem.
Edit: this is not an anti-government rant. There is far more vegetation in the wildlands of California than is natural due to forest fire-fighting policy dating back 100 years.
I'm sure someone will point out that CA does controlled burns on occasion, but it's not the same thing.
Edit: here's some reading on the topic: https://www.npr.org/2018/09/27/649649316/fire-ecologists-say...
The article is good. But podcast itself is also good.
Interestingly, controlled burning was a common tradition in native american cultures. Maybe we should've listened to the people who lived here for potentially thousands of years instead of oppressing the fk out of them to near genocide levels. But I digress.
Bad policy is a contributing factor to these fires. The Paradise Fire’s destruction and human toll was exacerbated because Paradise implemented “road diets” — squeezing critical evacuation and first responder routes. [1][2]
It’s a problem of policy being made by people that have no concept of unintended consequences: environmentalists opposing proper forest management as well as creating public safety risks by narrowing roads, “calming” traffic without the common sense to realize that roads serve an extremely valuable public safety function and that many so-called environmentalists promote policies that feel good to the simple minded, but actually do more harm. [3] It’s like the people that oppose hunting — they think they are saving an animal but they end up causing greater damage to the ecosystem because hunted species such as deer end up overpopulating and starving to death. You see similar effects with wild mustangs. People fiercely oppose culling herds even when not doing so results in mass starvation and disease that ends up harming more horses. The idea of shooting a wild horse gets “environmentalists” up in arms because they can’t see the forest for the trees.
My point is that forest and environmental management should be done from a position of rationality, reflecting reality rather than some emotional response. Examples: laws against cutting trees over a certain diameter without a time consuming permit process from CalFire. Opposition to logging, even when done responsibly. Protecting of trees when a spotted owl is in the area — even if those trees and associated underbrush become extreme fire dangers. More owl habitat is lost from an uncontrolled fire than would have been lost by properly managing it in the first place. You rarely see this sort of bad policy in other states even though other states do have wildfires. Californians do care deeply about the land — we want to preserve the beauty and majesty of the landscape — but some of us have no idea what we are talking about — we respond to emotional pleas rather than with informed maturity. It’s like having environmental policy designed by well-meaning, but naïve teenagers.
[1] https://www.latimes.com/local/california/la-me-camp-fire-dea... [2] https://www.latimes.com/local/california/la-me-ln-paradise-e... [3] https://thefederalist.com/2018/11/16/misguided-environmental...
Enron used the same tactic in 2000-2001. They took advantage of power laws to cause rolling blackouts in California in an attempt to raise prices.
Look how they turned out.
Just like our sales tax rates, income tax rates, our gas prices (due to taxes), sanitation costs, ad nauseum. We pay excruciatingly high rates for everything, and get mindbogglingly low returns for the money.
CA gov. and utilities are a textbook case in poor management and government out of control.
Where does the money go?
[1] https://www.chooseenergy.com/electricity-rates-by-state/
But in terms of experiments, if you want your MSc or PhD then you better make sure that you can go a couple of days without power to the grid.
If you are in a rural area, especially a previously "white" area as the pre-1994 NP government called it, then you need to have your own > 5000 L water tank. The town where I went to primary school had 9 days without water or electricity a year or two ago. Probably half of the town (it's a small town though) never have water between 9PM and the next morning since they simply turn of the pumps (whether there is electricity or not). The government officials also steal water to go sell for $1 per 100 L to areas where there is no running water.
You can think of it as a kind of osmosis between previously affluent areas and areas that had been mud huts. If you think of it that way it doesn't sound that bad, actually, but I think the net position is not the issue, the real issue to me is the corruption and apathy within the government and the rate of improvement. Africa should be ahead of China in economic growth if you look at the youth percentage and potentially massive workforce.
If you want to completely redesign an entire country's power grid and make it 100% solar/wind, then South Africa is the perfect candidate. That is if you could forget about politics for a moment.
If you have specimens that you cannot afford to have warming up, it’s inexcusable that you don’t already have a backup power source.
This is not normal.
A nation is a social construct that California could certainly fit into. Hell, Jerry Brown called it one. Examples of nations: Palestine, the Kurdish nation, Tibet.
The legal construct is called statehood. States can bind themselves into larger legal entities by giving up some of their own rights. Examples include: the US federal government, the European Union, and the United Kingdom.
I don't know anything about Jerry Brown but he doesn't sound very bright.
https://en.wikipedia.org/wiki/Nation
I studied these topics at a university level. A nation is not a legal entity. Sovereignty is a legal property, and it can be applied to nations.
The stable community is the USA. We share highways, military resources, language, government, traditions, laws, etc. This is a ridiculous argument.
I highly disagree with this sentiment. Very few of the US institutions are well designed, and I suspect that there will be a decline of American hegemony worldwide.
Jerry Brown was Governor of California, for four terms, and not knowing who he is in this context seems rather ignorant.
"Nation" has become colloquially synonymous with "nation-state", which lead to your confusion. But the legal aspect is from the "state" part, not the "nation" part. There have been historical states (eg. the Holy Roman Empire, Hapsburg Austrian Empire, Ottoman empire) that were not nations, and there are present day nations (the grandparent mentioned some: Palestine, Kurdistan, Tibet, as well as Basques, Zulus, and others) that are not states. Some present day nation-states (eg. Germany & Italy) were nations since medieval times but did not achieve unified statehood until quite recently (1871). The U.S. started as a state (technically, 13 states and one federated government) but did not really become a nation until after the Civil War.
If you can convince other people to identify with you, you absolutely can found your own nation. It won't be a state, though, so while might have your own customs good luck on having your own laws.
PG&E is big, but it doesn't cover the entire state. For example, much of San Diego is served by SDG&E, much of LA is served by SoCal Edison, and several larger LA satellites like Riverside (~400k pop) have their own city-managed power utilities.
edit - I'm looking this up and having trouble finding evidence of this outside of AB 1980 (proposition to break the state-sanctioned monopoly and allow competition among energy suppliers), which was from 1996
This obviously doesn't pan out when you've got a monopoly or captive audience. Around me we have a choice of energy retailers, so we get to deal with the "efficiency" of several competing bureaucracies instead of the "bloated" government one.
Politicians love it because it abstracts them from any accountability to the people, even our left wing parties are pro-privatization these days.
A private company would eradicate it since its effecting the bottom line, but public companies may choose to ignore because the people stealing may be a voting bloc to curry favor.
It may not be the case in US, but you are trading one set of problems with another and that is the best case.
Say what? PG&E diverted money earmarked for safety to executive bonuses. Is that not graft?
I’d predict it also tied up the collective conscious on funding its own mundane internal issues
We tried nationalizing in Europe, within a capitalist framework: it just doesn't work. The TL;DR of 70 years of that experiment is that the State (gov and legis) is an extremely bad steward of any business endeavor; whereas lack of competition basically means no innovation and rent situations (it becomes sort of a tax in disguise to just hike prices or keep them stable when they should decrease).
Most unregulated big markets devolve into some kind of cartel after concentration, though, so that's yet another dead end after a 50-year cycle generally (telecom shows it well, TV as well, construction too).
What seems to work, for now, is a sort of regulated hybrid framework:
- networks and transport (of elec, data, whatever) is opened to a few companies, which can only sell access to said infrastructure to other actors, not market it to end clients.
- classic "operators" thus rent infrastructure capacity to these major actors and sell it to customers.
This allows very small actors to emerge (like "virtual" ISPs and telcos, we now have very small energy companies all over Europe). It's a recent development but these new startups provide great customer service and modern end/site equipment (easy to monitor, plug to some IS, etc).
The infrastructure caretaker status is typically "slightly public", with a non-profit mandate to maintain the infrastructure and develop it, using influx of for-profit investments by the operators, some citizen oversight, special status by law, etc. Basically, we try to make sure it costs as little as it should (it doesn't always work, rails are in a very bad situation for instance, and the caretaker is so deep in debt it's a failure, but the oversight of that dates to 100% public times with government-appointed CEOs).
TL;DR / conclusion: find the right medium between "private/unregulated" and "public/monopoly". Both are disastrous, but there are sweet spots in-between. Note that I have no idea how to pull off such change in the US, elec or telco markets are simply too entrenched / congress too corrupted it seems. We have that problem too but currently we're circumventing most democratic systems (whether citizen opinion or lawmakers corruption) by invoking the EU: “we must do it, no choice, it's EU law now”. That works most of the time and is the reason why the EU is so hated yet so quintessential. I'm not sure the Federal level would carry as much authoritarian weight in US States.
Well, even if the mission isn't too difficult, the prowess of being a great counterexample should be noted and praised.
Now if you guys could just write the book so we can export your ideas... ; )
Once it does that, we'll come to appreciate their service as much as we appreciate the schedulekeeping of the MUNI.
It’s more likely that the more significant incentives would be other things, like, oh, not maintaining equipment and eventually causing wildfires.
Most electric utilities in most states aren't regularly given insane contradictory legal directives that can only be reconciled by cutting power. This is not normal.
Regulatory capture is a thing that happens and is a problem, but this seems to be a case of bad regulation, not regulatory capture.
This is not true. PG&E have been returning phenominal divideds. They have chosen to prefer to dump their asset management budget into dividends.
https://ycharts.com/companies/PCG/dividend_yield https://ycharts.com/companies/SO/dividend_yield https://ycharts.com/companies/DUK/dividend_yield
Bullshit. PG&E redirected around $100 million from their safety budget and spent it on corporate bonuses instead. While hopefully not part of their safety budget, PG&E spends about $400 million annually on stock buybacks. PG&E has plenty of money to prevent this bullshit but is more concerned with executive compensation.
I mean, for fucks sake, Caltrans had to threaten to shut down the Hwy 1 and Hwy 24 tunnels before PG&E thought about bringing in generators (not even leaving enough time to install them before the power was cut).
Regulatory capture is a thing that happens and is a problem, but this seems to be a case of bad regulation, not regulatory capture.
Who lobbied for the regulations? Hint: PG&E.
But I wouldn't count on it :)
Source: https://www.vde.com/de/fnn/arbeitsgebiete/versorgungsqualita...
Even then, one of the sources did describe that they have an e-power plug in their lab, for the -80 freezer. That's pretty much how it's supposed to be. If they have additional important fridges, those should also be plugged into e-power.
I'm sure it's possible there are some labs at Berkeley that may be compromised in backup power, there are a lot of old buildings and infrastructure there. That still doesn't mean that that experience is common, just that the people impacted are complaining loudly. Most research labs may be totally fine, and just waiting for things to come back.
Temperature of ultrafrozen samples raised to -75 Celsius and years of hard work and a collection of DNA reference samples for agriculture were about to vanish
Clearly it's a big inconvenience but they're not in danger of losing any grants.
In the UK there is National Grid Electricity Transmission plc, which as the name suggests is responsible for the transmission infrastructure across the country. They are a publicly traded company, and the parent company owns electricity & gas distribution in the UK, and some in the US. There are then smaller companies who are responsible for maintaining more local infrastructure (as I understand it's still owned by National Grid, but they are responsible for fixing and upgrading it).
As a consumer you can choose to buy electricity from a long list of companies, and easily switch between them. All that happens is a different company collects your meter reading and they send you a bill. A lot of these companies also own generation facilities (e.g. SSE, who are also publically traded) while some are just resellers (e.g. So Energy).
I've always thought that there must be hidden pitfalls of selecting a different supplier, since National Grid would have the ability and incentive to tilt the playing field.
Portable generators are so inexpensive that I'd think an institution like this could just keep some in a shed or garage, and shlep cords through windows for extended outages. I bought a top-of-the-line tiny Honda Inverter generator so I could keep my wife's breast milk frozen in case of an extended outage, but even something half the cost would be good enough to run a refrigerator.
But, even then, if you plan for it, backup generators and Solar + Battery are very affordable!
I recently installed a Powerwall purely for backup. Because I have solar, it's slightly more expensive than a standby generator, but with no maintenance or fuel I suspect it'll be a wash within a few years. (Backup generators need expensive maintenance every year.)
Even Generac bought a Solar + Battery company because they know they can't compete, and their solution is nicer than my Powerwall! (The Generac solution includes a critical loads panel inside of it, unlike Powerwall where the critical loads panel is outside.)
But most importantly, it's what didn't happen: the state of California did not bail out or grant PG&E immunity from these lawsuits. PG&E settled with the insurance entities and has since filed for bankruptcy protection, since the possible liability is larger than their market cap.
The blackouts are a way to minimize further risk and liability. But I would also guess that the intensity of blackouts is a not so subtle middle finger by the utilities to the state government for its resigned treatment of these issues.
Because Jeffrey Skilling just got out of prison and he wants to hold California’s energy ransom for payment to a random bitcoin address
Warmer yes, but dryer definitely not. Earlier in the summer there is still moisture left over from the previous rainy season. By mid fall it is much dryer, and the risk is increased by offshore winds coming from inland.
and the answer is.... individual PIs optimize for the easy case (that power stays on), not the disaster recovery scenario. There is no SRE thinking at the individual PI level.
Power Companies circa 1900 there were lots of power companies, I've seen pictures of urban areas where there were multiple circuits run on poles, how many companies tried to serve a certain area, I do not know. Through lobbying investor owned utilities (IOU), or in most cases a single IOU gained the rights to serve an area exclusive. IOUs concentrated, for the most part, on serving dense load concentrations. As such the US Govt implemented the Rural Electrification Act, so low density areas could be served. The 2 types of electric utilities are public power (municipal, REA, RUS, PPD, UD, etc) and for profit (IOU's typically). In most cases, the IOUs give something to the government in exchange for the IOU providing service exclusively, it might be a tax or it might be free street lights. Municipals tend to subsidize the local government in some way, either through returned dollars or free power (street lights, buildings, traffic lights). IOUs typically have a defined rate of return and are supervised by a governing entity of some sort.
Vegetation Management The recent SERC compliance meeting had a good presentation on vegetation management from the utility perspective and another on enforcement trends. In my opinion a lot of the issues in the W US is the result of the policy 'no fire is good', the sand pile game I think illustrates the issue where the longer sand keeps from falling results in a larger collapses (see Yellowstone fire). Since the late 1980's the issues with not burning has been known, I had an ecology class where if I recall correctly that was discussed for a couple of days. Tree trimming and clearing out undergrowth is done on a regular basis when the utility has an easement, but especially in urban areas folks tend to plant trees too close to power lines or even worse encroach on the easement with buildings. Most utilities patrol transmission lines at least once a year if not twice or monthly, sometimes this is aerial and other times it is feet on the ground walking the line. As an aside, the NESC governs clearance of electric lines to stuff and how stuff should be built; the RUS has publications on line design if you want to read about it.
Distribute Generation The electric grid in the US is divided into 3 areas, Eastern Interconnect, Texas and Western Interconnect; they all function essentially the same. If I have a generator connected to the grid, it has to synchronize to the grid before closing the breaker. If it is done correctly then there is very little mechanical stress on the generator, if done incorrectly then there is a large amount of mechanical stress on the generator. One mis-operation I know about involved the A and B phases being swapped during a re-wind, when the generator was closed in at commissioning it had a large bang/clunk and the breaker opened immediately. The generator then had to be examined, i.e. taken back apart, to figure out what went wrong and if it could be put back into service.
If I have a generator partially supplying a facility (this can save a lot of money for an entity) and a fault happens on the grid then my goal is to protect the generator, so the generator will either shut off or island the facility while shedding load above the generator's capacity. This happens very quickly. One instance I know of, the urban area was supplied by transmission (aka remote generation), a 30 MW generator was the closest source, the utility had a fault because equipment misoperated and the generator was suddenly trying to supply all the power to that fault such that the generator protection operated and islanded the facility. It was no issue to close the grid interconnect back in (once it was ensured it was safe to do so) but the facility had to shed load to keep the generator running without causing electrical issues to load and damaging the generator.
Once a facility is islanded and running on its own generation the phase angle is a don't care until it is time to synchronize back to the grid. As long as the facility can shed load to maintain frequency (there is a NERC standard on Under Frequency Load Shed if you want to read about it) and not ruin equipment by having a power quality issue. During dynamic studies for system stability, it can be observed that a generator will diverge from the system frequency phase angle but not trip off because it is isolated from the grid which requires verification that isolation is happening and the protection scheme will indeed work that way on the actual system.
My observation is that most facilities, data centers and other processing facilities (refineries) tend to be the exception, concentrate on first costs when designing their electric infrastructure. It is possible to design a resilient system but it has a cost and it will not be utilized 100% until something goes wrong. And if you are doing research then that can be an issue as you may lose a large amount of data due to the power going out or possibly being sensitive to transients on the system, e.g. a switching operation on the transmission system affects the end user equipment. Even if you have redundant systems (and/or power supplies) it is possible to have single point of failures on your system. As well if you have enough local generation to supply your load, it may be more economical to not run 100% of your generation as the market price for electricity is cheaper than your cost of production (and there are folks who don't like idle assets, not realizing the greater benefit is not using it or only having to use it infrequently).
One other aspect of distributed generation is the automatic separation of the DG when loss of voltage is detected on the grid side. Utilities do not want voltage on their system if they have an outage due to worker safety (and other reasons). Utility crews in hurricane areas will typically investigate if they hear a generator running when the power is out to an area to ensure it is not back feeding the distribution line. As a reminder keep your feet together if you are near a downed power line and hop away, or even better don't go near downed power lines.
They can't afford another lawsuit like that one. They couldn't afford the one that already happened. So they are taking steps to prevent it.
All because of a law suit. Great job California.
Sadly you need to plan for these well in advance and be willing to sink significant resources.
Possibly this is a case of California climate biting them. A lot of the big research universities in the Northeast have steam plants, originally for heat, but often with cogen facilities as well. Usually not much, a meg or two, but it is something.
I mean, I'm used to IT in general, and security in particular, being treated as a cost hole and given as skimpy as budget as possible and then some until something breaks. But at least that's more intangible, and I guess from a cold hold MBA point of view it's not as if most places are really punished that hard for security breaches unfortunately. But power loss and having valuable work/material lost every year seems like it should be more tangible.
It's hard to imagine a better advertising campaign for those products.
> Tesla factory unlikely to lose power even if most of Fremont does because of agreement with PG&E
https://twitter.com/PBRStreetGang7/status/118178393541343641...
( disclaimer: might be usual twitter humbug https://www.smh.com.au/business/markets/the-crowd-sourced-so... )
Who's doing the inspections, who gets contracted to clear brush, who gets power, who doesn't, etc.
It's kinda like MUNI, just not quite there, but I don't doubt Sacramento can get them to EDELCA.
To me, there feels like a big difference between a reliability SLA and an exemption from what is supposedly a vital emergency shutdown.
What 4th option are you criticising? Unless you're saying that if they don't immediately throw up their hands, try to do nothing, and go home when the power goes out, they're not "accepting the consequences"?
What ever happened with Bloom Energy? they don’t get much press. https://www.bloomenergy.com/
Not saying there aren’t other options, just the one most businesses have opted for, and has the best ‘vendor’ support (plenty of fuel companies to deliver, rental agencies to arrange the delivery, and companies that Veratis that setup the building side needs)
I’m merely talking about situations where everything is fine except for the occasional brownout or blackout.
We are trying to move towards greener electricity. Wouldn’t it make sense for parts of California, for example, to generate some of that electricity locally?
I cannot recall a single power outage in my home city that lasted longer than ~three hours. And even those "couple hours" outages are extremely rare, much less than once a year I would wager.
According to the statistics, the expected unavailability in my country is around 0.002 - 0.003 %. That's hardly something I would describe as a "normal thing" that "can happen at any time". That's a really rare thing.
The "SAIDI" value here is under 15 minutes. Meanwhile the median SAIDI of the US is ~1.5 hours. That still doesn't sound like a common thing.
Likewise the 2000 crisis where LA's DWP customers had power when PG&E did not.
Although I'm sure it makes me a "socialist" for pointing these things out, it almost seems that having private monopolies in utilities is a bad idea
Although not as widely reported, SoCal Edison is also doing rolling blackouts in at-risk areas https://www.latimes.com/california/story/2019-10-10/l-a-face...
In particular, if you only take into account the revenue made on a given day vs the likelihood of a lawsuit and nowhere is economic damage accounted for, then you will turn the power off a lot more often and a lot more widely than you should.
Also, since SoCal Edison is geographically adjacent to the DWP but Edison is investor owned while DWP is municipal owned, I believe you are adding support to my point.
However, to your point about LADWP, I think you're making an overeager classification. It's not a matter of being "geographic adjacent", but being liable for the infrastructure that would start wildfires in at-risk regions. Take a look at the difference in coverage areas of Californias utilities: https://ww2.energy.ca.gov/maps/serviceareas/Electric_Service...
edit: it is tangential to this issue, but LADWP does not have a very glamorous history, either.
No this is solely PG&E's doing. When you write the regulations, you own the blame. PG&E claims it's too expensive to underground its transmission lines (which would prevent fires like these) at a bit over a million per mile. Meanwhile they took blew their safety budget on executive bonuses. You want to tell me that undergrounding around 100 miles of lines isn't going to reduce the risk of fires? How about undergrounding 500 miles? Because in addition to the safety budget, PG&E also spends around $400 million annually on stock buybacks.
I don't particularly care if PG&E remains a private company, but there need to be consequences like jail time for actions that go far beyond mere negligence.
But PG&E are not the only offenders here. You seem to be arguing that there's regulatory capture (correct me if I'm wrong), but the Public Utilities Commission still exists, and they have the authority to regulate PG&E. If the commission is neglecting its duties, then it should be restructured as well. A president of the commission has been forced out before, and it should probably be done again. If they neglected such an obvious case of greed and mismanagement because of bribery (sorry, lobbying), that should be a criminal case, too.
Public officials can throw PG&E under the bus all they want, but if systematic changes aren't made, we'll be doing this song and dance again in a few years with a different utility.
While this combination is more likely in a desert, it's not true of all. For instance, the Salt River Project that oversees electricity transmission in Arizona switched to building underground lines in the 1970s. https://www.azcentral.com/story/news/local/asked-answered/20...
Uncharacteristic edit: Why the downvotes? It must be because you took the comment seriously and think the sentiment grossly naive, or heard the skepticism and thought it unwarranted?