How New York City Gets Its Electricity | Hacker News Reader

How New York City Gets Its Electricity

nytimes.com

183 pointsOsiris3051 comments

One of the things you'll probably not notice about this article, although you can be certain a lot of effort went into it, is the simplicity of language and sentence structure.

The article describes a hugely complex network of technologies, and also touches on how public policy shapes their use, and yet I bet that as you're reading, you don't struggle at all with understanding what it's talking about.

The writer, Emily S. Rueb, uses short, straightforward sentences, paragraphs of only a sentence or two, utility words instead of flowery language -- and yet the writing doesn't seem at all unrefined. It just seems natural and to the point. That's a real service to the reader, and a real show of talent by the writer.

I guess, like the incredible infrastructure that goes into supplying power to NYC, you know it's working well when you hardly notice it.

To follow up to your last comment: as you may know, the idea of an infrastructure being invisible until it breaks down is one of the major themes in academic study of infrastructure.

Gregory Bateson and Merleau-Ponty used the example of a blind person's cane as the paradigm of a tool that disappears to its user. Heidegger talked about similar notions of tools being present or absent from the point of view of perception. See p.208 of this book https://books.google.ie/books?id=U8IPBwAAQBAJ&pg=PA208&lpg=P...

Having said that (I'm not sure why I am saying it) a much more important aspect of infrastructure is the political one: does a given deployment of infrastructure reinforce existing unfair social advantages, or tend to equalize things? Fancy words like "apparatus" and "dispositif" can get you access to that discussion though I'm guessing it has a high ginger-factor for the HN audience.

For more infrastructure of NYC:

"The Works: Anatomy of a City" by Kate Ascher

I was a bit disappointed that it is so focussed on NYC, but it is nevertgheless a fantastic book with lots and lots of illustrations.

Also "Underground" by David Macauley.

As a non-native English speaker, I really appreciate this style of writing.

I'm sure many people who are less proficient with academic English will do as well.

Out of curiosity, I ran it through text complexity analysis and it came out at grade level 4.7, while a pseudo-randomly selected New Yorker article came in at 10.9

To me, this principles of this style are best described in the Economist's Style Guide: http://www.economist.com/styleguide/introduction

"Clarity of writing usually follows clarity of thought. So think what you want to say, then say it as simply as possible."

"Pomposity and long-windedness tend to obscure meaning, or reveal the lack of it: strip them away in favour of plain words."

"Do your best to be lucid (“I see but one rule: to be clear”, Stendhal). Simple sentences help. Keep complicated constructions and gimmicks to a minimum"

"Long paragraphs, like long sentences, can confuse the reader. “The paragraph”, according to Fowler, “is essentially a unit of thought, not of length; it must be homogeneous in subject matter and sequential in treatment.”"

"Get: Get is an adaptable verb, but it has its limits. A man does not get sacked or promoted, he is sacked or promoted. Nor does a prize-winner get to shake hands with the president, or spend the money all at once; he gets the chance to, or is able to, or is allowed to.

All of these suggestions apply to coding too!

I wish email writers would spend a bit more time on composition. I do realize code lives a lot longer than an email does, but if they put in just a fraction of the effort a programmer spends on structuring code, it would make email a lot better.

I noticed this too. The combination of a minimalist layout, simple stylized graphics and clear writing made this article a joy to read.

My first project as an investment banker was refinancing the turbine barges in Brooklyn briefly mentioned at the beginning the article.

The article should have gone into capacity auctions. It's an interesting process; generators participate in Dutch auctions to win contracts to be ready to provide a certain number of MWs onto the grid. The entire winning side of the market gets paid at the market-clearing price when the ISO gets enough MWs allocated. The capacity is auctioned off over three time horizons: a "strip" auction for the six-month winter period and the six-month summer period that happens about a quarter before the period starts; a monthly auction that clears a few months out; and a monthly auction that clears on the eve of the start of the month.

I was initially surprised to learn that clearing bidders are not even paid to generate; they're paid to be ready to generate, which means they have to stay inspection-compliant and periodically fire up the plant to ensure everything's in good working order. If they generate, they get paid for the MWs that go onto the grid.

For an asset like the Brooklyn barges, they might get spun up 20 days a year on a hot year, and actually put meaningful MW's on the grid for 10 of those days. All in July/August when it's hot out and people are blasting AC. The capacity market is highly seasonal.

The NYISO website has lots of data if you want to learn more: http://www.nyiso.com/public/markets_operations/market_data/i...

Its not that surprising when you account for the fact that utilities are government granted monopolies. In California, at least, the deal is that investors get to be the only game in town and in return they guarantee that the infrastructure will not fail. Depending on the size of the market, a power outage might cost the utility tens or hundreds of millions of dollars in fines, all but wiping out any profit for the year. AFAIK there are also some pretty hefty fines when the company fails to repair infrastructure fast enough, like after a car accident damages the power lines.

With such crippling fines, its cheaper for utilities to build a huge natural gas peaker plant every year than to get penalized for failing to provide service.

> I was initially surprised to learn that clearing bidders are not even paid to generate; they're paid to be ready to generate,

It is, however, a very serious commitment. If you fail in your generation requirement you can be fined significantly or excluded from future auctions.

I started reading the article specifically because I thought it was going to cover the capacity market structure.

My home province (Alberta) has recently announced it is moving from energy-only market to a capacity market. https://www.aeso.ca/market/capacity-market-transition/

This is one of the better articles I've read about the grid.

I don't work in this region, but I work for another ISO that controls one of the other grids doing a cross between DevOps and SRE on the systems that directly control the grid.

I've worked for startups (including my own) and public cloud providers, but this is the most stressful job I've had. If I fail people will quickly start to die as traffic lights and life-support systems fail.

The article touches on the complexity a bit, but it's huge job controlling a deregulated open-market energy grid. You have to scale for peak demand while providing financial incentives to support a mix of generation with widely varying prices. You have to deal with distribution congestion and natural disasters. Even generation disruptions -- a 2-GW nuke plant can shutdown at any moment. You need to deal with that deficit and quickly get something else online to meet demand.

And it all has to work 24/7 without a second of interruption.

It's an eye-opening experience. I'll never complain about paying my electric bill again. ;)

Which ISO or vendor do you work for?

ERCOT in Austin, TX.

Awesome! I've worked with y'all before. Great engineers!

> as traffic lights and life-support systems fail.

Has anyone on life support ever died from the power going out ? I've worked at a hospital and the life support machines have internal battery backups, they're plugged into a separate outlet that is (at least) got an instant-on backup generator and possibly another battery backup. Plus, the ventilators I saw have rubber balloons on them that can be manually squeezed to breathe the patient if the machine dies. I also thought ventilated patients are also staffed by critical care nurses that have at most 2 patients per nurse. I've never heard of a ventilated patient set off in a room alone without adequate supervision.

Not saying there's no risk elsewhere in a city from power loss, but I thought hospitals had this situation covered.

Probably multiple in New Orleans during Katrina. Of course, that was more than just a power outage, with flooding water destroying backup generators, more critical needs patients than non-evacuated staff remaining to support them, multiple days for support to arrive, etc.

This podcast contains a horrifying story about how life-supporting services and resources were managed when one hospital in New Orleans lost power during Katrina: http://www.radiolab.org/story/playing-god/

Things are not always properly maintained, human error happens, back-up equipment fails, etc. In hospitals I am sure it's very rare but I'll bet it happens. But not all people on life support are in hospitals...

http://www.timesunion.com/local/article/Nursing-home-death-d...

People die when the power goes out. :) It can be both accidentally and when people's power is cut off due to non-payment.

They also die from heat exhaustion when the AC goes out.

http://www.pressreader.com/new-zealand/the-new-zealand-heral...

If we fail at the grid management level and the entire grid collapses, it's a non-trivial process to get it back up. It can take hours to days to get everything up and running. It's a process called blackstart.

In the meantime hospitals and other facilities with backup gensets will quickly run out of fuel. Those fuel delivery contracts everyone has will go out the window once the entire grid is down and everyone needs fuel.

Your backup-electricity strategy has to be decently thought out. The Fukushima disaster would have been avoided had all of the switching stations for the backup generators been located in safe, or even elevated, structures[0].

[0] https://en.wikipedia.org/wiki/Fukushima_Daiichi_nuclear_disa...

What surprised me, is that 54% of the energy of NYC comes from what is arguably clean energy, if you include nuclear in the mix. That's even more impressive if you consider how long it's been since the nuclear power capacity of NYC has expanded.

Since it says that the biggest threat against the electrical supply system is trees, shouldn't it be possible to design a drone-based automatic transmission line inspection system?

I mean "just" have a drone with appropriate sensors fly the length of the lines, looking for tree branches that are threatening. Perhaps using machine vision, or something simpler like lidar/radar/ultrasonics/whatever. Signal human inspectors when something looks fishy enough, with video attached for quick review/triage.

Optimally designed to allow recharging the drone's batteries somehow from the top of the pylons (aren't there electrical fields that might be used somehow[1]?), to allow extended automated runs.

Obviously covered in rubber or something to make the chance of damaging the infrastructure in case of a crash minimal.

Crazy?

[1] No idea if this is possible but it would be super neat since it might allow it to scale to run on hundreds of miles of power lines without many humans and with good coverage.

See the "Deep Intelligence" section in this MS Ignite blog post. Lots of companies out there doing this, just a google search away:

https://blogs.technet.microsoft.com/machinelearning/2016/10/...

Why not spend the money to bury the transmission lines? At least the small ones.

As http://www.cnn.com/2014/02/12/us/winter-storm-power-lines/ explains:

"underground power lines cost five to 10 times more than overhead wires, don't last as long and cost more to replace..."

and

".... underground lines "are not without their disadvantages,... While more reliable 'under normal weather conditions', they take almost 60% longer to fix when something does happen to them."

Wait, underground lines have a shorter life?

> "Buried power lines are protected from the wind, ice and tree damage that are common causes of outages, and so suffer fewer weather or vegetation-related outages," it concluded. "But buried lines are more vulnerable to flooding, and can still fail due to equipment issues or lightning."

Does flooding happen that often that it significantly decreases their life?

I'm thinking there are two possible reasons why weather could cause an issue for buried lines, especially for lines in upstate New York feeding the city:

First, the cable would have to be buried deep enough where temperature fluctuations are minimal. If the temperature is allowed to fluctuate widely enough, the cable would be subject to expansion/contraction stress and could then fail. Overhead lines wouldn't have this same issue since the catenary shape of the wires allows room for this expansion and contraction -- solid earth doesn't.

Secondly, if it is buried deeply enough where temperature isn't much of a factor and something fails, now you have to dig that much further to get to/repair/replace the wire. Add on top of that the dead of winter and the rural rights of way that these buried lines would likely be under and that's another big mess.

Keep in mind I'm just speculating here. I'm no expert.

I worked on some small pieces of things mentioned in the article, specifically around the maintenance and reliability of the underground network in Manhattan.

There is a mix of submersible transformers and those that can handle water but can't be submersed in the city. The former are usually placed in areas that are prone to flooding, they are much more expensive than a regular transformer since they are basically waterproof and have a number of extra features built into them so they require less maintenance.

In Manhattan proper, the worst enemies of the underground network are salt + water and the build up of gases. Salt and water + eletricity = corrosion which leads to burnouts. There has been a significant move to try and preemptively visit underground vaults and clear them of the salty water after a snow melt.

Most underground cabling is enclosed in an oil filled casing. Over time these wear and leak. Those gases can fill the underground transformer vaults and lead to explosions. These events are usually pretty well documented and again, ConEdison has been putting procedures and inspection routines that help prevent these.

What is amazing about the Manhattan and Brooklyn networks is how resilient they are simply by the simplicity of design. A transformer outage in a certain area can go almost completely unnoticed by citizens because of the control centers around the city and how they will reroute power. It's something that doesn't exist just about anywhere else and allows crews to isolate issues and perform maintenance when needed.

Does flooding happen that often...

Every time it rains heavily, the water table rises temporarily, in low-lying areas. It's not a flood, but it swamps things that are underground. If an earthworm would drown, a cracked junction box would leak.

Physics, specifically their much higher capacitance. The buried lines cost more, but mainly they aren't buried because of physics.

"A typical 345,000V transmission line will be able to deliver no power when the line becomes about 26 miles long."

https://www.puc.nh.gov/2008IceStorm/ST&E%20Presentations/NEI...

In New York City specifically most distribution (not transmission) is already underground.

http://www.nyc.gov/html/planyc2030/downloads/pdf/power_lines...

Air is a better insulator

There are several vendors that do something similar with robotic inspections of underground pipes.

A robot is able to inspect (map and video) a 6" pipe better than a human can!

I knew someone who used to work at a place that built those bots for natural gas lines. Really interesting stuff. They nicknamed them pigs, if I recall.

You recall correctly:

https://en.wikipedia.org/wiki/Pigging

I think my introduction to these devices was one of the three Bond films they mention in the article, probably Diamonds are Forever.

If you want an extremely thorough look into the operation of New York State's (including NYC of course) electricity market I would recommend the market monitoring unit's State of the Market Report[1]. The deregulated markets have a lot of quirks, but I think their operation and set of outcomes is pretty interesting.

[1] http://www.nyiso.com/public/webdocs/markets_operations/docum...

if you like this, you may find this book interesting:

The Works: Anatomy of a City.

my GF got it for her post grad architecture degree and i found it really well illustrated and interesting! For example it outlines the steam grid in NYC and current vs historic uses

https://www.amazon.com/Works-Anatomy-City-Kate-Ascher/dp/014...

I thought a part of Manhattan was still coal powered -

Some 20 years ago we took a behind the scenes tour of the museum of natural history and saw huge "doors" where coal was dumped. Our insider said that the museum borders on the 1/2 of the island still powered by coal. And that 1/2 if the museum's property was still coal powered.

Btw - he also showed us the remains of the real Jumbo. Quite cool for a cub scout parent.

There's very little remaining installed coal capacity in Zone J (within the physical confines of NYC), and I believe all of it is dual-use, which these days would be running exclusively on nat gas. However, some coal-generated power from outside NYC does make its way consistently into the market.

The article says that 4 operating nuclear plants supply 1/3 of the electricity for NY state. That's impressive!

Sadly, 4 of the NY nuclear plants were commissioned in the 1970s. A 5th plant, in Shoreham, is the newest but is also no longer operating. Nine Mile Point has one newer unit from the 1980s, but all told, most of the nuclear power generation capacity in NY is 40+ years old.

Shoreham is actually way more interesting and complicated than that. The plant was never really in operation, but a deal was struck for it to briefly "turn on" so that cost recovery could be triggered and passed on to the rate base. It never entered real operation despite being completely finished.

It is, but the state is accelerating the shutdown of the major plant feeding NYC.

https://www.nytimes.com/2017/01/06/nyregion/indian-point-nuc...

Shoreham was completed but effectively never generated power. The evacuation method for an accident was "everyone drives away from Long Island" and Long Island refused to allow the plant to be turned on. The community paid back the entire cost of the plant through taxes because they had agreed to the planet before construction and effectively reneged on the deal.

Was partly a victim of external circumstances as well. It got federal approval in 1985, and might have eventually had an evacuation plan approved (over opposition) if Chernobyl hadn't happened in 1986, at which point nobody was willing to take on the political cost of pushing it through.

I live in NYC, and an important technology used here not mentioned is microturbines which can be used for co-generation and even try-generation. I spoke with an engineer that converted a Columbia Univ building from #2 oil to natural gas (Columbia is converting many/all buildings to natural gas) and used a micro turbine for the electric. He claimed the cost of conversion was $500K and the savings per year is $100K.

With tri-generation you get not only heat from steam and electric but also chilled water AC.

Unfortunately, microturbines don't get the tax breaks that solar and wind power gets.

During Hurricane Sandy we had a major power shortage below 42nd st (?) in Manhattan and everything was black except the Goldman Sachs building. There is a great magazine cover of Manhattan all dark except that one tower.

Here's the New York Magazine cover: http://nymag.com/nymag/letters/hurricane-sandy-editors-lette...

I'm pretty sure the blackout was everything below 23rd St (except Battery Park City) on the west side and below 40th St on the east side.

39th Street was the cutoff between the 14th St grid and uptown. The huge co-op development in West Chelsea between 23rd and Penn Station has its own generators, but the rest of the area was blacked out.

-signed, was w/o power for 4 weeks in West Chelsea

I have to wonder how long it will be until it's economical to use lithium ion batteries located in NYC to do electricity arbitrage. If natural gas turbines on barges are used, it can't be too long...

This explains why ConEd bill always has higher "delivery" charges in NYC per unit than actual consumption.

> "Although energy use is projected to flatten or decrease in the next decade"

This is a digression, but this has always bugged me in the Kardashev scale. Why would more advanced civilizations use so much power?

Reply on news.ycombinator.com