Hacker News Reader

Comment by adrianN | Hacker News Reader

The technology on the NYC subway is so old that they have to pay through the nose for replacement parts. I also assume that you have a hard time finding people willing to work at a place where they learn few transferable skills, because nobody else uses century old stuff. Especially if the pay is so-so and you don't guarantee a job for life.

Yet upgrading the system to modern signals and interlockings is even more expensive, especially since maintenance is extremely hard when the system has to run 24/7.

View on HN

"I also assume that you have a hard time finding people willing to work at a place where they learn few transferable skills"

You assume wrong; the MTA has no difficulty finding people to work on the old equipment (I have a close relative who actually works on this equipment, there has never been trouble finding willing workers). Also note that the old technology is mostly restricted to interlocking machines and telephones; everything else (train sets, radios, MOW equipment, etc.) is much more recent.

Signals are expensive? That kind of blows my mind. A chaveut dj LED light that responds to dmx commands can be had for a few bucks

Signals are expensive because of their safety guarantees (and because of lack of competition). That LED light you can buy for a few bucks doesn't guarantee eight nines of uptime if you keep to the maintenance schedule. You also can sue the manufacturer if a train runs a red because the LED broke. Signals and other trackside equipment make up a large part of the costs of building rail.

"You also can sue the manufacturer if a train runs a red because the LED broke"

FYI, MTA operating procedures require trains to stop when a signal is not lighted, then coordinate by radio before proceeding. There is also a defense-in-depth strategy: the interlocking maintains 4 red signals between trains, so a collision in theory would require 4 failing signals, 4 failing stop arms (which automatically stop trains that ignore red signals) and a train operator ignoring the rules.

Exactly what I was thinking: certification and insurance

Not really, it is all because you have this crazy idea that mass transit systems should be profitable and/or self-sustaining so with the intent of getting profit they cut corners.

Most other countries have subsidized or even free mass transit systems and they realistically don't expect them to be profitable, but helping people/goods to be moved is considered more important.

It is one of those natural monopolies where privatizing stuff doesn't really work.

> Most other countries have subsidized or even free mass transit systems and they realistically don't expect them to be profitable, but helping people/goods to be moved is considered more important.

Here is a good map of "farebox recovery ratio" (the percentage of operating costs funded by fares): https://people.hofstra.edu/geotrans/eng/gallery/Map_World%20.... Quite a few asian systems recover more than 100% of operating costs, and a couple of European systems do, but no American ones. London's public transit system, for example, gets 25% of its funding from the government. The NYC MTA is at 45%.

Generally I'd agree that it's better to just let the government subsidise it (or just run it) and accept the cost in exchange for the economic benefits.

However, there is one way to make it work privately, and that's to capture some of those economic benefits by buying the land around stations. If someone buys some relatively cheap land near a city and then builds a train line from there to the centre, the value of that land increases and so they can charge higher rents on any housing, offices or shop units built there.

This is illegal nearly everywhere but China, as the rail company becomes the 500 pound, rent seeking gorilla that has dominion over large chunks of a city, or state.

No, it is not. Plenty of examples around the world, and it generally works better. Copenhagen and Hong-Kong come to mind.

Japan as well, If you go outside major train stations in the large city there are big department stores run by the train company.

https://en.wikipedia.org/wiki/Kintetsu_Department_Store https://en.wikipedia.org/wiki/Isetan#Subsidiaries

Not sure who "you" is here but nowhere in the USA is public transit expected to be profitable (excluding private coach and shuttle services, airlines, etc. as those are not "public") or even self-sustaining. The general problem is like most government services they are money pits with incompetent, corrupt management.

The US subsidizes mass transit systems. The biggest subsidies go to the highway system.

That and actually not killing a great many people, as that dj light signal would inevitably do sooner or later. Rail signaling isn't just a convenience and throughput optimization like road traffic lights are, with little consequence in the case of failure. Failing rail signals are more deadly than falling air traffic controls.

The legacy parts manufactured half a century ago have eight nines of uptime?

Relays are pretty reliable if you exchange them regularly enough. Just because the stuff is old it doesn't mean that you can't build machines with enough redundancy to make them very dependable. It's not so clear that complex software solutions as in modern interlockings are more dependable than the comparably simple electromechanical solutions of yore. Cheaper, yes, more capable, sure, but more reliable?

Not just dependable in terms of systems engineering, these things were also massively "overdesigned" to be extremely reliable no matter what.

> Just because the stuff is old it doesn't mean that you can't build machines with enough redundancy to make them very dependable.

I can do the same with 1-dollar modern parts.

That's exactly what the companies building these things do. It's not the material that's expensive, it's all the paperwork necessary to prove that it actually provides the guarantees that it does.

The context was you saying "they have to pay through the nose for replacement parts" and exabrial's downvoted reply, to which you replied "That LED light you can buy for a few bucks doesn't guarantee eight nines of uptime if you keep to the maintenance schedule". I don't know anything about this stuff, but your arguments have not been coherent.

Solid-state electronics from 1967? Eight nines for individual components doesn't sound at all implausible to me.

Have you ever looked at industrial-grade hardware? You know, stuff that is actually certified to work for years on end in the worst conditions imaginable? If someone asked you, how much do you think a 12V power supply costs, would you say $5-10, because that's how much they go for on ebay? Try a National Instruments 12V supply - it's about ~$1200 for a power supply slightly larger than your phone charger. Why? Because it's certified to work in -80C and +120C without failure, in environments where explosive gases are present and in very high humidity environments. You can't use consumer hardware in industrial/commercial environments, and for a good reason.

That was the story told by SGI execs. It was also a story told by Sun execs. And it was a story told by Unisys execs. But I'm sure it is different this time around.

I mean sure, in a lot of cases, yes, a normal iPad is a lot better than this super custom commercial terminal for $10k/each. But if you work in an oil refinery where you can't risk an explosion of millions of gallons of petrol, or if you run a metro system that carries hundreds of thousands of people a day, you can't be using consumer grade electronics.

Have you heard of this company called "SpaceX"?

Also, in case you do not actually know this, a metric crap load of this special electronics that NY metro uses have Windows 98 running IBMs inside.

Source: Neighbor MTA grunt

Running two subway lines at two nines of reliability would be better for passengers than running one line at four nines, and might well be cheaper.

A signal failure can lead to dead passengers, so maybe you want to reconsider that statement.

If you are thinking about "verified" systems spent way too much in academia.

Fundamentally, verification system is a traffic light problem. In case you do not know, the blinking yellow is "i crashed". That condition is not checked via code verification of Red/Yellow/Green state machine. It is checked via crossed green detector: a circuit that checks for a condition "green energized" on both and should that condition occur it crashes the Red/Yellow/Green state machine. That state machine will never recover until Bob shows up at a traffic box, looks to see if a sensor is dead, looks to see if box is OK and resets it by hand.

Signalling is always fail-safe - as far as I can tell this is the case even when there are e.g. multiple redundant ways of detecting train presence, a failure of any one is treated as a reason to stop trains. Something like a cheap LED that burns out more frequently is not unsafe, it just means there would be more cases of trains having to stop because the signal light was out.

It must be one of those things that's harder than it looks - China had a train crash because they decided to roll their own signals.

Agree, it's obviously much different than a lighting effect in practice. What I'm trying reconcile is the hardware and software. Are there truly that many differences physically? The cost of signals probably comes from certifications I'm guessing

Did you take any computer science courses on software verification? That is, proving your program does what you intend.

It's complicated, difficult, and the people who can do this are expensive. Railway signaling systems, alongside air and spacecraft, and sometimes consumer devices that would be expensive to firmware update during the warranty, are the few places software verification is used.

The specification we are verifying the software against must also be proven to be correct, including failing safe.

The hardware also has to perform, at all reasonable temperatures, humidities, with vibration, poor power, and for an expected life of at least 50 years.

The people installing the hardware must be checked, to ensure the work is done correctly. A crossed wire is a critical fault, I imagine there's a long and detailed procedure for connecting every wire.

[1] is an interesting read, a retrospective on a bad crash in 1988 in Britain which led to many improvements in signaling safety.

https://www.railengineer.uk/2014/01/06/25-years-after-clapha...

But most of that work can be reused, you're not building the whole software from scratch each time, so those fixed costs get diluted.

According to Wikipedia[1], there are about half-dozen commercial systems that get heavily reused.

[1] https://en.wikipedia.org/wiki/Communications-based_train_con...

Unfortunately its not, yes framework is reused but business logic part is written from scratch most of the time and it is that part where most errors usually occur. That is why you can't just install SAP or Dynamics you need to program your logic and workflows into if first, the fact that framework itself is reused saves some time sure, but error probability increases with the amount of customization framework provides.

I'm talking about software designed for railway signaling, like SelTrac, not generic business platforms like SAP.

I don't know, but presumably there's still a lot of work

- planning and then (re)locating points/switches

- planning and then (re)locating signals or balises

- updating trains

- configuring the software to reflect the track and signals

- verifying the final configuration

Seltrac will not be accepted by MTA. They have their own CBTC specifications and Thales (seltrac supplier) is engaged in a large program to create a compliant product.

The signalling system probably has an extremely small amount of software in it, and if it's old the logic will be electromechanical; if a train is in the block, the entry signal will be red, because a particular relay is closed.

The signal lamp itself is probably a small part of the cost compared to all the wiring and the time spent installing it.

Edit: googling for "cost of railway signals" produces this very comprehensive PDF on the subject: www.credo-group.com/downloads/tco_of_rail_signalling_systems.pdf

If you just look at the signals as "a little red light that tells the driver to stop", then its not that complex.

However in the rail world signals encompasses much more than that red light. There's sensors on the track which detect when a train enters a section of the line and when it leaves, and on modern systems there's communication with the train itself to force the brakes on if the driver runs a red light, making it near impossible to enter a section with a train already in it.

All those parts need to communicate with each other, and they need to be ridiculously reliable, and capable of self-reporting when they fail, because in the worst case a failure in any one of those components results in a few hundred people dying in a dark underground tunnel.

You are thinking of signals as just the lights telling the driver what to do, but the signals used in the subway are much more complex and include the hardware to know, for example, whether another train is on an upcoming stretch of track, or how a switch is set up ahead, which affects what tracks the train is routed onto. These aren't just electronic systems, they include mechanical hardware to detect the necessary information. The design of some of the signals still in use basically hasn't changed in 100 years!

Reply on news.ycombinator.com