Trains were designed to break down after third-party repairs, hackers find
arstechnica.com
arstechnica.com
Polish Hackers that repaired DRM trains threatened by train company - https://news.ycombinator.com/item?id=38628635 - Dec 2023 (137 comments)
Polish train maker denies claims its software bricked competitor rolling stock - https://news.ycombinator.com/item?id=38570654 - Dec 2023 (2 comments)
Dieselgate, but for trains – some heavyweight hardware hacking - https://news.ycombinator.com/item?id=38567687 - Dec 2023 (292 comments)
Polish trains lock up when serviced in third-party workshops - https://news.ycombinator.com/item?id=38530885 - Dec 2023 (359 comments)
https://news.ycombinator.com/item?id=38641289
I'd love to see that angle researched more because I think it changes the game from something commercial to a far more important level.
It's also something the driver would easily avoid if they still have any control over the train, e.g. braking force.
> It's also something the driver would easily avoid if they still have any control over the train, e.g. braking force.
Yes, but: it would be much harder to test whether bricking this thing selectively does what it should do and for all we know right now you'd have a runaway on your hands. So this isn't just for shits and giggles.
And even a stopped train on a live track can under the right circumstances be extremely risky.
This is simply not the case. It is a fundamental feature of every signaling system, and has been for over a century. Any collision would be caused by a seriously defective signalling system, not the stopped train.
Anyway, I'm sure you'll find a new reason to say why it's perfectly ok to stop trains with passengers in them willy nilly and how that isn't a safety issue but I'm just going to let it go here. I certainly hope you're not in charge of anything that involves public transport.
(Edit: you might be able to find some of these online, as regulators sometimes publish them for comment if there is a waiver request… not going to dox myself though)
Your point about external grade crossing hazards is valid… but also completely avoidable unless the train is super long (IE: not a passenger train) or the train was already going slow (like just leaving a station or signal).
There are separate system safety plans to ensure that timely/safe evacuation is possible regardless of where the train stops.
In the absolutely worst case, this means coupling another train and waiting the ~30 mins for people to walk through the train to the new cars, then uncoupling.
The most recent major incident is two months ago:
https://apnews.com/article/poland-train-accident-1db1a088c31...
And there are plenty of others to choose from:
https://en.wikipedia.org/wiki/Category:Railway_accidents_and...
There are only two countries in the EU that have worse rail infra than Poland (Romania, Bulgaria). Rolling stock is off mixed quality and vintage, maintenance spotty at best.
Since then, workers get no days off, they reduced the amount of workers on a train to 2 and are trying to get it down to 1 as we speak. They increased the length of trans by literal miles, and these trains have killed multiple people by blocking roadways / ambulances and have run over children who have had to crawl underneath the trains to get to school. This happens multiple times a week where the road is blocked for almost a day.
These conductors have to walk the length of 4 miles to "inspect" the train, and deregulation has pushed the inspection time down to mere minutes instead of an entire checklist. They have taken all the power from the single regulation body and allow trains to run with brakes built in the early 1900s. We have many multiple "accidents" a year with chemical spills, all for profit seeking behavior. These workers are consistently over-worked and cannot take any sick days unless they are scheduled 2-3 months in advance.
Train accidents are inevitable at this point, the next one may be more disastrous than Palestine Ohio. This is the direct result to profit seeking by corporations (who pull in billions a year) and deregulation. It sucks because trains are f*cking awesome and could help society in a million ways.
https://badcyber.com/dieselgate-but-for-trains-some-heavywei...
They should have mentioned something about vendor lock or right to repair in the headline.
... Eh? Trains, to be clear, are not big polluters; not sure what made you think that.
But instead this was all about electronics and malicious locks, which wasn't apparent from the headline.
Modern diesel engines yes, but the old ones? They got barely any emission controls, only for locomotives built after the 90s there is regulations [1]. On top of that comes brake dust [2] which is only irrelevant in powered-car electric passenger trains with regenerative braking - every other train releases insane amounts of it, no surprise given that the brakes have power outputs in the megawatt range.
[1] https://www.epa.gov/regulations-emissions-vehicles-and-engin...
[2] https://www.railwaygazette.com/vehicles/reducing-brake-dust-...
Also regenerative braking... Trains are amazing
When I walk I kick up dust. And I breathe. I guess I'm also not emissions free even when I cycle? Brake dust and all that?
There is a lot of variation in train emissions, but most modern ones (electrified) have minimal emissions - and far lower than electric cars even, per passenger mile or per tonnage transported.
Even the oldest ones in common use (diesel electric) are more efficient per ton-mile and passenger mile than a typical car.
Wow, that's just... so wrong.
(1) This was scheduled maintenance service, not a quick repair. Maintenance is complicated and is expected to take more than 10 days:
> Maintenance a train is a complicated affair – it has to be taken apart, the parts sent to the various manufacturers, checked, sent back, the train put back together again and tested. The SPS carries out the maintenance procedures according to the relevant maintenance manual (some 20,000 pages) provided by the manufacturer, but the train does not start after being put together.
From [1].
(2) If this was a legitimate check then there should be a legible error code instead of the train randomly locking up with no explanation why. This was clearly designed to sabotage competing maintenance service companies.
[1] https://badcyber.com/dieselgate-but-for-trains-some-heavywei...
Complete checkup takes time, or one important mechanic is sick, delaying things or whatever. It is not the responsibility of the manufactor anymore. (A different company has the official service contract.) If you have other informations pls share.
After which the two specific trains this happened to "gained" GPS lockout.
This makes little sense. It's pretty reasonable that any machine that is sent for repair may take longer than expected in the workshop. Parts availability for one. Also, manpower shortages, scheduling (we don't need it back until next month) and so on all make this "heuristic" more likely be a scam. This idea that the manufacturer is entitled to a lifetime stream of repair revenue has to stop.
I shudder to think what would happen in the US if an auto manufacturer tried this heuristic on the average owner of a pickup truck.
Yeah if you're going to try to pull a scam like that the US is not your best option:
https://nypost.com/2022/12/27/texas-mechanic-executed-over-5...
https://www.thetrucker.com/trucking-news/the-nation/pennsylv...
https://apnews.com/article/auto-shop-shooting-florida-c4d45f...
If you just search "truck owner murders mechanic" you get tons of unique results.
Wow.
There's people posting like that here on HN.
I know HN is a great place to practice PR-speak, but could you please stop fucking with the gullible people?
>"The president of Newag contacted me," Cieszyński wrote. "He claims that Newag fell victim to cybercriminals and it was not an intentional action by the company. The analysis I saw indicated something else, but for the sake of clarity, I will write about everything.
>Newag president Zbigniew Konieczek said that "no evidence was provided that our company intentionally installed the faulty software. In our opinion, the truth may be completely different—that, for example, the competition interfered with the software."
/s
Parent comment is really an indication that society has lost the plot when it comes to ownership. The trains do not belong to the manufacturer after sale. They can't introduce anti-competitive code and pretend that users want it, like phone companies can.
"Lead time on the toilet is 21 days from the manufacturer."
"Uff, 21 days? That's more than 10! We will have to junk the whole car."
"Told you you would make it worse" incoming.
I guess the logic being that it could be an unknown service location.
Oh, that seems pretty damning. I wonder if this was a lone developer or ordered from above.
But I think we need another PSA that, if you are going to write some code that could land you in jail, make sure you get your entire reporting chain in writing telling you to do it, up to and including the CEO and the board of directors.
Repeat after me: The company is never at fault.
Note that the above is only believable before the rest of the investigation. Proper investigation proved the story wrong, but it is just possible enough to believe it could be true if the proper investigation hod come up differently.
Also: I wonder if their management realizes that they probably have a nice trail in the form of a bunch of repositories and commit messages. Would be nice if that leaked.
https://www.google.com/search?q=version+control+plc+programm...
You'd have to be pretty daft to do this kind of development today and not take advantage of version control and even the most visual versions of these systems eventually output (text) files. You may not be able to do an easy line-by-line comparison but you will have a commit log with helpful messages.
Look for 'engage in anti-competitive behavior' in the log message ;)
But then again, as a Dutch person I have enough issues locally that I can't even complain...
Ugly times.
He absolutely raves about them. It sounds like he's got some good coders.
No, pretty much just the manufacturer loses. Short term the operator loses, but I'm sure that the courts will award damages.
For me, this incident is a welcome argument with which I can tighten the screws on manufacturers in the next round of train buying (at minimum, they will agree to heavy contractual fines for anything like this; at best I get full source code for every train).
For too long the only priority in OT was safety (fine in the 80ies, but the second you integrate an IP stack that posture doesn't work anymore). This has been changing in the industry thanks to EU-regulation; this incident will accelerate the change.
> (...)I can tighten the screws on manufacturers in the next round of train buying(...)
But then I can see it might help change things for the better across the board, as you nicely described. Thanks for the illuminating comment!
EDIT: BTW having no version control would be pretty telling on its own. It's a critical piece of software, that controls a train..
Its on-disk representation of graphical 61131-3 languages (FBD / SFC) is text-based and somewhat human readable, so there's nothing technically preventing the developers from keeping all of this in any other VCS of their choice.
You likely won't see any 'feature branches' or frequent merges in this kind of environment.
Except merging things, and handling a lot of files...
There are lots of small things wrong with SVN. But it's indeed usable.
Merging things is different than in Git but it works. I use both, and I'm not religious about either, some things are easier in Git, some are easier in SVN. Git provides more footguns. And loads and points them too.
Of course before going on the stand the expert witness will work with a lawyer to word smith the above into something the court will better understand. however I think the generic idea is something everyone here will agree with.
I've used a thing that not only doesn't play nice with versioning (your local workspace is a collection of embedded db files) but doesn't play nice with multiple developers (no way to sync workspaces). I still managed to get it into version control, even if useful things like diffs didn't do anything useful.
Having said that a lone developer can come up with the idea, propose it to management with the expectation of some fat bonus. But why do this in secret?
There is just no possibility where upper management is not involved in this.
The ridiculousness of this defence makes it clear leadership was in the know.
IIRC, there was some "konami" code to re-enable the train after disablement that was removed in a first-party update after the third-party repair company found the sequence.
And not related to the third-party locations, one of the trains had some code where if the year>2021 & month>11 & day>? then the train would disable itself; ultimately doing it in the wrong year because the train was off during november / december of that year. This is a little excessive for somebody random to do.
https://kolejowyportal.pl/koleje-dolnoslaskie-odpowiadaja-ne... - this is from 2022-07-06. The train owner complains that they still didn't receive information from Newagg about what was fixed when unit had to be shipped to manufacturer after it refused to start.
So the issues with 3rd party servicing were publicly known well before the smoking gun was found in firmware.
In the last year they ran the contest (2016), the goal was to write a program that would compare two sets of measurements for similarity (nominally radiation emission spectra) but fail to give a correct answer when given a particular crafted input.
Guess. (Did I miss the sarcasm again? I always miss the sarcasm.)
Otherwise, they’ll find out how loyalty is rewarded nowadays.
Whoever implemented this, if you read this, you are a very bad person and destructive for society.
If you want this to not happen, you need to incentivize refusing to do this more than the existing incentive to do this.
Otherwise, punishment only changes who's in the chair next time.
(Future possible punishment is not a very good disincentive, which is why increased prison sentences don't deter crime.)
This is why we need very strong whistleblower protections.
Which is not really my point as to why they should be held responsible. The reason is the real world consequences of their actions and the scale and ease of introducing negative consequences by tech creators.
However, the "yeah, sure I can add in a GPS locator module" and the "yeah, I can add analytics that reports when the train is in a maintenance hanger" and the "the catastrophic program halt code module used in cases of extreme failure is located here, but why do you want to know that?" all seem less than unethical.
Theoretically you only need one unethical line of code, so how it got there, I think, is pretty important to know before passing ultimate judgement.
EDIT:
Of course for something like train control software, you really should have a process or at the very least responsible engineers that would notice a middle manager with limited technical skills asking suspicious questions and then pushing up a PR that is self approved.
I would be more than willing to entertain an ethical debate along those lines. Although, like I said, I think it's important to understand the whole story because the specifics really do make a difference.
(not expecting this, but would be nice)
But there is more than one powerful person in Poland, and Newag owner is far from the most powerful.
That’s a very bold statement right there! Also, from my experience, most if not all companies who build automation/robotic systems they maintain “backdoors” of some sort, not to disable them remotely, but to quickly and efficiently access the software remotely upon request to troubleshoot or fix issues, it doesn’t make sense to fly to the client and sometimes the other half of the planet just find out it was a local IP mismatch or a system service is down, so saying “impossible” is not true.
* Polish train maintenance company, SPS, was getting suspicious as trains made by a company, Newag, kept on "randomly" breaking and couldn't be fixed. They was getting fined millions by Polish government as they had a contract that fined them for being too slow with repairs.
* They secretly hired literal hackers (Dragon Sector) for 2 months to dig around Newag train code.
* Hackers found out some incredible things, generally that fit under the umbrella of "late-stage capitalism", or more specifically, corporate protectionism, sabotage, ransom, etc.
Some examples of the secret code that the hackers found:
* Breaks the trains if they go into geo polygons that are right around the warehouses of 5 Polish train maintenance companies, including SPS.
* Breaks the trains after 1 million kilometers.
* Breaks the trains if they don't move for 10 days.
* Secret button press combination (basically Tekken, Street Fighter, etc.) to disable the "malfunctions".
I'm waiting for the stuxnet-like report on this as much as anyone.
https://api.newag.pl/shouldirunornot?lat=&long=&trainid=
can't wait to see the swagger docs on this...
Does no one read the site anymore?
original news story discussion just over a week ago: https://news.ycombinator.com/item?id=38530885
And the followup from the company
Polish train maker denies claims its software bricked competitor rolling stock https://news.ycombinator.com/item?id=38570654
Not to mention the 404media dupe with a ton of votes just yesterday also!
I don't know, is it some kind of requirement to read every piece of news on the main page every single day to keep visiting? Or can I log in every few days, read a few stories, upvote the ones I like, then visit again few days later? Or is that not allowed?
Upvote what you see/like, sure, all good.
But if you're not around, you miss some stories, that's it. (especially when it's come up so many times over weeks) There is a current events aspect to HN.
I don't read the vast majority of sites linked here. I come here for the comments. I assume anything that's important enough to know will be directly quoted in a comment here.
There's ways to add back doors and other fun easter eggs in code, but wow they picked a really cumbersome and obvious method of exploitation.
I mean, I'm Polish and worked for such people in the past. The statement reeks of an approach to business that I wish went extinct already, but apparently you still see pockets of it here and there.
Essentially, if they're going to do this, a good route is for them to be the malicious actors. Just before shipping, someone trusted surreptitiously switches the binaries to the modded versions.
I can imagine the initial disbelief and shock as you start to piece together what the software is doing.
So malicious, I wouldn't believe it at first.
Edit: someone already added Newag there :D
On the other hand, trains that refuse to start don't immediately kill everyone on board, and trains that have fall off the tracks do tend to kill people. So when a train fails and people are dead, and people go in search of who is responsible, it seems decently likely that ambiguity between the third party repair shop and the company responsible for the train is a problem.
Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There should be an opt out system - something where the train operator contacts the train supplier and says "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Or to bring it closer to home, say one of us breaks into our tesla to exercise our God given right to change the software stack, and then it kills the driver and various people around it, to what extent is that Tesla's fault? Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
This is an interesting discussion, thank you for your nuanced opinion. I don't know where I stand on it but you are spot on about the demographics of HN. I certainly have a knee-jerk reaction in the direction of operator freedom.
Edit: Additionally, if something like this goes wrong (say the train derails) it will be a massive blow to right to repair. Even if it is later found that the third party repair is not responsible, the media will scoop up this story and it will forever warp the public's ideas about right to repair.
From what I've read, the train operator bid out the maintenance for these trains, and the manufacturer lost the bid to a third party. So your "opt-out" system should absolutely have been triggered here.
If there was some reason to believe that the low bidder would be incapable of safely performing the maintenance, the train manufacturer should have raised that issue publicly, not silently sabotaged the low bidder's ability to perform the maintenance.
The question of where liability falls after a repair isn't covered in the articles linked unfortunately. It seems plausible that it was initially whoever wrote the software and is now someone else.
I think it's extremely likely that this disable-train-on-various-conditions behaviour is exactly as specified and documented since train software is a bit obsessed with formal methods and documentation. It's then only "silent sabotage" to the extent that said docs were ignored.
It's credible that a way to easily disable these safety checks for the case where it's now someone else's liability wasn't considered worth paying for by whoever bought the train. It's tomorrow's problem after all. Also the customer may not be totally convinced of the necessity of the software dev paranoia, especially if they're not liable for failures.
So sure, maybe this is evil/negligent software people at the train company. I don't see that conclusion well supported by the article.
Update: here's a quote from the Ars Tech. article itself, showing a) no tampering with the trains' software or hardware was necessary to get them running, and b) there's no even slightly plausible case for believing the manufacturer was unaware of this. Furthermore, if it really was somehow unaware of what was going on, then these 'features' cannot be justified as a safety measure.
Dragon Sector got the trains running after discovering "an undocumented ‘unlock code’ which you could enter from the train driver’s panel which magically fixed the issue."
Update 2: It is barely plausible that the ability to track the trains' presence in other maintenance shops was initially added to gather evidence for liability and warranty purposes, and then someone had the dumb idea of using this to covertly disable trains when this happened.
Note that the use of a third-party maintenance operation was not a secret: the train operator solicited bids for the work, and the manufacturer tendered one. Clearly, third-party maintenance was not in violation of any contract (and if somehow it was, the manufacturer did not need to gather any covertly-acquired evidence for a breach-of-contract suit.)
From what I've read, the maintenance company did scour all documentation and found no explanation for why the train was disabled. It was an entirely undocumented feature. Maybe they were just lying, but this idea you have in your head that it was just a fly-by-night operation who couldn't be bothered to read the documentation is contradicted by the available public record.
Actually, the issue here is that the train company is on the wrong side of this issue.
> Further, if your company is financially liable for a product failure, and another company is not, that second company can totally fix it more cheaply than you can. As it's not their liability.
There are tons of cases where the repairer, not the OEM is held liable. The manufacturer is at fault for the defective parts/product. The repairer is liable for a defective repair. Courts have been sorting this kind of thing out for centuries and it's not a difficult thing to deal with.
> "we want to use this cheaper repair shop, and it's now our problem when the budget train fix kills people" - that seems totally legitimate to me.
Except that really isn't the case at all, and hasn't been the case. If the repairer does not do the repair correctly, the repairer is liable, not the manufacturer. Most mechanical shops even carry insurance and offer their own warranties on their work. Most cities (and hopefully consumers) will only do business with repair shops that are insured for this exact reason.
> Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Yes. It is not their car. It is my car.
Ford is not liable if that variable geometry camshaft I installed in my Mustang causes the engine to blow, killing six people with shrapnel. I would be liable for that. Why should Tesla be liable for someone hacking the software? The issue here is the world doesn't work the way you are describing, where manufacturers are held liable for the work of third parties. Tesla is held liable for their own warranties and laws applying to product safety.
So yes, it works just as you describe. (Planes, not trains, I know, but it's still a datapoint on how liability works out there.)
- there's a proper registry of who is responsible for maintenance of the train, including chain of liability. Crucially, the vendor wasn't in it for the affected trains - they'd only be liable if design issues were found.
- maintenance companies was appropriately certified and verified - we're talking professional MRO, not random workshop
- owners of the trains were also registered as responsible parties for maintenance and repair
- vendor was supposed to provide documentation that would allow such a workshop to perform maintenance and repairs up to P5 (largest scale) maintenance
- vendor never disclosed lockouts they implemented, thus committing fraud at the very least.
I definitely haven't gone looking for the documentation on the train system, partly because I assume it's unavailable to the public and mostly because I assume it's enormous in extent.
That is not the case here.
If that was indeed the case, one would expect the train's diagnostic systems to report that it has detected certain issues that need to be resolved before the train will start up again. But that didn't happen, instead the train simply refused to start, no error codes, nada.
Not only that, but further investigation revealed that the train's firmware contained code that would disable the train if it was present in a competitors shop for a considerable time.
> There should be an opt out system
This mechanism for disabling the trains, was never mentioned in the discussions or contracts for purchasing the train (which is why a third-party could win the repair contract). There was never an option presented to "opt out", that is why people are saying that the trains were sabotaged.
> to what extent is that Tesla's fault?
It isnt. You made unsafe modifications that can be clearly shown to have caused damage, it's your liability.
> Say Tesla made all reasonable steps they could take to detect third party mods and refuse to start the car, do we hate them too?
Depends, did their software present an actionable error or not? If the car gave an error code, and you look in the manual and it says that too much current is going to the motors (idk), and then you fix that and the car starts? Great! But if the car just refuses to start without any indication as to what is wrong, and further even if all repairs are reversed it still no longer starts? That sounds like deliberately sabotaging third-party repair.
This is just the next episode in an age old battle. Old cars were easy to repair, so people often went to independent repair shops. Companies obviously didn't like this, so they started using proprietary screws. Then laws were made to forbid this. Next round came when cars got computers and companies didn't give the repair shops the required software and/or hardware to work with them. So, laws got made which forced them to. And so on. Each time there's some new technology companies will try to find ways to misuse it to the detriment of customers. And each time law makers will have to get involved and put the screws on them. And then people later go around "why do we have so many laws" .. because companies are shit, if they can get away with it. It's their natural impulse.
Regulations don't work. There needs to be legal liability for the fraud perpetrated by the vendor against the customer, not prescriptive rules that vendors influence, then follow dogmatically so they can cite compliance in order to disclaim liability for their intended bad outcomes.
If only commercial airliners had this too...
https://news.sky.com/story/fraud-officers-arrest-one-in-dawn...
Because it's pretty normal for an aircraft to be in use longer than the manufacturer exists.