Hacker News Reader

Comment by atticora | Hacker News Reader

It would be so easy to get away with this kind of extortion at my work. Nobody reads my code that carefully, or cares if I don't get it reviewed and just merge it. Only one other person could understand it if he tried, and he has no interest or involvement in it. It could easily look like just a bit of incompetence on my part that requires some additional consulting from me after I have moved on.

That's not how I roll ... or sleep well, so my employer is in no danger from me. But there are many short-term devs who come through here, and I don't have the time to police them in detail.

But conceivably an LLM could do it. It could be just another step in a build pipeline. But, when LLMs can do this well, they can also write most of the code going into the pipeline.

View on HN

This doesn't sound like the sort of thing some rogue developer secretly slips into the codebase.

Exactly. This is a company initiative to increase company profits. It's smart business, as long as it's not illegal or the fine is insufficiently high.

Yup that is how I read it as well. Product decision.

Is it smart business though? Once disclosed it provides future purchasers with a strong reason to avoid your products. Who wants to spend millions on trainsets that could become unserviceable in the event that the seller goes out of business or makes some mistake in authorizing service centres or gets into a dispute with us over another matter?

It can be smart business if the probability of it being disclosed is low enough. Using fake numbers as an example, if you can make an extra $1 million on repairs and will suffer $100 million in fines / lost business if it becomes known, as long as the probability of it becoming known is less than 1%, it's a net positive expected value.

it's just tight-rope walking at that point. If your company has sufficient leverage within the market they can get away with murder.

see: John Deere

Ahem... Boeing 737 MAX, which was literal murder.

I would guess this is also why the code was found: it's parallel construction.

Somebody was told to take a closer look.

Otherwise it would be very weird to have 3rd party developers disassembling firmware code. I've never heard of that happening because a train didn't want to start.

When the trains your company serviced start experiencing failures, you look at your workers. When the trains your company was supposed to service, but didn't manage to touch yet start experiencing failures, you might begin wondering about alternative explanations.

I imagine someone in the company was someone who knew (or was a parent of someone who knew) someone in Warsaw Hackerspace, and introductions were made.

It's kind of amazing how blatant it was, they weren't even really trying to hide it much.

Similar to the VW emissions thing; if they'd been intentional about it they could have made it look much more like a mistake.

> But conceivably an LLM could do it.

It'd be kind of funny if an LLM did that "unintentionally", and wasn't able to unlock the code it wrote... ;)

The EN50128 safety standard for the European safety critical rail software places great importance on the development process.

Every change to the software has to be based on a defined requirement, and in order to validate the software you have to prevent evidence that every change was approved by a reviewer that is competent for that area of the software. The validation report contains the signature of this person.

If your code passes every test, but it wasn't developed in accordance with the process, it might as well not exist.

Of course I can't say how well this process was followed in NEWAG, but in theory rogue changes shouldn't be possible.

But how would you profit off of it? In the case here the company profits by forcing trains to use first-party workshops.

"Last time this failed, Bob was the only one who could fix it."

"Bob resigned a few months ago."

"See if he is willing to do some consulting. We'll pay whatever rate he demands."

I still occasionally have past employers call about things years after I left, and if I'd have been immoral enough to pull something like this, those systems could have been full of time bombs.

This mindset reminds me of the policies we use in the dev team at work. Any policy access that I suggest starts with the thought "If future me were to go rogue one day, how would present me stop me?"

But this is about a physical train that's in a first-party repair shop. How will the people who work in the repair shop know to call you, the software developer?

Reply on news.ycombinator.com