They clearly knew they were capable of reprogramming the control logic remotely (otherwise this idea for a test would've gone nowhere) and any engineer could've predicted this obvious outcome without the need for a real-world test.
Telling a business person they have a software problem is meaningless. Showing them losing a LOT of money because they ignored it gets attention. Ask any hospital about ransomware. Our local hospital is still running Windows 7 and older versions.
Everybody loves network connected new cars. Nobody thinks about the fact that they are "fly by wire", using software to control everything. Who thinks about changing the battery controller to cause a battery fire? Or controlling the acceleration to maximum? Or setting off the airbags? Or...
Use your imagination. What could go wrong in a 2 ton computer on wheels? Especially one that has a camera and software that can recognize "soft targets" (i.e. people).
It really should be illegal to have network connected cars. Any software security engineer knows that. But you can't convince a business person.
Automotive engineers think about those things. When we drive our cars we're trusting them to have thought about those things, just as we're trusting that the engineer who designed the wheel nuts did his work well enough to stop the wheels coming off, or the engineer who designed the steering linkage considered how we might need to go round corners reliably.
Not always. Hacking a vehicle's brakes is no longer the stuff of sci-fi, it's already happened in a poorly-designed Internet-enabled jeep, see [0]. (As an aside, I have no idea why they thought it was acceptable to do their proof-of-concept demonstration while driving at speed on public roads.)
[0] https://www.wired.com/2015/07/hackers-remotely-kill-jeep-hig...
irresponsible on the journalist's part too to consent to this. the point could have been made just as well on a private track.
To cause even more outrage.
If that were a reliable safeguard, we wouldn't already have several examples about the behaviour of connected vehicles being compromised remotely in one way or another. Since unfortunately we do have such examples, evidently whatever is going on within the engineering groups at the auto manufacturers is not sufficient to ensure our safety and security in relation to these new generations of connected vehicles.
Ask literally any ICS security professional their main concern, it's "[organisation(s)] think a system is airgapped when it isn't".
Whether a system is software-defined/hardware-defined, networked/standalone - these are not technically hard questions to answer, correctly. (No more so than, say, whether polymer O-rings will completely fail to do their job when cold.)
That they're not answered correctly indicates a big communication issue, and whilst I'm sure management has much to answer for, I'm not sure engineers escape the blame for that either.
Looking at CANBUS, I guarantee that they do not. Any device trusts any other device, access to the bus is the only security control. Connect your entertainment system - probably an unpatched Android tablet - so it can display any info from the rest of the car, instant compromise of everything.
I fully agree that automotive and medical equipment/IT could greatly benefit from not having security as an afterthought.
However, I don't know any vehicle with connectivity (other than that Jeep Cherokee controversy), which does not have safety critical CAN/FlexRay buses segregated from user facing 'infotainment' systems.
What that means is that the network bus in which your 'compromised' infotainment system is able to operate is completely separate from Engine, ABS, AEB, ESP, Airbags etc.
The solutions vary but there is usually a physical gateway that prevents a passthrough MITM attacks, so for example you cannot simple send a message frame from your infotainment pretending to be an Emergency braking request to your ABS system.
Gateways can allow for certain messages to pass-through, the point is that they preserve the garantee that certain safety-critical messages can only originate from within a specific network. It's basically a crude MITM mitigation.
That's the problem, though. Once bidirectional communication exists, you can pretty much guarantee there's an exploitable hole in it that can be used to break the firewall.
You would have to hack a gateway (on-site) or physically access a segregated bus, at which point you could argue that you could also physically tamper with brakes or engine without any software being involved.
The original claim was that vehicles should never be connected to any network because of unspecified online attacks that could actuate brakes or steering. I explained why this was unfounded.
Adversarial patterns against computer vision based ADAS are only a real issue for system which are not sufficiently redundant . Autonomous systems in particular should also apply a degree of sensor fusion between multiple sources of data such as optical computer vision, radar, long range ultrasound and LIDAR (once it becomes cost effective). If any of those systems provides erroneous data the remaining ones can negate that and allow for a fail-safe behaviour.
If you want my opinion, as someone who has moved away from automotive R&D a few years ago, Tesla's decision of depending too heavily in computer vision systems without addtional sensor redundancy seems like an architectural defect that has already cost lives. Either they are not integrating other sensor data sources or their voting weight appears underestimated.
I really wish people on HN would stop with the "I disagree, so you must have not read everything" comments. I keep seeing these recently, but they are unconstructive, and they are insulting to other participants in the discussion. You might like to consider the alternative possibilities that you weren't clear in your earlier comments and people are responding to what you actually wrote and not what you thought you wrote, or that you might not be fully informed and someone who disagrees with you might simply know something you don't.
The original claim was that vehicles should never be connected to any network because of unspecified online attacks that could actuate brakes or steering. I explained why this was unfounded.
Actually, what you have repeatedly said is that you aren't aware of any vehicles (except the infamous Jeep case) that don't fully isolate infotainment from safety critical systems, which is not the same thing at all.
You also said that the Jeep case was due to a poor architecture that came from the 90s. Even if this is true, the exploit using it to trigger multiple dangerous behaviours remotely was demonstrated in 2015, so apparently the manufacturer was a bit slow on the uptake of what you think they should have been doing for the last 20 years.
Moreover, at least one well-known auto manufacturer is (in)famous for performing OTA updates that can change fundamental car behaviour, so evidently there are still networked vehicles where safety-critical functions and remote communications can not be fully isolated. It doesn't take a genius to extrapolate from this to the not-so-distant future when auto manufacturers are pushing ever more autonomous functionality combined with OTA updates, either.
Then we have all the cars that now have remote controls that do more than just unlock the vehicle, affecting things like environmental controls, or even summoning a driverless vehicle over a short distance (in theory, at least) with some of the newer developments.
Next we have the security systems providing remote access, such as OnStar's Stolen Vehicle Assistance system that someone else already mentioned, which can in some cases interfere with speed or ignition systems remotely.
I think by this point it's safe to say that whatever explanation you think you gave, the evidence doesn't support a conclusion that modern network-connected vehicles are safe because their critical safety-related systems are fully isolated from external influence, which is what really matters here.
Adversarial patterns against computer vision based ADAS are only a real issue for system which are not sufficiently redundant.
And yet not so long ago I read this, which if you're interested in the field you've surely seen as well:
https://www.nassiben.com/phantoms
So again, evidently there are systems out there in production that are "not sufficiently redundant". You might have some personal opinions on what should be happening, but that doesn't mean what actually is happening respects your view.
In practice it's not typically necessary to go quite that far if a trusted communication processor is able to pass messages or set registers but only ever takes logic control flow input from the "safe" side. Plenty of equipment that wants to broadcast state over radio but never accepts any sort of input back beyond acks or whatever has done this for a long time.
(This is, of course, entirely my opinion. I haven't seen the code of these systems, but looking from the outside, nothing I've learned makes me feel like I can trust their security.)
I recently heard an "On Star" ad that proclaimed that they can "slow down stolen cars". Consumers are wowed by features that, by definition, are enabled by unwise security decisions. Manufacturers will cash in. IT security in cars will get worse.
But how can you include adequate firewalls in a car that, for example, receives OTA updates for autonomous driving software? It is inherent in any such system that incoming signals are received and that the software that may be affected by those signals has access to pretty much everything.
Even if the brake pedal is connected via an actual wire, emergency brake actuators can work by themselves on the system.
Electric assist motors on the steering column can also overpower a human with ease.
So, actual connections doesn't matter.
Absolutely zero benefit of the doubt should be given to systems where a single error could cause grievous harm to tens of thousands of lives. Such systems should require an absolutely ironclad public legally-binding positive assertion of security with strict criminal liability for failure (blame can not be shifted) that is independently validated before we should even think of accepting their use. We already accept nothing less for systems that are less dangerous such as nuclear reactor meltdowns and systems that are equally dangerous such as nuclear weapons systems. If systems with failure modes many times worse than the atomic bombings of Japan can not be made safe, then they must not be made. Anything less would be so criminally irresponsible that we lack a word for the magnitude of irresponsibility.
[1] https://www.wired.com/2015/07/hackers-remotely-kill-jeep-hig...
[1] https://www.detroitnews.com/story/business/autos/mobility/20...
I don't need to justify anything, I am stating they are isolated in all the vehicles I am familiar with. [0] The example from the Jeep Cherokee is a pretty crude architecture defect, the only way that could happen is in vehicles with a single CAN bus, which really is something out of the 1990's.
As constructive criticism you really ought to read the "In Comments" section of the guidelines at the bottom of the frontpage.
[0] https://www.st.com/en/applications/body-and-convenience/auto...
The Jeep Cherokee also did not have only a single CAN bus as evidenced by the diagram on page 8 [1]. The problem was that the radio unit was on both CAN buses.
I ask for justification of your statement because you counter-argued the previous comment that stated: “It really should be illegal to have network connected cars. Any software security engineer knows that.” by stating that the safety critical systems are segregated and thus “the network bus in which your 'compromised' infotainment system is able to operate is completely separate from Engine, ABS, AEB, ESP, Airbags etc.” which implies that solution achieves the desired security properties. I claim the desired security properties are: “safe enough to bet the lives of 471,000 people on them and take responsibility if they were wrong”. I think, and I think most people would agree, that is accurate assessment of the potential consequences of catastrophic failure. That is a consequence comparable to the detonation of a nuclear warhead in a city, so we should apply similar standards to the required level of security. Hardly any software engineer would make a claim that their software can be trusted with an amount of lives within 3 orders of magnitude of that number (note 3 orders of magnitude is 471 which is comparable to an airplane crash, so 3 orders of magnitude less is critical avionics level). Therefore, making a claim that implies that level of security has already been achieved is an extraordinary claim and should thus require equally extraordinary evidence to be believed.
Even with such justification I would still caution anybody else reading it since it would not be a legally binding claim by a liable entity, so even assuming you are commenting honestly and to the best of your knowledge (which I assume you are), and even correctly describing the truth as it is today, there are no consequences for any liable entity if they betray your trust. It should always be up to the liable entity to justify themselves to the people to our satisfaction before deploying systems with such societal-level consequences.
This is a common FCA design. CAN HI and CAN LO often go to the same device on different pins. My Jeep has an instrumentation bus as well and all three arrive in one box in multiple places within the vehicle. Friend with a Rubicon Wrangler discovered both engine buses connect to the electric motor to disconnect the sway bar. He discovered this when water got in it, shorted it out, caught the electric motor on fire while he was watching, and subsequently totaled the ECU and TIPM by transmitting electrical noise on both CAN ECU buses.
You’ve assumed one bus goes to one thing to reach your conclusion. That is faulty logic, which is no shortage of ironic given the parts of your comment I’m mostly ignoring in this reply.
My code reader has fetched “CAN HI” before and the code was contextually referring to the high-rate bus, not the physical wire. I’m not a deep Jeep tech, just dangerous electrically and with a spanner, so I may be wrong in how I’m spelling those and I’m following the lead of a trouble code. If you’re nice to a service dealer with a laptop you can get a quite lovely wiring diagram that explains it better.
The car can even be turned on and summoned remotely. Which means if nothing else it could be remotely driven onto a freeway to cause accidents even if it’s ‘autopilot’ had some kind of independent backup safely system to avoid collisions.
The important systems should not be permitted to be connected to the Internet. If an Internet-enabled entertainment system is fully isolated from the systems that really matter, there's no problem.
I could see that causing some secondary serious problems.
Could potentially amplify its impact by combining it with the GPS to ensure it occurs at a high speed and in busy traffic too.
Most older infotainment systems have a physical power switch which is a hard interrupt to power-off. It is in my view a design mistake to have full touch-screen interfaces on infotainment.
To be sure, that when I turn off a device, it is really off.
In terms of overall fatalities, distracted drivers on their phones can be expected to continue to kill far more than vehicle hackers. It's troubling though that hacking vehicles' brakes could allow for targeted attacks with little chance of being caught. The same applies to insecure medical solutions.
Now I'm curious. Does anyone know the manufacturer and model number for a microwave for which this is true? Have we really replaced hardware thermal switches, fuses, etc. with firmware?
It's a (perversely) interesting creative exercise to think of catastrophic consequences that might follow from the failure of a system we don't normally think of as critical. Let's restrict it to physical systems, excluding concerns like being banned from Gmail, and issues like misinformation and extremism spreading through social media.
• A bug in a sound driver could cause hearing damage, especially if someone is using earphones
• Perhaps a bug in a graphics driver could be dangerous to people with epilepsy
• A bug in a laptop's battery-management system could cause a fire on an airliner
• A bug in an overclocking tool, or in motherboard firmware, wouldn't typically cause a fire, but perhaps
• We know that bugs in satnav systems can cause inattentive drivers to cause serious accidents
• A bug in an app about food intolerances could have severe consequences. A graphics/rendering bug, perhaps in the OS, could cause the app to misrender.
AFAIK, the magnetron power supply is physicaly connected through multiple door switches, so this is an actual unoverridable and redundant disconnect. One switch shorts out the supply when the door is opened, so that if the others fail, it will still blow the fuse and cut power. I'm pretty sure that design is due to regulation, so I don't see it ever being replaced by software.
The real problem in these cases is internet-connectivity + software-control. Without internet-connectivity, your primary failure mode is random chance. The probability of failure of each system is largely independent from the others as there are few ways to induce simultaneous failure in all such systems. This means that the worst-case failure mode of any individual error is the worst-case of single-system failure which, for most systems, is relatively limited. However, once you add in internet-connectivity, you now introduce an easy way to induce simultaneous failure. Since these systems will all be running the same software, an error in one is an error in all, and you can now exploit the error in all simultaneously. This instantly adds a gigantic single point of failure whose failure mode is critical failure in all systems at the same time.
Suddenly, the software failure mode of an internet-connected gas stove with remote turn-on [1] is no longer one house blowing up in the middle of the night killing possibly 4 people, it is every one of the houses buying the 3.5 million gas stoves sold this year [1] blowing up killing possibly 14 million people. The introduction of internet-connectivity raises the worst case failure mode from a tragedy to approximately a nuclear first strike and surpassing Hitler as a mass murderer. They should not be releasing internet-connected gas stoves unless every engineer on their team is comfortable that their security would be good enough to connect the US nuclear arsenal to the internet since the worst-case consequences are equally bad.
[1] https://www.geappliances.com/ge/connected-appliances/ranges-...
[2] https://www.statista.com/statistics/221447/gas-range-and-ove...
You've made an assumption about the failure pattern. Perhaps thousands of stoves could fail at once, all encountering the same defect. Something like this could happen if the software fails after 10,000 hours of continuous operation, say. (A bug of this sort once resulted in the deaths of 28 US soldiers, when a missile defence system failed. [0])
> The real problem in these cases is internet-connectivity + software-control
I agree this is a very obvious and severe problem, and it's rather shocking that it's even legal to do this. Avionics software is famously strictly regulated, but the software in road vehicles isn't.
> Without internet-connectivity, your primary failure mode is random chance
It depends. There could be a failure after some number of hours of continuous operation. Perhaps a particular stretch of road could be dangerously mishandled by all self-driving cars of a certain model. There's also a risk that deliberately crafted fake road signs could cause the system to drive dangerously, opening the door to targeted attacks.
> once you add in internet-connectivity, you now introduce an easy way to induce simultaneous failure
That's a good point. It's similar to how we place enormous trust in Windows Update, except we're talking about a life-and-death system.
> They should not be releasing internet-connected gas stoves unless every engineer on their team is comfortable that their security would be good enough to connect the US nuclear arsenal to the internet since the worst-case consequences are equally bad.
Well, no, they plainly aren't, but I do agree with your general point.
I wonder if we'll see any kind of general software regulation (that is, not specific to a domain such as aviation) to cover this sort of recklessness. Currently, the company behind such a product would only suffer after the fact, were something to go badly wrong.
[0] https://www-users.math.umn.edu/~arnold/disasters/patriot.htm...
The main point I was trying to make is that mass failures that occur with a degree of simultaneity where the failure can not be corrected between failures are relatively rare outside of internet-connected software due to an access problem. Even the cases you mentioned are generally less dangerous because you can stop usage when an error is detected. If you discover that after 10,000 hours of operation the brakes on all of your cars stop working while driving, it is unlikely that everybody will encounter this problem at the same time. A few will encounter the problem first which will let you detect the error and then work to rectify it or recall your cars before it becomes a truly catastrophic error. It is a truly rare problem where a mass produced product contains a catastrophic defect that will affect a large percentage of them simultaneously. The only ones I can really think of are drugs with long-term unexpected side effects or a response to an equally massive cause like the Carrington Event coronal mass ejection which blew out the electrical grid in 1859.
I'd count the mass ejection as a natural disaster rather than the failure of a system, but the harm is just the same of course.
Related to your medicine example: bad dietary advice could be similar to this. We're straying some ways from technological systems here, of course.
I don't see it this way. Right now, there are so many people in cars that should not drive. Alcohol, drugs, lacking skills - to me, computers add to safety. There will always be a risk. I trust computers more in this case.
Imagine for instance there is a bush fire and your centrally controlled car decides that the fire is fake news and prevents you from leaving since it’ll cause congestion on the roads. No different from Twitter banning the NY post story. Maybe they decide NY Post headquarters is not a real place and refuse to navigate the car there.
Yeah we should implement some kind of social credit system so that people who do drugs can’t drive or take public transport. They’ve got that in the modern utopia of China right now.
If you want to push the safety related tasks into the computer, you have the obligation to make them hard to hack. And on the case of cars, that includes not plugging them into the internet.
And the latter is the really big danger with autonomous vehicles. The same consistency that means they aren't vulnerable to as much random variation as a human driver could also become a weakness. Yes, a drunk or some idiot on their phone or an unwitting patsy whose drink was spiked at the bar can cause a fatal crash. But they can't simultaneously cause 500,000 fatal crashes by doing the exact same stupid thing 500,000 times at once or by being the victim of the exact same crime 500,000 times at once.
Having a real world failure under controlled conditions - would show where failure points are and possible fixes.
Because uneducated people (like Congressmen and executives) tend to believe stuff more when they can see it with their own eyes.
It's like, in terms of information security, putting up sensitive information on walls that can be seen from the outside. Almost no one sees a problem with that because they believe it takes expensive equipment... until you go, take a 600mm+ tele lens and prove to them that their sensitive information can be stolen for under 200$ worth of camera rental fees.
Or, remember the immigration debate in 2015? The picture of three year old Alan Kurdi on the beach and its consequences in politics? Never underestimate the power of images and videos.
The article mentions the people responsible not grasping the full scale of the problem. Giving them the visceral image of the potential destruction might help them take on board the risk emotionally as well as intellectually.
2) It's very difficult to justify increasing security budget every time some infosec people says so - especially if their salary depends on such budget.