Hacker News Reader

Comment by ryukoposting | Hacker News Reader

As a firmware engineer, I'm one of the people who actually writes the code that goes inside the IoT devices. I'm very interested in what the FCC might be able to do here.

How does the FCC define a security flaw? Would updates only be distributed when there is a flaw that needs fixing?

Remote update mechanisms can themselves present security problems in some domains. Thus, some devices should only be updatable if the owner has physical access to the device. Will the manufacturer be liable for damages caused by attacks on vulnerable devices that were not sufficiently updated by their owners?

IoT is making its way into defense and enterprise environments where reliability is a matter of national security. An update nearly always results in some downtime for the device, even if it's just a couple seconds. Sometimes, it may be in the best interest of a device's owner to defer an update indefinitely, until that device's continuous operation is no longer mission-critical. Even if the owner can't control exactly what is in an update, they absolutely MUST be able to control when an update occurs.

View on HN

> Even if the owner can't control exactly what is in an update, they absolutely MUST be able to control when an update occurs.

+1 for this at the consumer level. My oven may have a critical update, but - for right now - *nothing* is more critical than finishing dinner. I'll let the update apply the day after thanksgiving when I'm doing the dishes.

There are a few connected appliances brands that do this well: updates are broken down into two classes - "critical/security" updates and "all the other updates" - and I get to pick from "do nothing, notify, notify and download, notify, schedule/auto-apply" for both channels. Unfortunately this is not common because it takes planning and more than the bare minimum to pull off.

I think this is oversimplifying things.

Is finishing the dinner more important than applying a patch that fixes actively exploited bug that locks your oven into cleaning mode and burns everything inside into ash over the next three hours? Or something that disables the safety checks and lets the oven overheat and burn your house down?

(Granted, the latter shouldn't physically be possible because it should have physical temperature killswitches etc.)

People always (understandably so!) consider software updates to be annoyances; but especially when you give an example like _an oven_, the potential for _catastrophic_ failures is too great.

One my favorite IoT botnet scenarios is an attacker taking control of thousands of ovens/air conditions/other high-wattage devices and using them to cause power outages. https://www.usenix.org/system/files/conference/usenixsecurit...

I wonder how the impulse to connect everything to the internet will be remembered.

The flip side of that is you can use control of all those high wattage devices to prevent power outages by shifting load to times when more energy is available.

Hopefully, that's how the impulse will be remembered.

Or by the rise of salmonela infections...

These seem like very rare scenarios. If we're concerned about dire threats like this, the manufacturer needs a way to remotely send devices into an internet-disconnected "safe mode" before anyone's even talking about updates.

How do you, as a user, determine whether your oven is in "safe mode" that the manufacturer has toggled, or in "safe mode" that makes the UI look the same as the real one, but is actually a malicious code that waits until 3AM to start the cleaning mode?

I agree that these scenarios are (relatively) far fetched _now_; but if we expect the future to include connecting appliances like that to the internet, then solving problems like those is table-stakes stuff.

I understand that enforced software updates are annoying; but for high-stakes environments like this they strike me as a lesser of the two evils.

If the attacker has total control, then all bets are off no matter what mechanisms you put in first. Adding a safe mode would at least allow manufacturer to stop any non-total exploit without relying on the more complicated update mechanism. Also, the appliance would more likely work in a kinda normal way in the meantime.

>If the attacker has total control, then all bets are off no matter what mechanisms you put in first.

Great, so we now agree that it's important to keep the system patched and up to date.

That depends on updates never introducing new vulnerabilities.

I didn't say otherwise. Only that a forced update mechanism is insufficient for stakes as high as burning down the house. Maybe the patch fails to download/install, or they take too long to develop the patch in the first place, or the patch doesn't work for all models on the first try. There needs to be a simple kill switch if we're talking about threats like this.

Besides that, the safe mode oven should still be able to cook food manually.

Just as smart people agree that the manufacturer knows nothing about the situation at the appliance, and a forced update mechanism a la the first iteration of Windows forced updated ("update's here, restarting, too bad you're currently doing important stuff that won't be saved") is a stupid idea and outside of private use often a non-starter. That kind of condescension is generally only something you can do to private tech users.

Though I guess making the experience frustrating could keep more people from needlessly connecting everything to the network because it sound futuristic...

> If the attacker has total control, then all bets are off no matter what mechanisms you put in first.

You could have a second SoC on the device running off ROM, whose only purpose is handling this safe mode and controlling internet access of the main device. Keep it simple and make it essentially just a fuse that can be blown (turning off internet access) by a signed message from the manufacturer. Keep the hardware capability of this SoC to just that, so even if you have a vulnerability on that thing itself, an attacker can't really do anything with it. Keep the code running on that SoC simple and preferably make it a FSM that is proven to be free of vulnerabilities. Also make sure the main device can't interfere with it in any way.

Once the fuse is blown, the user needs to press a physical button on the device to re-enable internet access. Preferably ship the devices with this internet access disabled by default.

Even better yet, don't build an internet connected oven... but that ship has sailed in many areas.

Ideally yeah, the bar for critical appliances would be high enough for this. If not, disabling IP at the OS level would still be pretty good.

> If we're concerned about dire threats like this, the manufacturer needs a way to remotely send devices into an internet-disconnected "safe mode" before anyone's even talking about updates.

This was exactly my thought. Download the update then take the device offline until it's applied if the update contains a security vulnerability.

If these are the kind of things we are worried about then the correct course of action is to ban needlessly internet connected devices. Your need for twitter on your fridge doesn't outweigh the botnet threat it poses.

Idk if that can be done, but anyway, I don't want to hear these scary threats thrown out as excuses for whatever policy the manufacturer puts in place. If the threat model really is burning down the house, you need a lot more than some forced updates to protect you.

Finishing dinner can be more important than the house not burning down. A burned down house is likely insured. A dinner with a potential business client is not.

Surely this has to be a joke.

Could be a reference to steamed hams

It’s just the Northern Lights.

I think it works best when the consumer gets to decide. If a car recall comes along because the seat belts in your car have been shown to actually kill people in accidents, would you blow off your weekend plans to get your car fixed as soon as possible or would you go to your thing then bring the car to the manufacturer on Monday? There will be people on both sides of that decision and, ideally, we should let them choose.

You make a good point, but this kills me. I've been writing software a long time, and there's nothing I trust less to control my oven than software. God help us. I yearn for simple physical controls

I'm wondering why you'd have a smart oven in the first place. Seems like all risk and no reward.

Probably for some feature or astetic that's unrelated to the smart features, but is only available on the smart oven.

This os definitely most of it. Also, someone figured out they could do something with the parameters of the convection oven and get close to an air frier. Got the mode a year later. Lastly, when the meat thermometer hits temp, I get a notification. That is actually useful.

The ability to pre-heat my oven without standing in front of it. That's really the big win. But also to be able to tell if spouse or children left it on.

But I can walk to my oven and turn it on, which wasn't a real problem even when I lived in a huge house. What am I missing?

We really can’t force you to see the obvious if you’re set on not understanding something.

That's alright, I'm good. Smart ovens are pretty rare, so it seems like a lot of people don't get this.

> But I can walk to my oven and turn it on, which wasn't a real problem even when I lived in a huge house.

Rushing to get out the door, "Honey, did we turn off the stove?" Presumably you can then check with your phone.

People with internet controlled ovens turn them on before arriving home to shorten the time it takes to get dinner on the table.

In my case, being able to start it heating when I'm ten minutes away from home, so that I can get the kids fed ten minutes sooner.

Ok, that makes sense.

A timer to turn on an oven has been a thing for 25 years or more, probably there were clockwork ones before that. So it is down to very fine control on timing, or not turning the oven on, say, if you're in a traffic jam.

I'd expect the network connection to go down and the oven not to turn on at least as often as 10 minute makes an operable difference.

Maybe you don't decide at the start of your day, rather it comes up later that the kids want something baked for dinner that is already prepared and just needs to go into the oven, and you also get home from work at like 7pm. Idk how niche that is, but if that's someone's situation, the remote control makes sense.

Yup. Or the buses and the queues at childcare pickup and the speed of walking home with a 3 and 5 year old means that arrival times at home vary dramatically.

Lucky guy, it sounds like you've never experienced overwhelming anxiety over having possibly left the oven on while out of the house.

That'd be a legit feature, but far less involved than a full-on smart oven. At most, it'd have a way to turn off the oven remotely (but not turn it on).

Anyway, last time I had this anxiety, I checked my Ring camera in the kitchen that's posted there for reasons like this. Works for the stove and faucet too.

While I have neither, I'd much rather use a smart oven to see its status than have a constant camera feed of my kitchen.

It only records on demand, mainly cause it's on battery. Also doesn't control anything in the house.

A simple timer that switches off after a couple of hours would do. The timer could reset every time the oven door is opened. This should solve most issues. Long cooking could have a bypass button or somethong (if the door open reset is not enough).

I just remembered many old analog electrical ovens had this already , the only way to turn on the oven is by setting a timer (and when you needed it on for many hours you had to set another timer for yourself to remember to extend the oven timer ).

Even if you left the oven on, I believe it’s very unlikely to burn down the house.

These devices are designed to work for hours with minimal supervision. Given the size of the user base, IMO ovens are extremely reliable. And electricity prices are too low to be anxious about the costs.

> These devices are designed to work for hours with minimal supervision.

I wish. A while back, I put some (food) stuff to dry in the oven on a very low heat, knowing it would take all night to dry fully. After six hours, the oven decided that I must have accidentally left it on by mistake, and switched off automatically, and that stuff was wrecked when I got up in the morning and checked on it.

Indeed. I don't think I've ever heard of an oven burning down a house. And given how pretty much everyone has one, there must be many ovens left on alone in the house at every given moment.

If you think the oven can burn the house down, how about the anxiety of trusting some companies IoT oven not to be exploited by script kiddies to burn your house down?

The more critical thing than finishing Thanksgiving dinner for your oven is not burning your house down.

I also write firmware for IoT.

IMO: one of the most important incentives towards secure IoT is: don't make it IoT unless it benefits the user greatly (instead of benefitting the manufacturer).

Don't let companies connect everything they want to the internet irresponsibly and if they must: force them to make it really really secure and long lasting (10+ years). Especially when there's high risk of causing physical damage to things and / or identity theft and / or privacy issues. Which en masse translate to national security issues.

Force them to sell devices that keep functioning even if the manufacturer (servers) cease to exist. Example: introduce a mandatory manufacturer IoT insurance that pays out in case the device ceases to function (securely) within x years (even in case of manufacturer bankruptcy).

Random chaotic brainfarts.

> Remote update mechanisms can themselves present security problems in some domains.

I've proposed many times that the device have a physical write-enable switch on it, not a software switch. That way, a malware infestation won't survive a reboot, and your backup hard drives won't get compromised.

I'm amazed that nobody does this. (Hard drives used to have a write-enable switch on them.)

Why doesn't MS do similar on all of their OS code by placing it on a read-only filesystem? Only allow updates when reboot into an update mode.

I have zero faith in software read-only modes.

Android does this. OS is read only. Updates happen on second partition and takes effect by swapping partition on reboot. Integrity is verified by secure boot.

Interesting. How is it verified? Maybe keep Merkle tree hash of eveything in TPM?

Many flash chips have write enable pins at the hardware level. So this support really does still exist in theory!

However, these pins mostly aren't used for any kind of switch in basically every PCB design I've seen. Either it's "always enabled" so the storage can always be written, or it's a chip that's programmed from the factory and always disabled to prevent any kind of user update.

I'd sure hate to have my system ransomware compromised, and when I try to restore from a backup drive, the ransomware encrypts it, too, as soon as I plug it in.

I also have, on occasion, flipped the from and to parts of the command to restore from a backup. I'd really like to have a hardware switch to make the drive read-only.

They exist, but only as very expensive specialty gear for digital forensic investigators.

https://digitalintelligence.com/store

> very expensive

Indeed, for a 2 cent switch.

These also exist for removable drives (not really any different price compared to normal drives, in wide use) in the govt/defense market, where using a hardware write switch when moving data between airgapped systems in a controlled way is common.

Thanks for this thoughtful feedback. I encourage you to file an official comment, especially regarding end-user control of update timing. Maybe my response here https://news.ycombinator.com/item?id=37394935 addresses some of your other concerns? We'd love to hear your thoughts.

Thank you for the reply! I have submitted my official comment.

This is a good point, some IoT devices really can't be designed to be physically serviceable, while still remaining reasonably compact, e.g. those that need very high levels of water resistance, especially saltwater resistance.

And adding any remote update mechanism at all would more then likely decrease overall security.

So there actually should be a counter mandate too, for devices that are impractical to design to be physically serviceable, while meeting certain size/weight/etc. requirements.

To make sure remote update mechanisms, of any kind, are never implemented, unless the manufacturer can guarantee that the update mechanism itself doesn't introduce new flaws.

It sounds like you're looking for a carve out so you don't have to upgrade your devices to have a modern microcontroller that supports remote updates and are using saltwater as a scary thing so no one challenges you on it.

You can conformal coat a ESP32 with a sensor and battery and a wireless charger, and get remote updating. If hobbyists are doing that without commercial backing, what industry experts like you have access to must be even better.

You appear to be under a lot of mistaken assumptions.

And in any case, the default ideal is to have an option to update or reprogram devices in-person.

It's never an ideal, that I've ever seen expressed on HN, to have a remote actor capable of doing so, unless in a totally air gapped environment.

I'm interested in learning! what are those mistaken assumptions?

Wouldn't a Starlink satellite qualify as an IOT edge device? How do you propose a user-servicable physical switch on a device in LEO?

Not all devices have easy or cost-effective physical access -- that's why IOT is particularly effective at bridging the digital-physical divide.

Having a category for inaccessible things in LEO doesn't indict my point that things not in LEO are typically accessible.

Good comments.

> The FCC recently issued a Notice of Proposed Rulemaking [2] for a cybersecurity labeling program for connected devices.

This sounds like it is intended for consumer products, and it also sounds optional. I would hope that users with a legitimate reason to do so (defense, enterprise) would have the capacity to not participate and forego the label.

The line between "consumer product" and "enterprise/defense product" can be blurry. For example, event security teams may use their personal smartphones to communicate with medical staff.

A lot of IoT companies (especially the startups) focus on the customers with the deepest pockets (enterprise and defense). If big-ticket customers demand this label, it generates a great deal of incentive for IoT companies to just say "to hell with it, we want that label on everything we make."

In any case, the words "national security" are usually a good way to get the attention of a three letter ;)

> Remote update mechanisms can themselves present security problems in some domains.

Not really if done right to be fair. It's just a matter of implementing a signature verification of the firmware updates that are installed on the device.

> IoT is making its way into defense and enterprise environments where reliability is a matter of national security.

If it's a matter of national security surely you don't use IoT devices connected to the public internet. At least the devices are in some private network, where the traffic is under your control. So if you don't do security updates it may be acceptable under that circumstances.

> It's just a matter of implementing a signature verification of the firmware updates that are installed on the device.

In principle: yes. In practice: signing keys seem to get leaked all the time.

It's not a mechanism I would blindly trust in security sensitive domains.

We could start by asking iot companies to be ISO 27002, stop saving passwords in excel files or in s3 bucket publicly accessible could help

Security is not an update, security is a cultural trait of an entity or an individual

> If it's a matter of national security surely you don't use IoT devices connected to the public internet.

Of course they do. That's the flip side of PaaS and reverse-NIH syndrome, the "opex > capex" thinking: "Industry 4.0" is built on web tech, with all the practices and assumptions baked in. Your critical infrastructure is, or is about to, be running JavaScript on a docker-compose cluster, and expecting to be piecemal-updated daily.

And then, there's also "shadow IT" - going behind the back of IT and using COTS SaaS to work around red tape is still... going around IT and giving untracked third-party vendors access to organizational information and operations. "Making its way" doesn't only mean "introduced by design" - those vulnerabilities just creep in.

I would love to see frank discussion on the record of consumer-grade vs infrastructure-grade practices and what label(s) would be appropriate for each! It’s not lost on me either that the roots of much high-ticket critical infrastructure is about to rest on web tech and highly evolved descendants of 8-bit micros.

> It's just a matter of implementing a signature verification

correctly

With rollback protection, a chain of signing keys to allow revocation, time stamping, correct parsing, enough scratch space to hold an entire separate image in A/B, enough processing power to verify image signatures, etc etc.

Doing this well is very hard. Given most IoT vendors don’t yet know how to prevent XSS, there’s near zero chance they’ll get updates correct in the next decade without solid open source frameworks to leverage.

Also with right to repair, you need to be able to disable signiture checks and upload custom firmware

Yeah true, that’s a fascinating challenge that I haven’t seen particularly well discussed anywhere in detail.

It needs to be really well protected so that the owner, and only the owner can disable the chain of security, but that they can do it without unreasonable overhead and without actually involving the manufacturer (in my opinion. To handle examples like manufacturer trying to lock-in or charge fees or simply going out of business)

Perhaps the owner mints a public key pair, and the device only unlocks with proof of the private key. But in a way in which is easy for your every day person.

Remote update mechanisms will be used to deliberately brick or downgrade the device (usually for petty commercial reasons), by unintended parties (e.g. Chinese spies with leverage over Chinese manufacturers), at the wrong time (e.g. one hour of outage when the smart oven should be cooking dinner), and so on.

Signature verification without human users controlling the updates means protecting an attack vector.

It would help if there were reference designs and testsuites for common embedded tasks like remote updates, configuration storage, input validation, etc.

I don't do much embedded work these days, but I remember a lot of shoddy, ersatz designs based on random OSS components. I remember a small mfg of hardware devices used widely in payment processing tell me they were secure because "they used SSL" despite not taking care of physical or system security in a coherent, effective way.

Honestly, companies need to have some liability or other incentives to even care about security a lot of the time.

Reply on news.ycombinator.com