Another Vulnerability in the LPC55S69 ROM | Hacker News Reader

Another Vulnerability in the LPC55S69 ROM

oxide.computer

160 pointsbcantrill55 comments

Since Oxide folks are reading: I'm curious about the details behind the choice of this specific chip - you found a problem last year during evaluation, it's now "chosen part for our product's Root of Trust implementation" and you found another problem. No reason to assume anything else would be any better? Nothing else really on the market that does some specific thing you need? Some of the BigCo's have opened their solutions to this problem - not viable for you to make/obtain, also broken, not actually as open as they claim, ...?

Yeah, big sigh here. So, a bunch of things. First (and to be fair to NXP), a secure microcontroller is a hard problem, with tricky bits in both hardware and software. And it remains to be true that the LPC55 is pretty good at a bunch of things that we need (e.g., a PUF).

Second, while it's true that we're (obviously?) not entirely satisfied with the LPC55, we have worked around the vulnerabilities we have found -- and the alternatives that we have found aren't better. In particular: there appears to be no alternative that is at once robust, mature, secure, and entirely open. And even in the RISC-V ecosystem, there is a disconcerting trend towards proprietary boot ROMs (!!). We definitely welcome alternatives though, so please prove us wrong on that front! (That said: any alternative will be too late for our first product; we will have the LPC55 in our system for at least the foreseeable future.)

In terms of the BigCo's having "opened" their solutions, they haven't really (sadly). OpenTitan -- which we were really bullish about! -- only existed as an FPGA during the time we were trying to build on it. We actually wanted to make a go of making a secure FPGA -- but the problem is harder and the ecosystem is more proprietary than with secure microcontrollers. There are (or were) "plans" to make an ASIC, but we were asked to pony up $500K to join the lowRISC Foundation to find out what those plans were. (!!) To be blunt: that's not open -- it's open-washing. (I hope OpenTitan has seen or will see the error of its ways and opened up an ASIC roadmap -- it's an important effort and we would like to see it succeed.)

As for the other BigCo's, of the ones that I know about, they are either not using an RoT (!) or are using one that's strictly proprietary (i.e., no idea) or they are using the (wait for it...) LPC55.

So, yeah. Big sigh. tl;dr: We would welcome something better; we need to live with the LPC55; we would love RISC-V to not repeat a bunch of proprietary mistakes; we would love OpenTitan to be ActuallyOpenTitan.

Why dont you use what Chromebooks use?

We would love to! But they use Titan -- like, entirely closed Titan. (Ditto Apple's T2.) But the Chromebook is very much a model for what we're trying to do; I've said this before, but I feel like the Chromebook is one of the most underappreciated products in computing exactly because of its approach to security.

On ChromeOS, our root of trust is actually the SPI ROM. We allocate a RO section of the SPI ROM that is controlled with the write protect pin. This RO section contains a full copy of the BIOS including vboot and the keys used to verify the RW-A/RW-B sections. We use this to build a chain of trust. RO FW verifies RW FW, RW FW verifies the payload (depthcharge), depthcharge verifies the kernel, the kernel verifies the rootfs using dm-verity.

Here is a great video by Duncan explaining the whole processes in detail: https://youtu.be/HwsTRThChn0

All the components are open source too. You can read more about it here: https://doc.coreboot.org/security/vboot/index.html

As for the security chip, all current Chromebooks ship with a CR50. This is a Google designed chip. The boot rom is closed source unfortunately, but it essentially just verifies and jumps to RW. You can find the RW code here if anyone is curious: https://source.chromium.org/chromiumos/chromiumos/codesearch...

The security chip gives us TPM2 functionality and some ChromeOS specific features like CCD: https://chromium.googlesource.com/chromiumos/platform/ec/+/c...

You should be able to use vboot with a different TPM. Reach out if you want to chat!

And who are you?

What I'm hearing here is "Yes, Google designed their own chip because everything on the market sucks."

A lot of the closed source boot Roms look to be SOC vendor code: they'd buy a subsystem ip from someone and have to redistribute the code for that thing but don't have the source licensed.

Unlikely that will ever be cleared up since the IP vendors are very precious about their stuff

Heh, so yes, everything is indeed terrible as feared. Thanks for the comprehensive answer, and especially sad to hear about "ActuallyOpenTitan"... which seems weird, especially since they are involved in the Skywater open ASIC thing.

I just want to add "low power" to "there appears to be no alternative that is at once robust, mature, secure". Finding a good micro for security applications is a nightmare.

Kind of unrelated here, but wanted to mention Precursor [1]. They just shipped, it's still an early product but... something works.

[1] https://betrusted.io

DO you think that there is actually a significant market for a secure open-source microcontroller? It seems like the BigCos have chosen to do things like Titan (open source but not really), Cerberus (also open source but not really), and T2 (extremely closed source). Traditional microcontroller companies don't seem to be interested. The whole root-of-trust industry is also moving towards extremely-closed-source microcontrollers inside your big SoCs.

In the interest of disclosure, I'm actually in a position to start a secure open-source MCU project/company, but it looks like a terrible dead end.

> DO you think that there is actually a significant market for a secure open-source microcontroller?

I don't think we'll ever know. It is kind of amazing how secretive chip makers are despite the absolutely massive moat they enjoy. Not only is there a fabrication barrier to entry approaching the size of a small nation's GDP, but there is a maze of intellectual property laws backed by supranational leg breakers. And still they operate as if none of that existed and only trade secrets provide them any measure of competitive edge. Now imagine how bad it would be if the US military hadn't put multisource requirements in place...

I am very much looking forward to the day that their BS argument for secrecy might actually make some sense, because that'll be around the time that it becomes practical for businesses operating out of home garages to fabricate their own designs and start eating NXP's lunch.

Shameless plug: If you want to hear a bit more history about the previous vulnerability on the LPC55S69, we talked to Laura about it in our podcast: https://unnamedre.com/episode/55

That plug isn't shameless at all -- that episode was great! Hopefully you'll have Laura back to discuss this one...

Credit to Oxide, the fact that they're finding this stuff implies they really know their way around their hardware/firmware. This finding actually gives me confidence in what Oxide are doing.

After hearing some stories from friends who work with microcontrollers IMO the most impressive thing about this is that Oxide were allowed to publish the vulnerability at all

NXP knew enough to not ask. To their credit, we got much less runaround on this vulnerability than with the vulnerability that Laura and Rick found a year ago.[0] (It should also be said that what Laura originally sent to them left little room for negotiation about the seriousness.) At some level, NXP seems to appreciate that we're helping them improve their products -- or perhaps they're just afraid of what we'll find next? Either way they were at least marginally better.

So all of that is an improvement, certainly, but it's still not what we need: the source code to the ROMs. We believe emphatically that we need transparency throughout the stack, down to its lowest levels. We need open ROMs, open FPGAs, open ISAs, open firmware -- not just because it's the right thing to do, but because it will result in more secure and more reliable infrastructure!

[0] https://oxide.computer/blog/lpc55

Aren't you afraid this might get your relationship with this supplier tougher ? Not necessarily with engineers, but if some C-level decides that you're giving them bad publicity and blacklist you or making you pay public price for all products ?

No, we're not afraid. Even with the partners that have been a real challenge for us, there is recognition that (1) we know what we're doing and (2) we want them to succeed. It should be said that that's not by accident: we have taken a deliberate, values-based approach to partner selection (and we differentiate between partners and vendors). Specifically, we explicitly look for partners that share our value of responsibility. Indeed, we have written an RFD[0] on the shared values that we seek in a partnership that we send very early on in the relationship; if a C-level doesn't buy into shared responsibility with the fate of their customers, we will find someone who does.

[0] https://oxide.computer/blog/rfd-1-requests-for-discussion

Lots of issues with microcontrollers, even secure ones, can be found around the ISP routines. I remember issues with NXP Micros (also LPCs, but of another generation) where, if you glitch the power at just the right time, you can get it to ignore code readback protection.

I'm not sure that getting the manufacturers to open source the ROM code is the right solution, per se, as even if it was known beforehand that there was an issue here, you would still be left with the same solution space: disable the affected code paths, and prevent access to them with additional (external) integrity checking.

While it’s true that it being open wouldn’t have changed the solutions, nor strictly speaking prevented the bug, it would make audits by third parties (like us) easier. And that’s really important.

Voltage glitching is a serious but broadly orthogonal issue (and one that has a very different threat model than, say, a compromised software supply chain). We actually do believe that opening this ROM would have prevented the problem because it is in fact glaring -- and the demographic for these parts really cares about this! So it seems highly likely that this problem would have been found many times over, and certainly would have been found in an early rev of the part that would have allowed them to fix it in a subsequent mask.

We need open source ROMs (and open source firmware!), and we collectively need to stop finding excuses for vendors to not provide it.

The usual convincing argument comes in the form of large amounts of profit or lack there of along with liability waivers. Nothing else convinces companies faster. NXP is arguable one of the better micros to use as well, at least the Freescale derived parts.

No amount of pleading or finger pointing is likely to work.

Alternatively an open source asic based on open source cores and IP that you can have on an asic or fpga that people start having made in large quantities cutting out the middlemen might work as a serious spooky moment for the likes of microcontroller vendors.

I can see it happening, riscv and litex make it dead simple to create your own custom soc on an fpga, perhaps with custom accelerators or I/O functions. It’s really quite amazing.

Imagine taking a $7 or so max10 or lattice ice40 and rolling your own. No it’s not as efficient perhaps, but efficiency isn’t everything. Flexibility to correct things at a logic level, after the delivery, could be a godsend in some scenarios where difficulty in servicing or replacing hardware far exceeds the efficiency gains.

There’s also some neat new chips that combine a hard cpu core, some hard ip for bus interfaces, and pl. perhaps the best of all if the pl is easy to use and program with open tools. See quick logic for example.

FPGAs just have a different set of attack vectors. Maybe it's smaller and more esoteric, but they're not a silver bullet.

Agreed that they definitely aren't a silver bullet! We went pretty deep down the secure FPGA route -- they (such as there is more than one vendor, which there arguably isn't) are very, very proprietary. (Not only do you not have source code to ROMs, there are entire hidden cores!) Indeed, we ultimately abandoned that route for quotidian reasons: we couldn't get their proprietary toolchain to work reliably on a Linux box, and we were getting the runaround. Our belief was that if their support was so bad on something so simple, it was going to be much worse when we found more serious issues with the device. Our view was (and is) that open FPGAs are a necessary (but not sufficient) step to get us to secure FPGAs.

You might be interested in Precursor and the Betrusted Platform.

https://www.crowdsupply.com/sutajio-kosagi/precursor

They are trying to have an hardware platform that can be inspected and it is based on an FPGA with a RISC-V Softcore.

Its by Bunnie, and he great talks about the choices and why he made them:

Keynote: Precursor - Trustable Open Hardware for Everyday Use - Bunnie Huang (https://www.youtube.com/watch?v=Fw5FEuGRrLE)

They are also doing their own Rust Message passing OS called Xous that might be of interest.

https://github.com/betrusted-io/xous-core

Interesting! I (obviously?) hold Bunnie's work in very high regard, but I had not seen Xous before. It seems to be similar in some respects to Hubris[0], but it's hard to discern its design center as most of the documentation appears to be forthcoming? Regardless, it's great to see more work in this important domain!

[0] http://cliffle.com/blog/on-hubris-and-humility/

Xous seems to be one step up from Hubris in terms of being oriented towards interactive applications and running full UI and dynamically loading applications.

I have not enough knowledge about the underlying message passing semantics to give an educated differentiation.

Maybe the most interesting feature is the Plausibly Deniable DataBase (PDDB). See:

https://www.bunniestudios.com/blog/?p=6307

Here is some of the design intent behind Xous explained by its main designer:

https://youtu.be/_pIr3Q7gqNI?t=959

He described Tock as to small and Redox as to big in one of the talks.

bunnie made a 1-hour video that walks through the Precursor tech stack (no, not ideal from an accessibility perspective):

https://www.bunniestudios.com/blog/?p=6336

IIUC, Xous is designed for environments that aren't quite as constrained as the microcontrollers that Hubris runs on. Precursor has 16 MB of RAM and 128 MB of SPI flash. There's some kind of support for separate executables, but the OS and applications are packaged in one image. Whatever flash isn't used by that image is for user storage. Also, they've got the Rust std library, or at least some subset of it, running on Xous.

Incidentally, bunnie is working with me on a version of Precursor with a braille keyboard and text-to-speech output. I got my prototype unit today. So I'll be digging into Xous more soon. bunnie wrote a nice article about the process of designing the braille keyboard here:

https://www.crowdsupply.com/sutajio-kosagi/precursor/updates...

and covered more recent developments on that front, including meaty technical details on the software side, in a recent status update:

https://www.crowdsupply.com/sutajio-kosagi/precursor/updates...

I still need to figure out the use cases that Precursor is best suited for, given the modest hardware specs, but it's still cool to have a fully open mobile hardware platform that's fully accessible.

I agree with you!

Is Oxide pushing the state of the art forward with any of the RISC-V vendors?

We’re not currently using RISC-V, though you never know what the future may hold.

How difficult is it to harden against voltage glitching?

Couldn't the manufacturer build the chip so that as soon as the voltage dropped below some reference it asserts the reset line or cuts power or something?

Fast brownout detection into reset is a hard problem. Especially as process nodes shrink. So the answer is difficult and it's a fairly high effort "can't you just..."

Here's some fun literature to read:

http://nice.kaist.ac.kr/files/attach/filebox/711/003/Interna...

http://hajim.rochester.edu/ece/sites/kose/files/conferences/...

Depends where you draw your boundaries. Physical access is always going to be complicated, but on larger scales (and increasingly as you can make things smaller) you can have a cap tank that must accumulate enough charge to do reliable brownout detection before device powers on.

Can anyone explain Oxide's product beyond the (fairly terrible IMHO) site? Seems like some kind of prepackaged VMware-alike including rack + hardware + networking fused into a single uncustomizable product?

What is the intended application?

I wrote this in the thread last week: https://news.ycombinator.com/item?id=30678324

Thanks. It seems extremely reminiscent of e.g. the Cobalt RaQ ( https://en.wikipedia.org/wiki/Cobalt_RaQ ), but maybe "AWS Outposts for the rest of us" might be a more suitable contemporary parallel

I don't know why people are so allergic to the terms private cloud, (hyper)converged infrastructure, cloud repatriation, etc.

It's because the well got poisoned by marketing teams selling things that were not actually a material improvement over what came before.

Why can't NXP use the ROM patch mechanism (which was the cause of the previous vulnerability) to fix this? Was that all in vain?

If it's anything like the patching in some common bluetooth controllers, the patching area was filled up a long time ago with new features and they don't want to respin the base silicon.

Yeah, looks like it :(

I read out the patch data from two LPC55S69 I have at hand, and...

MCU Date Code: 2019-07-20: 11/16 slots used 2019-11-05: 14/16 slots used

I find it very hard to believe NXP will open source their design, given that many of the logic blocks are probably used across their whole company. At best maybe they'll share code with Oxide under a hefty NDA.

Well, we just want them to open the ROM, but yes. For whatever it's worth, they won't even let us look at the code, let alone share it: not that it's our preferred model (at all!), but we offered to review the code offsite in a disconnected room, etc. They were... not amenable.

The fact they're not willing to share it raises eyebrows... security through obscurity eyebrows.

Or there are licensing issues - in some cases there might be non-transferable licenses for bits of things in the ROM that are held by NXP but which are problematic to show to someone not employed by NXP.

Sometimes it also happens that company finds out they broke a license and there's no way to fix it silently...

More likely just bureaucracy. All the agreements I've been involved with negotiating were worded to cover disclosure to contractors, third party security review, etc, provided such parties were covered transitively by the same disclosure restrictions as the licensee.

It gets even worse. MS can't open source Windows because there's licensed third party components. A lot of silicon vendors can't open source their design for the same reason.

I would think that the bootloader code wouldn't use any funky IP blocks - it's not some sort of driver code into which applications call, it's just a special application that runs before anything else. It has to be able to talk UART (or some other interface), check signatures, and write flash. All of these the user code can do as well.

There are many "user-space" bootloaders as well for various chips. The factory bootloader is only different in that it sits in ROM. For example, RP2040 ROM bootloader is here https://github.com/raspberrypi/pico-bootrom

I am currently trying to find their Common Criteria certificate for that product, but it seems like there is none for the LPC55S69 according to their forums [1].

[1] https://community.nxp.com/t5/LPC-Microcontrollers/Does-the-L...

The NXP supply chain is broken. Can this part even be gotten today, or a year from now?

NXP has already moved Kentits delivery times out multiple times. Companies are on the verge of bankruptcy because NXP is not delivering.

Queries go unanswered or get a stock "52 weeks", after being told "52 weeks" 53 weeks ago.

About a month ago, I took notice of LPC processors in general as "unbrickable" to the extent they all (AFAIK) have a serial bootloader, etc.

At some hazy and facetious level, does a vulnerability such as that described here connote a deeper level of being "unbrickable" -- say if a vendor lost its own key?

Reply on news.ycombinator.com