Adventures in reverse engineering Broadcom NIC firmware | Hacker News Reader

Adventures in reverse engineering Broadcom NIC firmware

devever.net

136 pointshasheddan21 comments

Author here. This is finally being given as a talk by me tomorrow at 37C3. I've wanted to give this as a talk for many years at CCC, but CCC's long hiatus got in the way. This will be my first time at CCC.

If anyone attending 37C3 wants to talk more about this, or anything else, or about open source firmware/owner control in general, don't hesitate to get in touch with me in person or online.

I don't have access to my usual email when traveling for security reasons, so use my travel email: https://www.devever.net/~hl/contact

I'll also set up a DECT phone at the event (4526/HUGO).

Comments and questions welcome!

Mind explaining your travel security protocol in more detail? I’ve never heard of using a separate travel email address, and I’d like to understand your reasons for doing so.

Impressive work. Looking at the presentation slides which accompany the 37C3 talk, the author covers their "Your princess is in another castle" experience. Turns out they fully reversed the firmware for the MIPs cores (one core per port) only to discover the MIPs cores were almost entirely vestigial relics retained from past generations. I think anyone who has done real world reverse engineering can empathize with having spent a significant amount of time and hard work reversing what turns out to be a ton of dead code. In the end, success in reverse engineering is mostly about being relentlessly persistent in your pursuit to understand what is going on and getting back up after big set backs and disappointing dead-ends.

There's a lot to be said for when dynamic analysis is possible; it doesn't have to be perfect in order to eliminate code that isn't involved in an interaction (worst case you cause that code to die horribly & observe the fail, or lack thereof)

A few years ago, I was playing around with an even older variant of this Broadcom chip. It turns out you can configure the PCIe PHY into root complex mode and hook the NIC up to other PCIe devices. You just need to supply the 100MHz REFCLK for both devices. And connect the TX pair to the RX pair of each device (crossover).

Registers 0x7d10 (sequence number) and 0x7d38 (TLP data) allow you to send custom PCIe TLPs, which let you do the configuration space writes required to get the other card to talk to it. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&d... (see page 383)

It's quite suprising to see one NIC blinking the LEDs of another, by writing to its registers over the point-to-point PCIe link (I used twisted-pair wire-wrap wire to connect them together).

Here's the code to do it, you run it on the MIPS processor: https://pastebin.com/ZJdaUpRZ

This is enough to load custom firmware on a GPU, and get the two talking together, with network access, by the way!

Also the Intel i210 NIC can do this as well, it has an ARC Tangent-A4 processor in the management engine. Same for the i225, but I believe that one has secure boot to prevent unauthorized firmware from running.

The 2019 submission which had some commentary from the author: https://news.ycombinator.com/item?id=19679640

What would it cost for a philanthrope to produce a fully-open-source chipset/soc/board for Linux?

Nothing, because they already exist.

A few millions, not more -- maybe even less but I am not up to date with post Covid ASIC engineering costs.

We obviously want 100GbE because frankly why bother with less? So you need to translate https://github.com/corundum/corundum into an ASIC and then produce probably a few (ten) thousand ICs to make it worth it. However, you quite likely can get away with an old node -- the Intel XL710 40GbE chip, for example, is produced on a 28nm node.

So the production will be cheap, the initial ASIC engineering and prototyping is going to cost you a bit. But the (very) hard part is luckily already done.

You run into ToCToU issues. Even if a chip is "open", how can you be sure the chip you have corresponds to the specification? If that isn't a concern, any FPGA based SoC fits the bill, for example the MNT Reform Kintex-7 module (https://shop.mntre.com/products/mnt-rkx7-fpga-module).

You could use a Lattice ECP5 FPGA as a fully programmable/customizable NIC, you'll need an external PHY/transciever, this FPGA has 100% open source tooling available. There's an open source BMC implementation for this FPGA, it should be easy to modify to function as a plain Gigabit PCIe NIC.

https://news.ycombinator.com/item?id=27105813

https://www.youtube.com/watch?v=OLoKQ1nAkis

This work resulted in what seems like the only open source firmware for Ethernet NICs. Is anyone aware of any other ones?

https://wiki.debian.org/Firmware/Open

Noob question but why is it the firmware that needs reverse engineered and not the driver?

In the simplest sense, because they already have source for the driver, and not for the firmware.

More broadly (no pun intended), NIC vendors want to work with Linux and the GPL means they have to release the source of a driver to do so. No such legal requirement applies to firmware.

Because the open drivers are good enough but have to deal with a proprietary blob that can't be fixed for bugs and has undefined features/behavior.

In this instance, it appears the author's motivation was to facilitate a clean room reimplementation by "producing a natural-language specification for others to reimplement". In other instances security researchers might reverse firmware in order to find vulnerabilities. As the article states:

> One example motivating the production of open source firmware for the BCM5719 is that it's the only closed-source firmware blob found in the Talos II, a high-performance POWER9-based system otherwise wholly free of firmware blobs... Once this is delivered, it will be possible to use Raptor's POWER9 systems with purely 100% free, open source firmware. As far as I am aware, there is no other machine in the same performance class which can make such a claim.

>All Raptor systems shipped after May 10, 2021 use the open-source Ortega firmware for the BCM5719 device.

https://wiki.raptorcs.com/wiki/BCM5719

My old team did some work in this area a few years ago. We got the Talos II BMC code to be binary reproducible, and had a go at automating David A. Wheeler's compiler diversification to stop compiler subversion. We checked the boxes we intended to, though never got enough funding to polish it up. It's probably broken now, but we did post a portion of our work on gitlab: https://gitlab.com/deepthirst.

Because vendors have realized GPL condoms are a thing and have started basically sacrificing the driver layer to the legal requirements of GPL, while keeping the secret sauce secret through firmware.

Firmware is the new proprietary/FLOSS boundary layer.

Didn't the GPLv3 work to eliminate this? With the "Tivoization" thing

Unfortunately (for better or worse) GPLv3 flopped

There are a lot of misconceptions around Tivo. Tivo did not do what people think Tivo did and refer to as "Tivoization". Tivo broke their proprietary software when you modified the installed Linux kernel. Both the GPLv2 and GPLv3 allow what Tivo did. Both GPLv2 and GPLv3 require that users can modify installed GPLed software.

https://sfconservancy.org/blog/2021/jul/23/tivoization-and-t... https://sfconservancy.org/blog/2021/mar/25/install-gplv2/ https://events19.linuxfoundation.org/wp-content/uploads/2017...

The situation around GPL "condoms" would be the same for both GPLv2 and GPLv3 too, if the firmware can be considered a derivative work then there may still be a GPL violation, but more likely the driver<>firmware interface would be fairly high-level, so all the functionality is in the firmware.

Reply on news.ycombinator.com