Packets of Death
blog.krisk.org
blog.krisk.org
Debugging deep down the rabbit hole, until you find a bug in the NIC EEPROM - and the disbelief many show when hearing a software message can bring down a NIC.
I for one would enjoy reading more content like this on HN that what qualifies as best as a friday-night hack
Shouldn't be a surprise to anyone. Firmware is just software, and it necessarily deals with raw bytes. Not really surprising that it can contain bugs that are triggered by certain byte patterns.
For any given piece of gear in your server room there is likely a massive list of firmware bugs for it and its subcomponents. Sometimes they are fixed, sometimes not. When you run in to one that affects and need to deploy a firmware fix to dozens or hundreds of servers, it's not a whole lot of fun.
This invalidates my assumption that a shop like Intel probably uses formal verification in firmware development.
Its also scary to consider how many very important (nuclear/damn control, etc) systems while themselves might be formally verified are dependent on services of lower level software (OS, drivers) and hardware (firmware) that are not...
PS You should be very frightened. The industrial control systems industry are slow to adopt new tech/new best practices/anything else new; and even more reticent to touch anything that "works." Tons of vulnerable equipment controlling dangerous things with default passwords (if any at all). I could go on for days.
In the wild what is more common in my experience is to use a formal specification language like Z or Object-Z and then recreate the the program in a "normal" language using pair programmers and/or extensive code review to attempt to ensure it conforms to the specification. While not perfect, for critical things this approach is far more time consuming and less bug prone than typical development.
I have also seen limited attempts at writing code generators that take Object-Z specifications as input. My understanding is that this is an area of ongoing research, but even a program that generates code for a subset of a Object-Z specification can save a lot of time.
Example research: http://www.jot.fm/issues/issue_2008_11/article6.pdf
Just browse through the ./drivers tree of the kernel source some time and look at all the "quirks" and "workarounds" there. Recognize that virtually all of those could have been fixed in firmware, but weren't because no one cared.
It's just deeply depressing. This was a virtuoso debugging performance, but it didn't have to be that way if hardware companies were sane. But they aren't, and we all pay for it.
(edit: I just checked for myself, the Linux e1000e driver apparently doesn't support runtime firmware update via the kernel API, so linux-firmware wouldn't be expected to have this. I don't know what the process is for an affected end user to get a copy. I suspect there is none.)
HIGHLY recommended to anyone who enjoyed the article.
It's a remote PHY injection at the hardware level - basically hacking the chip directly thanks to "special" content, i.e. putting a packet header within the packet.
One of the more painful summers resulted from a QLogic HBA which sometimes, for no apparent reason, injected a string of hex digits into the data it transmitted. There is a commemorative t-shirt of that bug with just the string of characters. It lead NetApp to putting in-block checksums into the file system so that corruption between the disk and memory, which was 'self inflicted' (and so passed various channel integrity checks) could be detected.
Here at Blekko we had a packet fragment that would simply vanish into the center switch. It would go in and never come out. We never got a satisfactory answer for that one. Keith, our chief architect, worked around it by randomizing the packet on a retransmit request.
The amount of code between your data and you that you can't control is, sadly, way larger than you probably would like.
We had problems when some NFS traffic would end up getting stalled. Our NFS server would use UDP packets larger than the MTU and they would end up getting fragmented.
Turns out the NIC would not look at the fragmentation headers of the IP packet and always assume a UDP header was present. From time to time, the payload of the NFS packet would have user data that matched the UDP port number the NIC would scan for to determine if the packet should be forwarded to the BMC. This motherboard had no BMC but it was configured as if it did have one.
It would time out after a second or so but in the meantime drop a bunch of packets. The NFS server would retransmit the packet but since the payload didn't change, the NIC would reliably drop the rest of the fragments of the packet.
Of course Intel claimed it wasn't their bug ("it's a bug in the Linux NFS implementation") but they quickly changed their tune when I coded up a sample program that would send one packet a second and reliably cause the NIC to drop 99% of packets received.
While it turned out to be a fairly lame implementation problem on Intel's part (both by ignoring the fragmentation headers and the poor implementation of the motherboard) I have to say it was very satisfying to solve the mystery.
Having the NIC inspecting incoming frames and potentially diverting them to the management controller sounds like a scary proposition. I'd almost rather just have dedicated Ethernet hardware for the management controller. The decrease in switch ports needed is certainly seductive, but I wonder if it's worth the risk.
(Do you happen to recall which Intel motherboard this bit you on? I was just getting out of whitebox Intel motherboard-based server builds about the time you're describing, but I'm just curious if only for the nostalgia.)
Yikes! Sounds like system management mode in a BIOS!
You'll enjoy this (or be horrified by it): http://fish2.com/ipmi/itrain.html
Yeah, be careful with it. Firewall it silly. But recognize that it's a tool that can be very useful.
Being grafted onto the same NICs on the server computer that you might potentially expose to the Internet makes firewalling them a more difficult proposition. I get a lot of piece of mind from having management interfaces on an an out-of-band control network whenever possible.
Or, "Sounds like an advanced persistent hardware rootkit!"
If there's denial of service hiding in there I wonder about what other security bugs might be lurking. It's scary stuff, and pretty much impossible to audit yourself.
Edit:
Also, I'm a little freaked-out that the EEPROM on the NIC can be modified easily with ethtool. I would have hoped for some signature verification. I guess I'm hoping for too much.
Edit 2:
I wonder if this isn't the same issue described here: https://bugzilla.redhat.com/show_bug.cgi?id=632650
Depending on what functionality is being offloaded to the NIC (are there still NICs that do IPSEC and crypto offload?) there's the possibility for information disclosure vulnerabilities in the NIC itself. Yikes.
Yes, but this coprocessor has access to the host's PCI bus. That is enough to totally pwn the machine, since it gives read/write access to all memory (well, all memory below 4GB, IIRC, but that's enough.)
Generally the EEPROM does not contain anything like executable code, although it probably can contain patches for microcode on many NICs.
I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again.
This is a very excellent case for fuzz testing. My thinking is that you want to whip up your Ruby and your EventMachine and Redis going and run a constant fuzz with all sorts of packets in your pre-shipping lab.
The idea is that you want to create a condition where you do see it, and the other handful of lockups that are there that you haven't yet seen.
Since it's caused by a specific byte at a specific place, surely you'd only need to fuzz an average of 256 packets (of the required length) to find it... which suggests it wasn't done at all... zero...
A true fuzzing run for something like this ought to run in the tens of millions of packets.
There were 4 buggy bytes. 3 would crash the card. 1 would fix the card ("inoculating" it). If they only fuzzed once, or fuzzed with the same dataset (random.seed(1) // seed chosen at random), then they wouldn't catch it.
Even then, it's pretty sloppy.
Given that, for all we know the relevant parties did conduct extensive fuzz testing and your condescension is misplaced.
It wouldn't surprise me at all if there are other issues that don't depend on these exact circumstances to get something to fall over.
As a telecoms engineer predominantly selling Asterisk for the last 4 years and Asterisk experiance extending back to 2006 it's shocking to see this finally put right. For so many years, I have avoided the e1000 Intel controllers after a very public/embarassing situation when a conferencing server behaved in a wierd manner disrupting core services. Not having the expertise the author has, I narrowed it down to the Eth. Controller, Immediately replaced the server with IBM Hardware with Broadcom chipset and resumed our services in providing conferencing to some of the top FTSE100 companies.
Following this episode, I spend numerous days diagnosing the chipset with many conference calls with Digium engineers debugging the server remotely. In the end, no solution, recommendation to avoid the e1000 chipset and moved on.
Everything gets serialized/deserialized these days, so there's all kinds of boundary conditions where you can flip just the right bit and get the data to be deserialized the wrong way.
What's more interesting is that it bypasses all of the checks to prevent this from happening.
Here is the wiki page on the INVITE OF DEATH which sounds like the problem you hit:
Can you elaborate? I recognize the words, but not the meaning.
"Oh yeah, I used to work at Intel - that nic's got a YAML parser in it"…
For e.g. a NIC, it's not that many things that need to go wrong when you encapsulate a packet it should forward out on the wire so it rather looks like a control packet triggering some undesired results on the chip. Or vice versa.
Definitely not this bug, the one linked is not intel specific.
"With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death - and kill client machines behind firewalls!"
That's worrisome, I'll bet there are lots of not-so-nice guys trying to figure out a way to do just that. There must be tons of server hardware out there with these cards in them.
Edit: Per http://www.kriskinc.com/intel-pod it does work on broadcast frames. Yikes!
The test would be to find out how common that particular NIC is out there - and grab a few and test out his method.
Looks like it would be fairly trivial to setup duplication given the author did all the heavy lifting in finding this. Putting a small AWS bot-net up that just sweeps massive IP blocks would be easy - heck you could do it really easily from a single machine, it would seem.
If you can take out a machine with a single packet....
off-topic:
A long time ago we found that a certain ping packet would be dropped with about 30%, which in turn triggered a monitoring system to register 'server down' when enough packets in a row were missed.
This would happen about once every day or so, leading to an operator being paged (usually at 3am). Very annoying problem and incredibly hard to debug. We'd replaced just about every piece of hardware except for a stupid little T-connector. My buddy Jasper and me looked at it and we both more or less at the same time said 'it can't be'. We swapped out the T-connector, problems solved.
It took the better part of a day to nail that one, I still remember the hostname (chopper) of the SGI box that the thing was connected to (SGI Challenge, an Indy sold as a server with one of those silly thinnet adapters dangling off the back, even though it had a UTP connector too).
Some bugs... I can't say I'm mourning the demise of coaxial ethernet and the bus topology.
Heck, even just a few months ago I spent 2 hours troubleshooting a 10G fiber connection on all brand new gear before swapping out the brand-new cisco 10G SR SFP module which was DOA.
Thinking the same exact thing after swapping out everything else, including all patch cords on both ends "It can't possible be this SFP module"
Yup.
And the more disruptive the attack the better. Because it shall help have better process put into place, at every level, to mitigate such issues in the future.
The last thing I want is a deadly packet not breaking havoc only because it's payload is not known.
If a single packet can bring down a big part of the net, then by all means make it happen and make it as bad you can. And then we can start thinking about how to build a more reliable and more secure net.
I have a total digital schadenfreude for such things, except for stuxnet and duqu which are evil and scary.
$ wget http://zorinaq.com/pub/intel-packet-of-death.txt
It has 0x32 at offset 0x47f regardless of the size of the IP and TCP headers. Try to run the wget AS SOON AFTER HAVING COLD BOOTED the machine (it is the very first packet of 1152+ bytes that determines if the NIC will crash or be inoculated until the next cold boot; well... unless it is the "no-op" packet).
Edit: fixed link.
/edit Ah, I spoke to soon; the author has updated his page here with diffs between affected and unaffected EEPROMs:
Sometimes bug fixing simply takes two people to lock themselves in a room and nearly kill themselves for two days.
You'd think the odds of getting a packet with 0x32 in position 0x47f is almost 1/256 per packet? So why aren't these network cards falling over everywhere every few seconds?
# awk '/eth/ { print $1 }' <(ifconfig -a) | cut -d':' -f1 | uniq | while read interface; do echo -n "$interface "; ethtool -i $interface | grep driver; done
eth0 driver: e1000e
eth1 driver: e1000eThe linux e1000e may support many chipsets, so the fact that it's in service on your box doesn't necessarily mean you're running the suspect chipset, or that it's vulnerable.
Check with lspci -v, and check with the concrete test using the cold boot+magic packet others and the OP have posted.
Intel 82574L ethernet controller looks to be popular too. Intel, Supermicro, Tyan and Asus use it on multiple current motherboards and Asus notably on their WS (Workstation) variants of consumer motherboards, e.g. the Asus P8Z77 WS (socket LGA 1155) and Asus Z9PE-D8 WS (dual CPU, socket LGA 2011).
Could this be a bug in the code of the EEPROM that handles TCP offloading, or one of the other hardware features that are now becoming more common? (https://en.wikipedia.org/wiki/TCP_offload_engine)
There are subtle differences between the offsets I get when I run "ethtool -e interface" versus those in the article that indicate an affected card (but they're quite close).
Mine are:
0x0010: ff ff ff ff 6b 02 69 83 43 10 d3 10 ff ff 58 a5
0x0030: c9 6c 50 31 3e 07 0b 46 84 2d 40 01 00 f0 06 07
0x0060: 00 01 00 40 48 13 13 40 ff ff ff ff ff ff ff ff
Output of "ethtool -i interface" (in case anyone wants to compare notes):
driver: e1000e version: 1.5.1-k firmware-version: 1.8-0
I tested both packet replays by broadcasting to all attached devices on a simple Gbit switch and no links dropped.
EDIT: it's difficult to tell definitively doing it remotely but I still can't re-produce the bug after a cold boot.
Basically one linux box with NVidia embedded gigabit controller could take down the whole segment. It would only happen after a random period, like after days when the box was busy. No two machines connected to the same switch would be able to ping each other any more after that. I suspected the switch, bad cables, etc. In the end I successfully circumvented the problem by buying a discrete gigabit ethernet card for the server in question.
He was instrumental in taming the Soekris and Alix SBC boards of old and creating Asterisk appliances with them. If you've got a little asterisk box running on some embedded looking hardware somewhere, it doesn't matter whose name is on the sticker, its got some Kielhofner in it.
I live about a mile from Star2Star. I ought to pop in one of these days and see what they're up to.
I'm currently working on an open source project where we are chasing "hang really hard and need a reboot to come back" issues with exactly this same ethernet controller, the Intel 82574L. I wonder if it's related!
Our Github issue: https://github.com/SnabbCo/snabbswitch/issues/39
I suppose I can send just the right ICMP echo packet to router to make it send me back an innoculating frame.
Correct me if I'm wrong (no, seriously) -- aren't both "packets of death" just poor handling of said malformed packets? Violations of their respective protocols? (TCP/SIP)
Personally, I am not at all surprised that this sort of thing exists. I'm sure there's lots more defects out there to be found. turning completeness is a cruel master.
They spent a long time "showing" us that a different version of the Linux kernel didn't exhibit the problem so it must be a Linux kernel bug. Turned out the different version just sent data differently so it didn't trigger the same bug with the same data. Other data would have triggered it.
I wouldn't be surprised if the majority of "bugs" they receive reports on turn out to not be bugs in their hardware. There's probably parallels with the reports of compiler bugs, most end up not being bugs in the compiler.
The unfortunate truth is that responsibility of proving it's the vendors bug falls on the customer.
I had to write a proof-of-concept "exploit" to show the problem was with their hardware, effectively troubleshooting most of the problem for them.
It's always someone else's testing procedures, someone's else's hardware... The thing is though, most of the time it is. Tech support at the lower levels especially are used to dealing with people who have bad configurations are are using the products incorrectly. The annoyance comes in when you as a customer narrow a problem down but can't get anyone on the phone who can help you at that level.
The “red telephone,” used to shut down the entire Internet comes to mind.
You discovered howto immunize friends and kill enemies in CyberWars.
Do governments have an Internet kill switch?
Yes, see Egypt & Syria they're good examples. We know China is doing Cyberwars, they are beyond Kill-Switches.
Techcrunch: http://techcrunch.com/2011/03/06/in-search-of-the-internet-k...
Wiki: http://en.wikipedia.org/wiki/Internet_kill_switch
We know Goverments deploy hardware that they can control when needed. Smartphones are the best examples for Goverment issued backdoors, next to some Intel Hardware (including NICs).