In Which I Savagely Impugn the Honor of IPMI and its Friends | Hacker News Reader

In Which I Savagely Impugn the Honor of IPMI and its Friends

fish2.com

63 pointsdfc16 comments

Pfft, this guy only mentioned the obvious horrible security implications. Aside from that glaring flaw, there is also the small matter that IPMI software is all implemented by raging idiots who got fired from BIOS vendors, and then got fired again from the people who write mobile radio firmware, and then got fired again from the company that writes the software in your electric toothbrush. They never work in actual outages, cause more host outages than they solve, and are in no way a substitute for having humans on site. I had a rack full of garbage from Penguin Computing that would hang on reboot with the message that the BMC was not responding. Thanks, that's helpful. And dozens of horror stories from every vendor ranging from SuperMicro to HP and IBM.

I encourage you to guess whether Facebook, Google, and Amazon use IMPI to manage anything. Have a look at the OpenCompute designs for a hint.

http://opencompute.org/wp/wp-content/uploads/2012/05/OpenCom...

These designs?

Doh. Just goes to show, you can open up your process, but you can't make people have judgement and taste.

And/or that you should look before you speak.

The Open Compute 1.0 servers had reboot-on-LAN instead of BMCs, but then Facebook went back. I wonder why.

"this guy"? This guy is Dan Farmer[1], I realize that does not make him infallible but he is definitely not someone looking for his 15 minutes of fame in order to make a name for himself.

[1] http://en.wikipedia.org/wiki/Dan_Farmer

I read the comment not as a critique of the author, but rather as a further rant against the target of the author's ire.

After rereading it I think that's probably a fair interpretation. I guess I was thrown off from the fact that the author was the subject of the first sentence and therefore made it seem personal.

Very true. I searched long and hard to find Intel's vPro capable motherboards which support KVM. Lots of whitepapers for managers on the Intel site. Not much in the way of hard information about which motherboards work. My favorite was their suggestion that you download a tool and scan your network to see which ones work. Because that is easier than maintaining a list, I suppose. I can just buy one of each and work it out.

That it took so long to figure out which motherboards even supported vPro KVM should have been a clue. I think they must be embarrassed by the thing and and don't want anyone to actually see it.

For my efforts, I have VNC access to my motherboard that hangs when I try to connect to it from OS X, works a little bit, sometimes, for a while from Linux, then hangs until a power cycle.

For extra delight, the ethernet chipset that supports this useless heap has long standing bugs with the Linux kernel with no work around. It works fine, unless you want to push more than a few megabits per second through it. Then it hangs, disconnects, and reconnects.

Oh, it also took me two days to figure out how to even turn on the KVM without a Windows machine. Their tool won't run on Wine because its written in the new "portable" way, and for some arcane reason you can turn on lots of things from the BIOS, but not the KVM, you have to come in from an external machine with some misguided "one protocol to manage them all" system and an arcane text key.

Late last year I was building infrastructure with some higher security requirements and was quite shocked at the IPMI situation - concerned enough to email Redhat (who obviously support lots and lots of clients on hardware from lots and lots of vendors) via a personal contact in a global security team there, with my concerns and some evidence related to our particular hardware vendor.

After some time to ponder, they got back to me with "your concerns look legitimate". Discussions followed, but I am not aware of any specific advisories resulting from this.

I also understand that recently there was an IPMI-related presentation at Ruxcon Breakpoint in Melbourne, Australia: http://www.ruxconbreakpoint.com/speakers/#Igor%20Skochinsky ... no idea of the content, though.

Honestly, I think the first thing is for vendors to be a lot more honest: if someone has root on your host, they can most likely trivially obtain root on and backdoor the IPMI controller. Once they get root on the IPMI controller, they can do anything they want on the IPMI LAN. This reality differs greatly from the 'dedicated' and 'separate' words used in vendor marketing literature and documentation, and is no doubt a direct contributor to insufficiently paranoid systems architecture, resulting in real world vulnerabilities in some pretty important systems.

Why do I say pretty important? Well, outside of remote KVM, IPMI is frequently used for node fencing in high availability (HA) cluster scenarios, which means that this technology is likely to co-occur with some fairly paranoid / 24x7x365 systems (air traffic control, stock markets, etc.). Attackers with a foot-hold can thus use IPMI-based vulnerabilities to (permanently) compromise such higher-end systems.

What are we as implementers supposed to do? Here are some ideas. (1) Run a relatively paranoid, receive-only, anomaly-based NIDS on your IPMI segment to encourage 'after the fact' detection of bad behaviour. (2) Run IPMI on a dedicated VLAN. (3) Run your IPMI VLAN on dedicated hardware (4) Program your network infrastructure to ensure that regular IPMI nodes on your average server cannot communicate with any other peers on the link layer (ie. other servers) except for the expected source of management operations (5) Make sure the expected source of management operations isn't running an IPMI controller (or at least, one from the same vendor on the same subnet) (6) Don't use IPMI at all

Honestly, IPMI can be very useful in some cases, though it's probably a massively unspoken vulnerability in some high end systems. As always, don't trust any single component... security only comes in partly-effective layers ...

Are these IPMI things ever installed on desktops and laptops, or are they strictly for server hardware?

You may be able to find it on high-end workstations sold by server companies (read, a 4U server turned on its side, with added feet), but by and large it's a server-only thing. The reason being it's only really necessary to have the level of remote access offered by these systems if it's difficult to access the hardware, or you have a very large amount of the hardware (in most cases where it's used, both of these are true).

You can also buy discrete bmc pci cards from a few different vendors. I believe super micro sells them, but I've always loathed any environment that uses super micro servers as they tend to be low to mediocre quality compared to the "big 3" tier 1 server vendors.

On desktops and laptops Intel has AMT, which is somewhat similar.

Anecdotal, but the IPMI software I have to use regularly for access to a specific server is, without doubt, the worst server utility I've ever used.

I love having the BMC there, but you are really at the mercy of vendors to update their IPMI/BMC firmware. Guess how often that happens.

IPMI is so bad, I don't even use it for my servers when it is available! Better to just wait and call someone at the datacenter who will take care of reboots properly.

Reply on news.ycombinator.com