Comment by xoa | Hacker News Reader

Comment by xoa | Hacker News Reader

Parent Full thread

>but I'd guess the vast majority are never even touched, much less thought about, by the people who manage those systems.

Really? I'm certainly no big data center sheep dog, but even for the tiny handful of systems I manage IPMI/BMC always seemed to come with big, flashing red lights just from their mere description. They're enormously powerful and valuable tools, but seems pretty obvious that can work both ways. Beyond their own credentials (and it's a real shame so few support smartcards/hardware tokens for login), having them on their own isolated VLAN seemed about the bare minimum. If budget allows/there are enough systems flat out having their own 100% independent physical infrastructure isn't such a stretch either, not like a simple switch and console system or if necessary a decent minimal bastion is that big a price tag these days. In some cases may make the most sense to just use them to setup then disconnect them.

You may well be right but still, yeesh. It's remote super root access, should make any sysadmin treat it with healthy fear from just a description. Although what did give me the willies was how it seems common to have IPMI be hybrid with LAN1 by default rather then it's own physically independent port. I switched from HPE to SuperMicro kit because at our scale and application it made sense for a lot of reasons anyway, but that HPE wants a lot of extra money for a dedicated port and full fat features was a real factor.

Unfortunately I have to concur with GP, my experience in penetration testing is that many sysadmins seem oblivious to the existence of ILO systems, or don't believe they can cause any harm. Alternatively, sometimes one simply slips through the cracks, and all it takes is one system with cached credentials to wreck havoc to a network.

Ironically, the more expensive ILO systems can be easier to exploit. Just throw a Linux live rescue image onto virtual media, reboot the machine, and you can grab everything sensitive fairly quickly.

Well (and to sibling comment), TIL, thank you :(. That's scary though, that anyone could even use the BMC and not at least pause a bit on "wait a sec, what could someone else do with this", same as hypervisor access.

Though maybe some of the blame really should be on the vendors themselves here. At least credentials are randomized now, but they could be much more aggressive about making it something that has to be actively turned on, always has it's own physically isolated port, and requires some level of secure setup (won't communicate untagged maybe?). Or these days making hardware token/smartcard usage required perhaps. There's clearly tradeoffs to be made in terms of security vs convenience/recovery from bad setups, and the balance has fallen fairly decisively in favor of the latter. Kind of depressing though the industry constantly has to do the same thing of ignoring security right up until it gets really, really bad.

In their view, they place a great deal of importance on security; at least for marketing purposes. Sustaining a secure system after it's sold and work has begun on marketing the next new supersecure thing... that's a different story

That’s been my experience as well

They were either not plugged in, or definitely not on their own vlan in any of the datacenters (dozens of them) I worked in from 2000-2007. Many admins have no clue they exist or what they do.

On most boards where the ILOM / BMC has a dedicated LAN port it usually also has access to at least one of the main ethernet ports on the MAC level to support hooking machines up with just one cable without loosing ILOM.

Just isolating on a separate network isn’t enough — if an attacker compromises a single machine, then they get access to that network.

> Just isolating on a separate network isn’t enough — if an attacker compromises a single machine, then they get access to that network.

Eh, no. You'd isolate it on a VLAN where only people who need access to ILO have access to. Now, lets zoom out for a moment, does the sales dept. need access to ILO? No. Does the CEO need access to ILO? No. So machines on these VLANs do not have access to that VLAN. And, if the machine of a sysadmin is compromised, you're into deep shit as it is.

You’re missing the point.

Suppose you have 500 servers. You carefully put each of the 500 iLOs on the special iLO VLAN, you can enable 802.1x or MACSET or magical locked Ethernet jacks or whatever and make absolutely certain that only perfectly trustworthy IT admins using perfectly trustworthy computers can touch that network.

And you still lose! Because an attacker can compromise an unimportant sales computer, escalate to root (or SYSTEM), and compromise that computer’s iLO via the internal pretend-PCI transport as discussed in the OP. And now the attacker is on the supposedly secure iLO VLAN.

Replace iLO with any other BMC or BMC-like solution (AMT, for example), and the scope of this issue should be apparent.

To mitigate this, either proper hardware rooted security for BMCs is needed (giving a strong zero-trust model) or a very carefully configured network that isolates all hosts from each other. Or, preferably, both.

Who said the unimportant sales computer (desktop) uses ILO? You said you have 500 ILO servers on a special VLAN (or even galvanic separated).

It could be an unimportant sales server. [0] Not everyone has a fleet of essentially interchangeable Kubernetes servers, and a credible threat model should allow compromise of a single server without permanently infecting the entire fleet with an APT.

[0] Windows Server, Citrix, etc are real. Just because it has an old fashioned GUI doesn’t mean it doesn’t have rack ears and a management port.

Sure, that makes sense. Thanks.

Reply on news.ycombinator.com