Our BMCs are not great at keeping accurate time | Hacker News Reader

Our BMCs are not great at keeping accurate time

utcc.utoronto.ca

58 pointszdw32 comments

BMCs are horribly insecure and probably intentionally so. They should always be on a dedicated ethernet port and never shared (looking at you, Supermicro - no port should ever be able to accidentally become shared), and the network to which they're connected should be 100% disconnected from the Internet, which means you'd obviously need to be running your own NTP server.

If you weren't previously using NTP with your BMC, then that might suggest you haven't put enough time and attention in to your configurations.

If you're not already keeping your BMCs from talking to the Internet, then you really, really should, as soon as you can.

> If you weren't previously using NTP with your BMC, then that might suggest you haven't put enough time and attention in to your configurations.

> not all vendors include NTP as a free feature on all of their BMCs. Charging licensing costs for some features is a good way to insure that we don't use them, unless the licensing cost is trivial (which, spoiler, it hasn't been when we asked

If you bought servers that nickel-and-dime you on NTP that means you haven't put enough time and attention in to your purchasing.

Some BMCs have get their clock synced with OS which is connected to internet. This way BMC does not need outside connection.

Are there any resources that talk about the specific concerns of the bmc having outbound Internet access?

It is general advice across most server application deployments to limit internet egress (see for example NIST NIST SP 800-190, section 4.4.2 “Unbounded network access from containers”) . A machine that can talk out to the internet usually provides an attacker more avenues to compromise it and control/exfiltrate data from it post-compromise. A classic example is a SSRF bug, like how the recent log4j/log4shell bug used an attacked-instigated outbound LDAP call to a malicious internet host to achieve RCE.

You remember those Bloomberg articles about SuperMicro servers being bugged? The details they came up with for how the bugs were inserted were kind of out there, but the scenarios for what could be done line up with a compromised BMC.

If you assume something will get remote execution on your BMC, it's likely going to get full access to the BMC, because no updates. Then, what does your BMC have access to? DMA to the host, ability to monitor and trigger lots of exciting things, etc. It's pretty scary if you think about it, but IPMI replaces serial concentrators and remote access PDUs with zero rack space and not much security, so yay?

But if the BMC is compromised, then basically all the protections you put in place to seperate its connection from the host are useless anyway, since the BMC can easily masquerade as the host. I can see why it's a good idea to restrict access to the BMC itself, since it's a likely quite soft target for exploits, but once it's exploited, there's not really much benefit to restrictions which target the BMC specifically.

If you can restrict access to the BMC before it's comrpomised, you might have a chance at a secure environment... You definitely want it to always use its dedicated port, preferably with no factory way to use the host nics (although, once it gets compromised, dma says it can use the host nic if it's sneaky)

Sounds like you need a server that can run OpenBMC or similar. Or just run a regular Linux distro on the BMC, some of the vendor BMCs are running Linux under the hood.

As far as i can tell the openBMC support list is quite small :/

Most vendors are indeed running Linux. For example, HPE iLO 5 is capable of starting smart storage administrator from bios, and when doing so it just launches some flavor of centos, launches SSA server then starts an antediluvian build of Firefox pointing to it.

I wouldn't be surprised to learn that the chip serving iLO was Linux too.

It is.

The industry has really build around 2 vendors for chips, and two vendors for software solutions for those chips, AMI and Insyde. These systems are generally pretty locked dow , but there are was to get to the Linux core(sometimes serial headers).

All others except iLO is Linux.

I'd dearly love to do this. I'd even settle for keeping the (presumably ancient) kernel shipped by the manufacturer and just replacing the user space to ditch the hardware manufacturer's pile of hacked together crap and clunky web interface.

All we ever use in practice would be covered by dropbox/opensshd on a minimal rootfs with a shell where we can (presumably) toggle GPIOs through sysfs to power on/off/reset and interact with the serial console over a tty. This is almost certainly what the horrible manufacturer interfaces already build on top of.

But as far as I know, the damn things only accept signed images and aren't easily stripped down and replaced like this?

I expect these devices have plenty of vulnerabilities over the years and there have been Linux remote exploits, so probably you could hack them to replace the software or even just brick the device to avoid the vulnerabilities.

Also companies like Raptor Computing, Facebook and others are creating open BMCs people can use.

OpenBMC only supports one vendor's SoC.

OpenBMC supports SoCs from Aspeed and Nuvoton, and HP were/are working to add support for their GXP (iLO) SoC.

Given a random board with an aspeed 2500 on it, I assume that's not sufficient to allow the manufacturer firmware to be replaced with vanilla openbmc by a user though?

Or just enable NTP client in the BMC.

A more than a decade old BMCs based on Aspeed 20xx had the NTP baked in. Because you know, it's already running Linux there.

I don't remember on hw/os ancient iLOs/iDRACs were running on, but they had NTP client too.

I'd say everything newer than 2010 should already have NTP client.

The oem vendors have been adding weird surcharges for the bmcs to use features. For example hp used to charge(and I assume still does) for remote console. The author mention dell charged for using ntp, which was a little surprising. But, “enterprise” features like ntp and ldap are things vendors get excited to charge you for.

> adding weird surcharges for the bmcs to use features

While this is a true statement, I don't remember NTP client being on this list.

SuperMicro only wants a license for a online BIOS update, HP/HPE wants money for iKVM to a running OS[0] and some features making sense only in enterprise environments, like AD integrations and Dell's iDRAC... well I haven't used iDRAC without aN Ent. license for a decade, so I can't bet my $5 on the feature set.

Overall, most BMC implementations does have NTP client free of charge.

[0] You can have any time you need to install OS. You have a one or two minute limit in already running OS. Enough to for the troubleshooting.

In Dell iDrac 7/8, NTP is a licensed feature.

See table here: https://www.dell.com/support/manuals/en-us/idrac8-lifecycle-...

Thanks. It's dissapointing, there is a ton of things what make sense only in Ent. environment, but having a correct time is not Ent.-only.

Enabling network interaction using outdated probably insecure software running on the vendor BMC Linux distro sounds terrifying. I would much rather hack the BMC to run OpenBMC or a regular distro than enable NTP on an existing vendor BMC distro.

It seems to suffice to log the difference between BMC time and host time periodically. Actually correcting BMC time seems pointless.

The article mentions KVM over IP licensing. AFAIK only Intel and its OEMs used to do that.

Who else provides licenses for features?

Hewlett-Packard has never heard of a rent they didn't want to seek, so iLO definitely has bullshit extra fees for remote management after the firmware exits.

HP also unlike Dell requires a maintenance contract to get the latest ilo updates. The free ones are years behind, so if there are zero days for ilos then many price-sensitive users purchasing them on the secondary market are vulnerable.

Dell iDrac requires the enterprise license to get KVM-over-IP.

> Who else provides licenses for features?

Every enterprise grade hardware provider as far as I am aware.

Even with shared network BMC's, NTP will work fine. I'm not sure what BMC they are using that does not have NTP - it's been around all aspeed BMC's for a decade.

I'm not aware of anyone charging for it, either.

There are many things that BMCs are not great at. Add this to the list :)

Reply on news.ycombinator.com