Hacker News Reader

Comment by roughly | Hacker News Reader

Someone once told me that nobody ever returned a server because the BMC sucked too much, and that says everything you need to know about BMCs.

Incidentally, at a previous gig, the software that handled issuing BMC commands had a BMC reboot as the first step in any script it ran.

View on HN

Not returned, but i've seen clients refuse to go with HP again because iLO sucked too much and come upgrade time went Dell. Unfortunately for them right around the time iLO got okay-ish and iDRAC got shitty.

This was me though I ended up moving to SuperMicro instead. Not fancy but their BMC seems to get the job done, and also doesn't cost a significant chunk of money extra for basic functionality. And then even more money to have IPMI be dedicated connection not shared. And HP's frigging BIOS wouldn't even work with their own HP rack console! But it was happy with an Apple mouse and keyboard. Argh! Making me irritated again remembering it. But yeah, we didn't return them but did sell them off to some other poor sucker and never bought an HP again. They may be great at large scale with the higher end central management software suite and support etc, but the BMC definitely was a dealbreaker at an SMB level. It's one of the few differentiating features vs a pile of other competitors who do the same basic thing, so I do think it's not entirely meaningless to get it badly wrong.

That pisses me off, when some vendors want to charge extra for KVM on stuff that already have hardware to do it and competitors do it...

Yes, and as essentially a "hidden flat tax" it burns a lot more for large quantities of lower end hardware vs single bigger iron. Looking back at the records, at the time adding their iLO/M.2 comm card (needed to have a dedicated iLO port on those servers) was $65 from Provantage and then the license to actually do basic stuff with it like a web console was and $227. So ~$290 all told extra. We've got a few higher end systems, $10-20k NAS or larger hypervisor systems, and at that level sure an extra $300 stings a bit but is against a lot of other stuff. At medium range, more like $3k-5k, obviously worse. But some use cases called for say 6x $500-600 systems instead of a single $3k-6k system, and for a few projects someone was trying to go as cheap as possible and grabbed some still pretty new (gen10) plain vanilla ones off of used/bankruptcy sales for like $300-400 because the price seemed so attractive. But then using iLO properly could nearly double the price, and at the lower end there is a big difference in the hardware you can get for $600-700 vs $300-400, and then multiply that per unit.

Meanwhile competitor systems all have dedicated ports out of box. SuperMicro does have a paid "full unlock" for their BMC, but the only thing it does is add bios updates and such. All the core normal management functionality is there by default. It also only costs $30.

Granted HPs had other irritations like really wonky proprietary fan control (heck, proprietary fan cables too) that would mysteriously fail to function with different flavors of the same OS, and couldn't be overridden from the BMC (then what's the point!?). Also they were slow with EPYC options when that's what a lot of us really wanted to be switching for, the performance and value propositions were getting really good vs Intel who were also jerks.

Like lots of big players the experience may get different if you're buying hundreds to thousands or more units and have a dedicated account manager who takes care of all this for you etc etc. But x86-based servers is a pretty damn competitive market and at some point one has to stop and ask why hours are being burned futzing with stuff when literally the entire basic point of getting "server class" hardware with remote management functionality is to save man hours by NOT having to futz. So yeah there's a little rant I didn't even know I still had in me years later :). HP you silly goofs.

Datacenter / Cloud Service hardware is a race to the bottom, we want simplicity and ability to work with off-the-shelf and widely available tools (Redfish (cURL), IPMI (freeipmi, openipmi, ipmitool)) so we can integrate these things into our hardware management platforms.

In this world, the legacy vendors like Dell and HP feel like they have to differentiate their hardware somehow, or they lose all the margin. So they'll charge you for all those "value-add" things like KVM or OOB Firmware Updates, because they can't make money on the machines themselves anymore.

The irony is all that extra garbage they add to their servers is exactly the opposite of what you want at high scale and really only serves the "enterprise" market that tends to deploy VMware and hand-manage servers and need point-and-click stuff since there's no incentive to write software to manage small environments like that.

> SuperMicro instead. Not fancy but their BMC seems to get the job done

Kind of depends which generation of servers. I had worked with a lot of x9 and x10 (xeon e-2600v1-4) which was alright, as long as you don't mind outdated java (well the newest x10 bmcs do html5 consoles too, IIRC); but I recently started renting an x8 server personally, and it's worse... My favorite is when serial over lan just stops responding when you go from console redirection to os opened serial port (and back)... real helpful for inputting disk encryption passphrases. Oh well, I'm renting this server because it's cheap, it's also 10+ years old, and it works enough.

Totally fair. Yeah should have specified I'm talking fairly new stuff, at least new enough it all has HTML5 consoles and mildly more polish. Not that there aren't problems, less a "bug" then a "cutting edge for newbies" it does near zilch certificate validation for example. So when the friendly new guy wasn't paying attention and generated a client instead of server certificate and uploaded it to a remote one it cheerfully accepted it and whoops! Operator error of course but the real point was that remote rectifying it proved surprisingly impossible despite still having admin serial/ssh access. Documentation is bad and didn't seem to like typical reset codes or tools.

I guess after lots of problems with expensive "high end" fancy stuff like HPE's and co I kind of felt resigned that all BMC/IPMI was kind of crap, and at least with SM's I felt less obviously squeezed. Like I said math will all be different for high end stuff and big herds. But neither Dell or HPE's struck me as "oh yeah that's worth an extra $300 on every single last server!"

I was just fiddling around with a SuperMicro X8 IPMI the other day. The X8 IPMI stuff is terrible, e.g. the warning that your Java installation is outdated on opening the website etc.

Turns out, you can actually install X9 IPMI firmware on X8 boards as the platform files are still shipped. Might be worth checking out, if this improves things for you. It did for me.

Check out https://github.com/devicenull/ipmi_firmware_tools for unpacking (and repacking) the SuperMicro firmware. The developer just merged my patches making it work with some of the X8 boards. As long as your board is listed in /etc/defaults of the IPMI tree you should be good.

Anecdote, but: I've seen a previous employer blackball a hardware vendor because of terrible BMC support.

I've personally been involved in decisions like that, as well. If you're throwing stuff into a datacenter and the BMC doesn't work right[1], the hardware is basically a brick. Vendors should be blackballed for their incompetence.

[1]: all things i have personal experience with:

- chassis bootdev pxe => doesn't do that, just reboots to normal OS

- chassis power off => doesn't do it (oh, here's an ipmi raw command you can use for this BMC version. NO!)

- dhcp server sends an option in an offer it doesn't understand, drops the offer, no IP at all.

- gets scanned by auditor-checkbox-as-a-service (qualys), locks up and sends the host CPU into 100% and locks up that up too.

- Not supporting IPv6 properly (if there's a place to start deploying IPv6 it's BMCs), i.e. uses SLAAC properly, but doesn't use the gateway from the RA so you actually can't use it from outside it's own segment - needs a firmware update to fix but uhh, we didn't dual-stack the BMC network because the whole point is to get that IP space back.

We had to write a test suite for vendors to run against their BMC and validate these things and you were disqualified if you failed.

Reply on news.ycombinator.com