Over 20k servers have their iLO interfaces exposed to the internet
isc.sans.edu
isc.sans.edu
These companies don't care if there's no way to stop a remote control service (iLo, iDRAC, IPMI) from binding to a motherboard ethernet port. In the case of Supermicro, you can't configure via the built-in BIOS, there are no jumpers you can use, and you must have a full network setup and installed tools to configure a new machine. This makes deploying in the field much, much more problematic.
If the BIOS battery dies and settings set are lost, the motherboard defaults to joining whichever ethernet is active, with default credentials. It's incredibly stupid and insecure.
I will never buy Supermicro again, but for the hardware that was already bought, I mandated loopback plugs for all IPMI ports so the IPMI wouldn't switch to the system ethernet.
"A lot of devices and services we have seen during our research should never be connected to the public Internet at all. As a rule of thumb, if you believe that "nobody would connect that to the Internet, really nobody", there are at least 1000 people who did. Whenever you think "that shouldn't be on the Internet but will probably be found a few times" it's there a few hundred thousand times. Like half a million printers, or a Million Webcams, or devices that have root as a root password."
Um, like web is in the name. Othewise, good points
I once connected a server to a network and it wouldn't take the room's VGA KVM, so I started a dhcp server on my laptop to get onto the ipmi
63 leases were given out in a few seconds -- the lights-out (ilo, ipmi) that were on that network had never been configured, were just sat sending out dhcp requests and getting no response. The servers themselves were all statics, but nobody had bothered setting up the ilos. Most were of the age of a default password (ADMIN/ADMIN for supermicro for example)
Now this is on a private network with internet via a proxy, so fairly bad, but not catastrophic. It shows how easy it is to connect something to the internet accidently though.
Personally I put ilos on a dedicated vlan with an ACL blocking access to anywhere other than the management servers, it's not just incoming connections from the internet or from compromised machines behind your firewall, it's outgoing connections from the ilo too.
https://threatpost.com/plaintext-supermicro-ipmi-credentials...
Here's a search for some of them (requires a Shodan account):
Supermicros default to using the dedicated nic for IPMI, but they also default to using the piggyback if there is no link on the dedicated nic. :(
People on here love promoting dedicated servers over cloud VMs, but it’s so much easier for this sort of thing to go wrong with dedicated hosting.
a chip that sits on the southbridge of the server and has management capabilities (power operations and access to the underlying os being the 2 big ones).
But yes, some providers put the BMC on the internet because it's easier, a provider I used once did this and I was quite displeased as iDRAC's are quite weak and suffer under the weight of bot-spam. -- even if there were no security issues.
It’s nice to slide in a server, watch as the arps/dhcp requests go out and see the machine spring to life without human intervention.
Easier if there’s a known username/password.
It was printed on a slide out tag.
Some earlier generations had paper/cardboard tags tied to them I think but I cannot remember if passwords were there.
Or did Azure not have endless issues, similar in size and scope.
Well yeah, different tradeoffs; many people believe that dedicated has more advantages than disadvantages. That doesn't mean zero disadvantages.
Though what helps is that most servers have a dedicated iLO interface and you really have to choose to configure it on the regular ones along with normal traffic. So out of band is default.
So in this case it's only people who have deliberately configured this. I think this is why it's not hundreds of thousands.
I worked with a good number of SuperMicro servers that - to my surprise - the IPMI interface was by default set to "fail over to bridge to primary interface". That is to say if the IPMI port was disconnected, it would bridge onto your primary NIC with a second MAC address. The way to disable it was to download an obscure utility from SuperMicro's website, which only ran under Windows, and to pass an equally obscure hexadecimal command line flag.
Fortunately in my case, these ran on a network with no DHCP server, but I can't assume that's true in every case.
IPMI is a very different thing.
My best guess is that the vast majority of these 20k servers have their management interface deliberately exposed. Only takes 2000 people with 10 servers each to think this is a good idea.
See, e.g., https://www.armis.com/research/nat-slipstreaming-v20/ and https://samy.pl/slipstream/
Some of them could also be intentional, in order to provide management capabilities.
At least iLO has no default password (each server comes with a label with the password)
http://fish2.com/ipmi/itrain.pdf
all servers in a datacenter have this management interface (iLO is just one type).
if the management network these sit on is poorly secured (like here) your servers are literally powned.
On one side, an ethernet port, that you plug into your management plane. On the other side, a serial port, to access the host's console, a device-side SATA port, so it can present a virtual boot disk to the host, and a wire to the power supply, to turn the host on and off. Maybe also some read-only interface to motherboard-level monitoring like SMART and so on; maybe you get that by cooperation with the host's firmware, over the serial or SATA interface. That would let you bring a machine up and configure it, reboot it remotely, force it to boot from a recovery disk, etc. But a hacked BMC couldn't be used to subtly interfere with the host.
Also, does anyone know what Oxide are doing about BMC?
Actually, I can imagine a fair few are still in use as small enterprise servers, with iLO being visible for remote admin, which is a shame, as that suggests they are not behind a tunnel, so those addresses probably indicate larger problems than a visible iLO.
Walking around a DC, no they have not been "chucked out of datacenters"
1. Disable public ipv6 on the iLO interface
2. change all the passwords
3. Set-up tls on the web interface and force http to https redirection.
Ironically, point 3 alone is really most of the protection: the web interface is so slow in https it's basically unusable.
Either their IT people have been begging to fix it and management won't give them the resources, or they don't have IT people who even see it as a problem. Either way, a couple special agents having a sit-down with the CEO might change some course real fast.
Discuss.
I don't think the fbi would be any good at this scenario described.
1. You don't want the device even considering requests from anyone but the sole person(s) responsible for accessing the mgmt interface. Someone might not be able to get in, but they could enumerate your hardware characteristics, or perform a denial of service.
2. Don't confuse SSH with secure. You don't know what version/brand/make/model of SSH is running on the embedded device. It might be ancient. It might not even be OpenSSH.
3. If an exploit becomes known for your management interfaces on your machines, you are screwed. First because someone might have already exploited you. Second because now you need to patch your boxes, and that might require a hard reboot of the entire machine. If a patch even exists.
Really, it isn't worth the trouble. There are too many risks.
The security group at HPE called everybody that they could identify at that time to help them fix this.