Hacker News Reader

Comment by LennyWhiteJr | Hacker News Reader

> The attacker gained access to the server — which had been active for about a month — by exploiting an insecure remote management system left by the datacenter provider, which NordVPN said it was unaware that such a system existed.

This screams for clarification and I'd love for someone more knowledgeable in the area to elaborate on it. Is this common practice for data-center providers? Do I now not only have to worry about my own infrastructure security but also worry that my IaaS provider hasn't installed some backdoor to my servers?

View on HN

I work for a web hosting company in the US and at least in our case, it's quite common for remote management to be enabled on pretty much all of our dedicated hardware. However, because of the inherent dangers in opening this up to the public internet, unless explicitly requested by the customer (or Managed Colocation), the NIC used for Dell iDRAC or HP iLO is on an isolated network unique to the physical datacenter. Remote access for our techs is managed through a secured bridge that requires all sorts of security hoops on our company intranet, and remote access for general internet traffic is not available due to the firewall restrictions. While it's plausible for remote access to be gained this way, it is extremely unlikely and would require several exploits at different points along the path.

I cannot speak for the industry as a whole, but remote management systems like this are bound to be common; any large enough physical datacenter is going to need a more efficient way to access a misbehaving system than sending a tech physically running to the box to plug in a keyboard and mouse. It should be extremely uncommon to have these management interfaces open to the public though, and I'll bet that's what NordVPN is surprised by. Generally these systems should be private and isolated due to the power that an attacker can wield through them.

IPMI does not have to be open to the internet to be open to a wide audience. Many of these out of band management interfaces are hosted on an internal network, but not isolated by customer.

Cheap datacenters are favored by VPN providers for their unlimited bandwidth and lax abuse policies.

Many of them allow access to IPMI only over a VPN, but do not isolate each customer’s IPMI to a customer VLAN. I personally know at least three large budget datacenters which allow all customers access to each others’ “private” IPMI IP addresses.

Which cheap data centers are you referring to? Curious as someone unfamiliar w/ the space.

Sorry hacker, not today!

Haha! If I was up to no good I wouldn't be using my real name in my handle to ask such questions :)

Generally speaking, there are four (4) tiers of "public" data centers are on the market, ranging from essentially a big room with some alright AC and a line out, to huge, highly secure (cameras, fingerprint readers, SSAE certifications, etc.) buildings with redundant power and HVAC systems.

The higher end ones are usually newer-ish, but there are lot of older "computer rooms" that offer acceptable-level benefits for a reasonable price. Lots of legacy customers and ISPs in these rooms, for the record. You get what you pay for but a lot of older DC spaces are just fine for most users; everyone thinks they need 99.999 but most don't.

There are also re-sellers and managed services companies who take out a footprint in data centers and then lease space in their cabs, sell bandwidth, IPs, etc. Using a series of resellers you can often get around restrictions as to what you're doing -- small fry MSPs don't ask a lot of questions -- but still get a data center footprint. These are sometimes one-man shows, and their quality and professionalism are often sub-par, which is how you end up with default iDRAC creds and the like.

Check out Data Center Maps or WebHostingTalk for some examples.

Source: data center ops manager for several different companies.

My social network for dog walkers definitely needs 5 9's. Thank you for the insight.

I have decided to contact them directly rather than publicize.

Maybe it wasn't open to the public Internet, but the VPN exit is inside the datacenter and connects out to the public Internet. Is it feasible that NordVPN provided their customers with a secure tunnel into their own datacenter's management software?

Sounds like an iDRAC exploit (assuming Dell servers).

But, yes, remote management is pretty common in datacenters. The fact that NordVPN wasn't aware of them just shows incompetence.

Depends on what kind. In case of idrac, yes; but it's weird that it was insecure by default in the first place. Usually credentials are configured and provided to the customer. Makes me think there might have been some other interface. Clarification is definitely needed.

It could have been something like https://www.zdnet.com/article/vulnerabilities-found-in-the-r...

There were many IPMI/iDRAC/etc. exploits published in the past few years. Throw a dart at a list of them, and you'll probably find one that was unpatched in most systems as of March 2018.

How the hell do you pwn a server with iDRAC?

In certain configurations, iDRAC gives you an rdp connection. If idrac is left at default, windows admin login not being changed isn't too much of a stretch.

If you can reboot it without anyone noticing? Really, really easily: iDRAC gives you access to the local console, like a remote KVM. Reboot into single user mode, change a password, done.

Oh, IPMI and friends are a total mess. Some implementations allow one to take control of a running server remotely especially if they use a shared ethernet for management ( popular in supermicros ). I once had our security geek demonstrate it by taking over the running server, rebooting it using network emulated USB stick, adding a file into /etc and rebooting the server again.

In secure environments one pulls IPMI module from the server or only uses the modules that have their own dedicated NICs that have to be wired to their own management network.

The first time I booted a server using a virtual CD-ROM (iso on my laptop shows up as a hardware CD-ROM on the server) over IPMI I was simultaneously relieved (because I could fix the machine remotely) and absolutely totally horrified.

iDRAC is a full onboard whitehat rootkit manufactured and supported by Dell. It runs independently of any OS and has control over the system. It is intended to be a substitute for physical access.

iDrac have a default password : https://danblee.com/dell-idrac-default-username-and-password...

There also quite a number of sysadmins that connect idrac's to the "regular" network, instead of the sysadmin VLAN ...

User root, password calvin. That's the default. And, if I had a dime for every time I've seen one of these in a data center, I'd be a rich man. I have literally begged sys admins to change the default password, but they say, "Why... we're behind a firewall using RFC 1918 addresses. No one can get to these." The rest, as they say, is history.

> we're behind a firewall

This is the dumbest thing I've ever seen... unless your firewall is between your host versus every other host and there's no multi-tenancy, this will suck.

In well maintained networks the management interface (IDRAC, etc.) for each server is placed on a separate VLAN which the servers cannot access. This isn't to say that cheap providers actually do this, or that the VLAN can't be accessed by a compromised technician's workstation/laptop.

So it's a fail-open design, given the rarity of well maintained networks, and the lability and inobservability of said state.

Never trust the network.

Yes, this kind of firewall is always supposed to be between the management hosts and everything else. Only the sysadmins at the data center a very limited set of applications is supposed to be able to access it. The very real risk is misconfiguration.

HPE iLO also had critical vulnerability: CVE-2017-12542.

I’d guess that the DC got owned, no need for iDRAC exploits when lazy VPN company staff never changed the pws.

Pointing fingers without having the details at hand is not competent either.

It doesn't make much sense to me, even with iDRAC/some other console access you don't really have access to OS unless you reboot & go to single user mode etc at which point they should be noticing their servers rebooting etc. would love more info

It opens an exploit chain, in a normal circumstance you are correct. In a malicious circumstance, it is always feasible irrespective of the likelihood.

Just set up your code as a boot-once config and wait for the owner to reboot their machine. Make your code end by booting the installed OS (or even by just rebooting again, most people will just curse about the damn slow server boot process).

You can't do that as you don't have any access until it's being rebooted. It's basically like you're standing in front of the machine so there's not really much you can do when you're just looking at a login prompt, you have to be able to stop grub from just booting with the default options and instead boot up using init=/bin/bash or maybe if the server supports iPXE you can just chain load some payload off the internet.

You can manipulate boot settings using BMC commands. No need to mess with Grub or the running system. Instead, tell the system to boot up from an emulated USB drive (image can be attached from some remote server, often including your web browser).

Now wait for the machine to get rebooted (or do it yourself using the BMC, e.g. 'racadm serveraction powercycle' for Dell/iDRAC machines).

Even SecureBoot won't help as you can just turn it off using the BMC.

See here for a bunch of examples for Dell machines using the BMC's HTTP API:

https://github.com/dell/iDRAC-Redfish-Scripting/tree/master/...

Why would they notice their server rebooting? And why would they not just assume it was a glitch or power failure?

When someone gets notified by their monitoring system that a server was unavailable (because it rebooted) they might investigate and see that the IPMI logs don't mention power loss

Power failure would require both of the power feeds in the DC failing simultaneously and would be easily verified by contacting the DC and asking if they had any power outages reported at the time. Of course there are cheapskates who don't go for redundant power supplies so it's possible but would be indicated in the IPMI logs

Servers can reboot for any reason. There are tons of kernel issues, especially since Meltdown & Spectre, that cause machine reboots in Linux especially on high traffic machines.

I've worked in production environments with thousands of machines and random reboots are a completely normal event for some workloads. Combination of hardware issues & kernel issues with hundreds of thousands of lines of code makes it inevitable. I would be surprised if NordVPN even noticed and their architecture wasn't designed to automatically start everything at boot.

You can't be perfect at scale - you just need to design your work loads to be redundant and fault tolerant.

It seems NordVPN fucked up themselves are are now trying to avoid responsibility: https://www.theregister.co.uk/2019/10/21/nordvpn_security_is...

""All servers we provide have the iLO or iDRAC remote access tool, and as a matter of fact this remote access tool has security problems from time to time, as almost all software in the world. We patched this tool as new firmware was released from HP or Dell.

"We have many clients, and some large VPN service providers among them, who take care of their security very strongly. They pay more attention to this than NordVPN, and ask us to put iLO or iDRAC remote-access tool inside private networks or shut down access to this tool until they need it. We bring [iLO or iDRAC] ports up when we get requests from clients, and shut them down when they are done using this tools. NordVPN seems it did not pay more attention to security by themselves, and somehow try to put this on our shoulders.""

Of course you have to worry about all of that! When you don't self host you have to assume whoever you are renting from hirers the lowest paid employees they can get to manage infrastructure for you. That's how they get profitable. You are not outsourcing expertise.

Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers.

Even a lot of VM/cloud systems have some kind of virtual management console (Linode has their LISH system that lets you SSH in to console and Vultr/Digital Ocean have similar web based consoles .. AWS surprisingly doesn't. You can get console output but can't send VMs any console input).

Not only should have NordVPN been aware of this hardware KVM, they should have secured it and had version checks on its firmware as an essential part of their security. I could see this oversight with other companies, but not with one whose primary business claims to be security.

> Yes, network KVMs are expected of any co-location center. You want to be able to access the console and the power switches of any real physical server without having to send someone out to the center, and is a common feature of most high end data centers.

Power on/off should be done via APIs that issue commands to a PDU, like Atlantic.net started doing in the early 200s.

And there's nearly zero reason to access "console" - configure your server to always but off PXE and fall through to disk if that intercept is not needed.

.. AWS surprisingly doesn't.

Why is this surprising? AWS seem to know what they're doing in general, and this is obviously the right policy in this particular area.

How is it “obviously” the right policy? If implemented, console input would presumably be part of the AWS API, guarded with IAM permissions like everything else. If you have full IAM permissions, you can already take over any instance by temporarily attaching its disk to a different instance and modifying the data from there. (That requires rebooting, but so would takeover via console input.) Indeed, I’ve had to do exactly that on multiple occasions to fix broken config files on my personal instance; it would have been much more convenient to have console input.

This "personal instance" model doesn't really fit AWS: if the instance is borked then fail over to another one that isn't. No need for console input.

FreeBSD has an interactive kernel debugger that you can use with a serial console. Super useful to track down some things, not usable on AWS -- you'd need to (somehow) do a core dump and hope you can figure it out from there.

Amazon has security critical functionality on an unauthenticated http endpoint on a link local address. That's pretty damn dumb in my book.

Support for your assertion: https://rhinosecuritylabs.com/cloud-security/aws-security-vu...

That's a great link, but they're looking at "misconfigurations". How is that relevant here?

> Since the metadata service doesn’t require any particular parameters, fetching the URL http://169.254.169.254/latest/meta-data/iam/security-credent... will return the AccessKeyID, SecretAccessKey, and Token you need to authenticate into the account.

They talk about the AWS “Metadata Service” Attack Surface and how juicy a target it is. I was just providing support for that opinion.

Yes, software that runs on the instance can learn instance metadata. No, that is not a problem. Running e.g. user-supplied scripts on the instance would be "pretty damn dumb", but no one is that dumb. Any widely distributed software that did something shady with instance metadata would get busted PDQ. Just like any widely distributed software that did something shady with e.g. root credentials, which is about the same threat scenario.

It's crappy design which bypasses important security mechanisms of the OS (lower privileged users) by allowing every application with network access to access such critical functionality. One sane approach would be passing this information to the OS through the hypervisor which then exposes it as a properly ACLed file system.

This is like an author of a website vulnerable to CSRF (because it relies on IP for auth) blaming browsers for allowing cross site requests instead of require proper authentication. Except that Amazon is powerful enough to get away with pushing all the effort onto developers and admins.

Yes, in some situations it will take effort to not allow network access to untrusted software and/or users. For those situations, EC2 is not a good fit.

You can use iptables to limit metadata access to certain users but that takes effort so no-one does it.

I guess a machine-local service that takes ownership of the metadata service and implements additional restrictions (such as limiting access keys to privileged users) might be doable.

If they're going with a bottom-dollar host, it's possible that the out-of-band server management tools were exposed. It's less likely to be a software backdoor, and more likely to be Supermicro IPMI or other baseboard management controller.

I know that public cloud providers like Rackspace and Azure insert their own accounts and services into cloud servers and VMs mostly under the guise of being able to support said servers and monitor them and their health.

True data centers where you own the hardware shouldn't... they give you an ethernet cord and everything is on you.

Very few people go to "true data centers". Those are very expensive because you are buying power, space, cooling and cross connects. Racking machines, replacing hard drives, building a PXE-boot infrastructure, building a remote access infrastructure that bypasses the customer facing network is expensive and time consuming.

>> Is this common practice for data-center providers?

Absolutely. We had similar situation with one of the DC vendors.

So what can you do about it?

Use AWS/GCP/Azure

doesn't that shift the vulnerability/risk to the configuration of the firewall, security policy, and machine configuration?

Those cloud computing vendors know better than give open access to the word to your infra by default.

Full Disk Encryption is an option, but that’ll entail needing the use the IPMI console to enter the password at every reboot, essentially turning it into a manual operation.

If you have physical access twice, full disk encryption usually doesn't help really at all. IPMI seems about as powerful as physical access.

If you just ignore random reboots, sure. But why would you?

I don't think that'd be necessary, couldn't you make changes and then wait for natural reboots?

You couldn't, not via console access anyway.

However, if the attacker gains RCE many IPMI implementations theoretically allow for DMA, but this is a significantly more complex attack to mount in practice with no public PoCs available.

One can do LUKS with access via dropbear in initramfs.

If you're lucky during network install, you'll never need out of bound access.

But rebooting is still a manual process.

- Bring your own network and limit LOM/IPMI to a private network accessible by VPN only.

- Or, request a private network with no connectivity and a VPN device connected on that private network.

- Or, hardware disable LOM.

There's no reason you shouldn't request an ASA from your datacenter provider in this day and age.

On Supermicro hardware and maybe others, IPMI has a very dangerous default setting: if you're not connecting the dedicated IPMI port to a network (typically some closed network dedicated to management), it will use the first ethernet NIC on the motherboard (sharing it with the host), possibly making it accessible through the internet (with default, insecure credentials adding insult to injury) or at least neighbouring machines.

Leaving default creds on IPMI/iLO is not uncommon,some providers don't allow internet access,you have to login on the web console and use a java applet to access it iirc. I can't imagine a well reputed provider exposing ipmi to the internet but the nature of their business means they have to diversify server and network providers.

Not sure if it is still the case but a while ago it was standard on OVH servers for them to put some public keys in their root-like's user authorized_keys. I think they used that to perform tasks requested from the web management system.

They mean IPMI. All servers have IPMI and there’s remote root exploits against many versions of them.

> All servers have IPMI

Most (not all) servers have out-of-band management; of which IPMI is just one of many such solutions.

It's also worth noting that the hack could have been against in-band management if the Nord used an OS image provided by the DC hosts. However OOB feels more likely given their description (as vague as it was).

I'm confused why a factually accurate post was down voted. anyone care to enlighten me?

Don't worry about it. It could have been a mistake or a bot randomly voting. Either way complaining about down votes is against community guidelines.

Typically data centers have compliance requirements like SSAE 16 specifies controls around physical access. Most any major retail data center would have that certification and others.

One presumes that because of NordVPN's business, they're colocating a server or two in many very many "POPs", presumably not all of them have tight controls on physical access. Its likely that there are none available in many areas where they seek to maintain a point of presence.

Yes. Dedicated servers generally have IPMI / ILO / IDRAC / whatever. It's the only way to scale out management of hosts and provisioning.

To the person / people down voting, can you share your ideas for how you can go about providing at-scale management of large estates of physical machines? I've never seen anything else that's both as practical and as affordable as IPMI / IDRAC / ILO systems that ship with servers, that doesn't introduce weird new failure conditions that can impact significantly more than a single host.

Reply on news.ycombinator.com