DIY out of band management: remote console server | Hacker News Reader

DIY out of band management: remote console server

michael.stapelberg.ch

57 pointssecure17 comments

My solution was buying an old Avocent serial server. It's basically a giant raspberry Pi running a MIPS CPU. I should blog about this.

In the USA this is the best low-cost OOB Cell/LTE service I know of.

https://www.amazon.com/IoTDataWorks-Unlimited-12-Month-T-Mob...

It's off T-Mobile's network. Latency sucks unless you are close to the IP gateway they use (Chicago I think) but when shit's down and needs to get fixed it works.

I bought an old HP server for $150. It came with remote management built-in. Coupled with a cheap FQD and you can access it from anywhere.

The problem is that many others are also able to access your server - running an old version of iLO - from anywhere [1]. While I have iLO enabled on my DL380G7 I do limit access to it to a VPN so as to avoid these problems.

[1] https://www.servethehome.com/tens-of-thousands-of-outdated-h...

The idea that anyone would make these accessible from the Internet is preposterous, and the fact that so many are exposes an underlying problem in IT. Maybe it’s lack of knowledge, low skilled people, or the intense focus on cloud has eroded basic IT skills to such a point that things like this could be putting everyone at risk.

This is not an erosion of skills. Since I started in IT around 2008, it has been rare for most people to care about security. Even now, it's hard to get any traction. "We're not a target." I've heard it in more or less words, so many times. I'm burned out on fighting this fight.

The senior staff at that time were often the worst. They didn't want anything to make their job harder, and they didn't see a threat. So if you pushed security hard, you were the enemy.

It's a little better now, but not much.

You're not alone. I've had the same experience since pretty much back when it became clear that security would become an issue "down the road". I've been repeatedly ridiculed as "paranoid" over the years for my stance on security being an important factor in networked or multi-user systems, and it left me feeling pretty "burned out" as well.

a lot of times this can happen unintentionally— these systems can coopt the primary ethernet port when the dedicated one is unplugged and many default to this behavior

Yes but they should have a different IP to the one that is forwarded to because shared devices will have differing MAC addresses and default setup.

For example a lower end Dell without a separate iDRAC port will share with the first NIC with a IP that is either DHCP assigned or a built in default - both of which are unlikely to be forwarded by accident.

Older Dells had a standard root password of "calvin" which was an appalling idea. Newer ones do as HPE and have a default autogenerated monstrosity printed on a pull out slide. Also you can set jumpers on the mainboard to reset it. Look at the manual or the inside of the cover.

If you deal with ordering lots of Dell servers you can have them customize the BMC security settings(nearly any setting actually). I know of many people who have them either turn the legacy root password back on or use a password they tell Dell.

I insist on a certain boot order from the factory and have the initial pxe boot of the server lock down the idrac. It isn't perfect but it beats having someone pull and scan all those tags.

Here's a recent discussion about these beasts: https://securityledger.com/2014/06/ipmis-inconvenient-truth-...

IPMI is one of the foundations of what iLO/iDRAC n that do for a living.

They are really useful - I've used them countless times to do updates and so on from afar. It's a bloody long drive from say South Somerset to say Hull in Yorkshire to pick up the pieces from breaking a boot loader. Instead I can mount a boot .ISO locally and after a while a remote box is running an installer or repair image.

However, ideally put them on their own VLAN to protect your stuff from them and do only allow access from the outside via a VPN. Never port forward to one directly or IPv6 allow direct inward access.

I never said they weren’t useful; in fact they are extremely useful. But they should never be exposed to the Internet, just like any other administrative device.

I've been doing IT for 25 years but FQD has passed me by! What is it and please tell me it involves a VPN somewhere along the line.

Fully Qualified Domain I'd say, e.g. ´hack-me-please.example.org' or something along those lines. Usually the term FQDN (FQD Name) is used.

Great job and has a VPN built in. All the components used lend themselves to monitoring too - Linux with monitoring-plugins, MQTT (doddle to watch from afar) and so on.

This also lends itself to remote monitoring and control of quite a few really big and expensive things that have serial ports.

I like the look of this ... pulls wires out of a discarded Pi experiment ...

Not a bad idea to have a BMC running software you can trust. We’ve recently made something similar from spare BeagleBone Blacks we had a few dozen of. Except it runs FreeBSD and also serves installation media (from the host point of view it looks like an USB flash drive).

You can build a wifi smart plug with an ESP8266 for 2 euro from aliexpress. A GSM module costs 2 euro.

A serial console over SSH with an old raspberry 1.

And the whole ensemble would take much less space.

Reply on news.ycombinator.com