Vulnerable industrial controls directly connected to Internet
arstechnica.com
arstechnica.com
In a company I worked at, there were PLCs controlling sensitive processes, including heaters for welding plastic, connected to the company network via Modbus-over-TCP. A visitor with a guest access, nmap and pymodbus could read AND SET temperatures of any of the devices. A black hat could easily have started fires or leaked acid in our production floor.
If you want to beef up the network security to the PLC, be my guest, but don't disparage ladder logic. It's the right tool for what it's used for.
The difficulty arises when trying to implement something in ladder that is too complex for simple Boolean operations, and would probably be much clearer if written in a few dozen lines of Structured Text (assuming an IEC 61131-3 PLC). Unfortunately, company policies (especially in North America) mean that engineers are often forced to use ladder, resulting in a ladder logic program which is inscrutable for both the electrician and the engineer.
What would be sensible is using the right tool for the job: ladder for simple Boolean operations, and ST for more complicated stuff. Tech schools should also teach electricians and technologists the basics of how to read structured text - they do this in Europe, but I don't think it's common in North America yet.
Generally I prefer function block, especially in Schneider's Unity, since I find it is easy to watch a process operate the way function block diagrams are animated. The function block also promotes code re-use by creating your own blocks.
I use structured text only when the problem can't be well represented in function block, or more like I haven't figured out how to well represent it in function block yet. It is much harder to observe the operation of ST as the tag values aren't animated. Of course you can't use breakpoints on a process without stopping it, which usually you don't want to.
Also the inherently binary and visual nature of ladder (and HMIs) makes version control impossible or expensive. Many PLC workflows I have seen are basically just a directory of files that you hope someone has been keeping up to date. Maybe there are previous versions. You will also have a hard time getting a code change history. Orchestration and configuration are also pretty primitive.
Mostly I have dealt with Rockwell, maybe Siemens is different.
Edit: These already exist. [1] ;-( Oh well. Just need to have facilities start using them throughout the site.
[1] https://www.lifewire.com/best-vpn-enabling-devices-4140254
[0] https://www.tofinosecurity.com/products/Tofino-Modbus-TCP-En...
You are implying that C++ code is generally safer than C code, even in the hands of developers who introduce buffer overruns in the first place.
This is so wrong I don't even know where to start. In C++, I doubt they will ever get it out of door. Which would solve the security problem, incidentally.
Of course if you're going to leave C, why not use real time Java or some other safer language.... C++ doesn't seem like a very ambitious sort of upgrade.
Also, it is relatively straight-forward to compile C code with a C++ compiler and adopt C++ constructs while throwing out all the old insecure C constructs. Doing this would vastly improve security and is achievable in a short period of time with little or no performance loss (it will probably run even faster).
Yet, all I read about on web forums such as HN are unattainable suggestions such as re-write X in the newest, non-standard, corporate backed/controlled language that isn't nearly as tested nor as performant as ISO Standard C++. Maybe it's just me, but I find these suggestions to be totally out of touch with reality and really uniformed about the vast differences between C and C++.
I may be remembering this incorrectly, but vec.at(x) is safe (i.e. does bounds checking), but vec[x] is not safe.
I'm going to try some test code here and check, but I'm pretty sure vec[x] will happily overflow.
Edit: https://gist.github.com/tonyarkles/e702e4d5b530a9b7cd3fc9837...
Definitely overflows and segfaults while unwinding the stack (presumably while deallocating the stack-allocated vector)
What your example shows is an attempt to access a vector element that does not exist (std::out of range). And, that is undefined behavior and is documented.
Also, that code is 95% C (not C++). Even the includes and prints are C. Pure, idiomatic C++ would use iostream rather than stdio.h, and a vector iterator to access the elements rather than looping over the vector (in C like fashion) using an integer (which is unrelated to the vector) in an effort to access elements via index.
His example shows the precise opposite: the vector didn't grow automatically. Instead, memory outside the vector bounds was accessed, i.e., the vector overflowed.
Yes, idiomatic modern C++ makes that less likely, by reducing the use of explicit indexes. Even then, you still can overflow a vector with innocent-looking code.
In C, buffer overruns are undefined behavior and are documented. Back to square 1.
g++ -Wall -pedantic vec.cpp
Worser still, even the people producing the newer tools to allow for OO, do not really test these tools, so you develop in a environment where you are basically a beta tester on a budget (TC3 - looking at you).
Nobody dares to move, to not break precious backwards compatability and devalue the experience of "experts" who for several generations simply did fall into every trap state-chart based design can provide. Means, every project runs into complexity walls at the end, there are no automatic proofers to check the state charts, there are no unit-tests (If i here the poor excuse -"But there is no hardware to test against" one more time).
In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you.
Half of the projects in this area are not ond budget and not on time. There is actually not even planning for that- just a factor, which usually is bigger then 2.0.
Im so sick of the excuses. You cant compare conventional coding and PLC programming. Yeah, you cant, one moved forward and the other got stuck with the lowest common denominator - due to lots of non-software guys trying to get a food in. All these fb lego-players, with no clue what they do...
</Rant>
>In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you.
They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place. For them it's just this nasty thing you need to throw a lot of money at until it everything works. The mindset would need to shift to software being the main space in which problems are solved, because in 2018 it is. I guess this also turned into a </rant>
This has largely changed in the automotive industry over the last few years. We still aren't quite there yet on getting all managers to spend on security preemptively, but at least we now have had the big honchos hand down the gospel that software is important.
You can get that by using Python or similar languages; you can get that by programming in something like a Haskell DSL and hiding the underlining libraries; you can get that by creating an actual DSL and interpreting it on the fly.
But C or C++ state machines manually encoded into structured code is one of the things that won't get you there.
And a DSL is the exact opposite of what we're trying to accomplish: An electrician with no prior knowledge of the system needs to be able to read the code and understand which switches and sensors need to be turned on in order for that motor over there to start. Having to learn an entire new custom language at 3AM when you're losing $700,000 an hour in lost production is not an option.
(Edit: I should clarify that I'm talking about mining here. High end factory automation engineers may very well be more familiar with 'traditional' languages.)
Even the automation professionals I work with whose jobs are to write and maintain PLC programs look at the 'structured text' PLC logic I've written (a constraint solving algorithm) and can't be bothered trying to understand it.
Controlling a physical system is different from programming computers. The simpler the logic is to read and understand the better. Direct and verbose is better than layers of indirection and abstractions.
And they want this with the conditions you described.
It it true though that the mindset is decades behind the web-oriented world.
OO would allow for easier code reuse, by composing objects out of pre-existing objects, something that is usually done by copy paste today. OO would enforce encapsulation and prevent global periphery fondling by serveral free running state-charts, where its really tough to find out, who in what order did flip the switch and cause a crash in 1 of 1000 runs of machinery.
List goes on... Sorry, but for my use cases the complexity is usually not completely avoidable.
The problem is- as with all state-charts fast grows in complexity to a level where nobody has a overview. Are we the first industry to encounter this. No- chip designers have this all the time, game industry has this all the time, every fucking industry using state-charts has this - all the time. So how comes, we are the only industry taking a lot of overtime all the time?
<chirpin ciccadas since decadas>
I have people whos VMs literally collapse under page-long state-charts. So how about breaking them up, just have small state-charts in FBs and small state charts in FBs marshalling them.
Not happening. And they use assembler, not for a final tweak but as base language.
And usually, when the whole mess is collapsing in on itself, like a black hole of bad design, the project manager call in some external consultants, who should be happy to work a project that is "that far along -its allmost done".
A castle made from dinosaur bollocks.
I hear you saying "but you have debuggers/simulators!", for anyone actually having worked in the field you will know they are pretty much useless for big changes in machines that speak to hundreds of other systems, sensors, motors, etc. At least that's what my experience was and nobody has a backup factory to test changes on. In the tight schedules we had during production stops (Sunday nights, mainly) we were busy enough maintaining everything else than having fun on a PLC.
I completely agree that this is all a big mess and people don't tend to write maintainable programs, they just want their machines running and this is where everyone needs to be trained and improve.
Need automated tests in simulation, preferably based on data collected in real systems. Ideally also hardware in the loop testing.
Have you never met a game developer?
These days there are a lot of studios working on so called perpetual games (MMORPG, etc) which have constant maintenance and improvements so they don't need to let go of people. Also the other AAA game companies are bigger and can shift people between projects as needed.
What terrified me most is not only the state of ICS nowadays (it's pretty much fucked), but also the fact that cyber-security vendors try to make a profit from that by shoving their products down the throats of scared managers.
And what @donquchotte says is pretty much true - on facility there I practiced, all PLC were connected via TeamViewer to company network, passwords were written on stickers and slapped on monitors. They used Modbus over TCP, plain-text packets easily viewable with Wireshark.
On the first day I accidentally broke my S7 because of wrongly uploaded configuration, it took them 3 days to find some special Siemens laptop to connect to this PLC and reset it. Also tried to play around with Metasploit and follow the work of Beresford [1,2], but the most trivial attack of mine was just flood on PLC's IP with random junk - it stopped responding to any commands.
These things do not belong anywhere near the internet, patched or not.
This isn't just about hacking. A device that crashes hard when it receives a POST with an empty body may do the same thing if there's a physical glitch with its connection, or if a cosmic ray flips a bit somewhere. Devices that control industrial machinery should be more reliable than that.
If this were a hobbyist project, or a toy, that would be one thing, but these are specialized devices that cost a lot of money. They're fragile liabilities that don't belong anywhere near a shared network.
Yeah - I can see even the simplest of network vulnerability scans wreak havoc on a device like this.
This business reality is going to get worse as time goes on. Companies would be receptive to change if people at the executive level were educated first. The carrot being more cost effective system development.
This is already turning into a major problem with IOT devices from supposedly reputable companies i.e. home thermostats that let pipes burst because of a bad or incorrectly pushed update. Now consider if that bad update could instead kill someone.
bad updates are bad updates. they are incompetence on the part of the programmer. but the worst they should be able to cause is minor equipment malfunction or the physical system has been poorly designed.
Or something dumber, maybe they pushed a temp variable to the wrong type of memory (say flash eeprom) that updates every cycle of the control loop, and then did a push to an entire line of manufacturing equipment. Congratulations, all of those are going to brick in an hour or so when the memory hits it's maximum write cycle lifetime and you'll have to get new boards shipped in while your entire line is down.
Remote pushing needs to be handled very carefully when controlling real world equipment.
I have made a total of at least 7,500 remote updates over maybe 25 different control systems. In fact we basically get it working well enough the operators can handle day to day stuff and then go remote after that, because who wants to stay away from home at some dirty industrial site eating crappy food. The exception to that is backup power for hospitals, no remote access there and it is a simple enough system we can test all the different scenarios and then walk away.
Generally the last thing I put before an output in a PLC is some rate limiting. If it is a discrete output it won't be allowed to operate more than once every 5s for example. If it is an analog output it is limited to achieve the maximum desired actuator velocity or acceleration. This is a good catch all to avoid damaging equipment. I watched somebody learn this the hard way as a DC motor starter exploded when it was told to start and stop the motor 10 times a second.
Certainly I have made errors. A bad one was I forgot to limit a position so that it could not be less than 0. The position was subtracted from some other number. Substracting a negative number is adding! That was a nasty positive feedback loop that resulted in fast oscillations in the position of a 1m diameter pressure reducing valve. However that was during on site commissioning not remote.
Certainly for major changes I will require a shutdown and co-ordinate with the operators, but it is a judgement call on my part as to what I can program and test at the office and unleash on remote equipment vs. what needs to be tested on the actual equipment. Most testing on the equipment is required to determine the equipment characteristic.
You usually don't worry about anything other than the task your PLC is supposed to perform (we're in a factory after all and need to make money, so better do it quick). I don't think I can remember ever updating the firmware of any of our PLCs, most of them even rot in a storage room for a couple of years before even being put in use.
Everything is connected to the local network for monitoring purposes and if it weren't for another department strictly managing our network I reckon there would be a lot more of them accessible directly from the internet.
It's kind of like making programs to run on Windows 95. You don't have to worry that much about making your program reliable because no one will notice a few extra crashes.
First off, the damage to third parties might be extremely severe, especially in a targeted attack. Think of long-lasting power outages due to damaged equipment. Or think of environmental damage (e.g., suppose you got a nuclear power plant to vent radioactive gases, or worse, to have a meltdown).
Second, the damage to the owners of the industrial facility might be severe even if not to third parties. Here the attacker might benefit if, e.g., they are a competitor or have shorted the victim's stock.
The first case is of extreme interest to the public, don't you think?
Let's not dismiss concerns about industrial security.
For your first case, you failed to provide a possible. motivation. I will assume "terrorism" as that seems to be brought up as a motivation for most everything these days. There is more than one problem with this.
First, it is pretty much impossible to prove that an attack over the internet was actually carried out by any particular entity or even any entity at all. Things fail on their own. Second, it is usually easy to entirely prevent another attack. So no ongoing sense of vulnerability.
For your second example, it has always been fairly easy to damage industrial infrastructure but there have been almost no attacks of this kind. Stock market movements are not that severe in the case of even serious catastrophe much less the sort of trivial problems you could cause by attacking industrial controllers on the net.
A couple years ago at a Fortune 100 company, I dealt with a WinCE panel that began randomly spewing millions of broadcast UPnP packets - enough to make Wireshark lag badly. Took us about fifteen minutes to find and unplug the offending item, and the recommended mitigation from Rockwell Automation was to just disable UPnP. No patch was available for a problem that's been known since about 2006 or so, on an HMI product that is still being sold today.
Due to some terrible switch configurations on the part of the customer, the traffic made it out to the campus network and DoS'd most of their intranet servers. About 3,000 employees were sent home early for the day because none of them could read their email.
I don't allow the HMI computers to access the internet, but then they remain unpatched. As an OEM I'm out of the picture once the plant is working.
This seems more like a satire than real world.
It however appears to be offline but it isn't hard to create your own.
So far the news stories seem to have shown unsecured x ray machines, farm equipment, cctv, maybe oil wells, bank computers, cashflow software, waste treatment plants, parking lot systems, petrol station storage tanks [1] and a Swiss water reservoir [1].
It shows everything wrong with the lack of incentives for security!
[1]: https://securityledger.com/2016/03/vnc-roulette-feasts-on-in...
I can't see why that wouldn't be achievable. Yes it would take some extra processing power on the PLCs and would generate a bit more heat.
It would have to be implemented by Schneider and Rockwell, who have been pushing all responsibility for security to the network level.
At least the latest schneider PLC, the m580, allows for requiring a password to connect using the programming software[0], and also some ability to restrict what client IPs can connect[1]. The m580 was released in the last two years, so everything prior to 2016 doesn't have those features and comes set up to allow anonymous upload of new firmware by TFTP.
[0]: https://www.schneider-electric.ca/en/faqs/FA238215/ [1]: https://www.schneider-electric.com/en/faqs/FA282558/
A better solution would be a shared symmetric key that is manually programmed into all devices. However, this complicates install and maintenance. Its a hard sell when your controls are installed by contractors paid by the hour. Pretty soon everyone is setting their 128-bit key to 0x12345678 so nobody forgets it during emergency maintenance.
if the underlying mechanisms just work in every case except for key mismatch and give you a good error message in that case, requiring your contractors to enter the same string in two devices that they already have to program is going to be hard for them to justify as a great additional expense.
The biggest share of toll free numbers with modem tones were fax machines and but many were 9600 baud terminal for some CLP. I guess these were remote maintenance accesses for isolated machinery - I wonder if people still do this (may be an updated version with 3G or 4G).
Now, I've learned a lot about mistakes I've made, and I was never a perfect senior sysadmin, mostly because I wasn't playing the meeting room presentations/reports and politics game. So I'm not trying to make myself out to be perfect in any fashion.
That said, when I requested a full time t2 person. Denied. Requested a full time t1 person. Denied. I literally had to poach some part timer who was sweeping floors and train them myself... which did nothing to alleviate my time crunch dealing with desktop users in order to tackle the substantial infrastructure security issues, which takes time and thought (did I mention it was an open office area, so good luck getting good thinking work done...) Requested contractor structured cabling job (just horizontal)... denied...
Us sysadmins can secure this stuff, mostly through really good network policies and firewalls, etc, but if management is cost-cutting corners left and right and would rather hire some contractor-MSP type that does 1/4 the work at x3 the cost, while refusing to support internal IT teams with the tools and funding they need to get their job done properly, it's no fucking wonder insecure systems abound in the industrial world.
This is managements problem, and until the public or others start punishing them for not seeing IT and security as an investment instead of just a janitorial cost-sink, it will continue to happen.
That's the reason behind my last burnout. Now I'm happily on break pursuing my data science degree. I've been on the contractor side too, (dropped out of college after the Marine Corps to start an IT support company that is still alive today even after I left) so I've seen the inside of hundreds of companies, from fortune 500 oil to 20 man lawfirms, so I'm not just pulling this info from a couple of jobs. I saw it everywhere.
A good CIO or CTO should be able to address many of the technical-political disconnects, but they hardly exist. Go try asking /r/sysadmin how many of them even get a real budget. Half the time they just have to "ask" and hope they were convincing enough that management approves. Usually to a CFO or CEO since the CTO/CIO doesn't exist.
Oh, and of course they all want to immediately jump on the IoT bandwagon but still don't want to hire the people to get them out of their old technical debt, much less the new massive technical debt many IoT devices will bring!
Don't even get me started on the "big data" issues IoT and many PLC type devices bring.
edit: The key to the increase of this issue is the direct LTE connections on the PLC's, whereas you used to have a satcon to a router and everything was internal from there, that's less true these days. Even so, you still have issues with the security of that edge.
If you go look at the current postings for these roles you'll find that the average salary is between $15-$25/hr, you'll be on call 24/7, and you have to work physically demanding jobs in relatively harsh environments. I've seen a few outfits that don't even offer to pay their employees for commuting time between sites, only for when they are actually sitting down and working on the boards.
You know the only people who weren't defined as cost centers by management? Themselves.
But in an IPv6 world where NAT is no longer required, don't you fear that a lot of device might get exposed?
I know it will be impossible to scan the whole address space, but I am sure other means of discovering IPs will arise.
A stateful packet filter does.
Stateful packet filters work with both IPv4 and IPv6.
I guess it will depend on how ISP will deploy IPv6, but where I saw it deployed, the ISP did not provide NAT by default and all allocated IP were routable and inbound traffic was not blocked.
The single IP does not prevent inbound connections. The stateful packetfilter does. This has nothing to do with NAT.
Also, NAT is not usually done by the ISP, but by the customer's router.
Yes, that could change with IPv6. But it still doesn't have anything to do with NAT. Just because stateful firewalling happens on the same device as NAT, doesn't somehow make it a property of NAT. There also usually is a web interface on the same device. Now, with IPv6 that could change as well. Does that mean that web interfaces are a property of NAT? Or that it would be sensible to say "NAT is commonly deployed with a web interface"?
Also, as you usually still have only one (non-)IP, the CPE still does NAT as well.
And in any case, NAT does not prevent inbound connections, not even CGN. CGN might prevent inbound connections from the public internet, but your ISP can still connect to your internal network if you don't have a stateful filter on your side.
The disruption may be temporary as things would recover quickly, and the perpetrators would be hunted down by law enforcement and thrown into jail for a long time.