Hacker News Reader

Comment by donquichotte | Hacker News Reader

PLC manufacturers and the controls community in general is lagging behind technologically by twenty years. It is very frustrating for a security-minded, computer-literate controls engineer to work with decade old hardware and software.

In a company I worked at, there were PLCs controlling sensitive processes, including heaters for welding plastic, connected to the company network via Modbus-over-TCP. A visitor with a guest access, nmap and pymodbus could read AND SET temperatures of any of the devices. A black hat could easily have started fires or leaked acid in our production floor.

View on HN

I mean, you're not wrong. PLCs are a digital technology evolved from... a electromechanical computers of the 50s. PLCs are not just twenty years behind. They're really just 60+ years of legacy.

It's true that PLC's use ladder logic, which is a representation of relay logic from 60 years ago. There's nothing wrong with ladder logic. It's easy for electricians to understand and change. More importantly, though, it's easy to see the logic flow that makes that conveyor belt that could rip off an arm, or that 1000HP fan, or that critical pump or that critical valve, function. It's important that the safety interlocks, switches, timers, and sensor values that control that piece of equipment be all grouped together visually to help avoid the frequent surprises (bugs), so common with text based programming languages.

If you want to beef up the network security to the PLC, be my guest, but don't disparage ladder logic. It's the right tool for what it's used for.

Agreed. Long live ladder logic!

It seems like there's opportunity here. A little box with raspberry pi or similar device and two CAT5 connectors to be placed inline between the PLC and the network, as well as between the network and all other network based displays, interfaces, and sensors. This would allow non-hardened equipment on a non-hardened network to be easily retrofit with encrypted connections.

Edit: These already exist. [1] ;-( Oh well. Just need to have facilities start using them throughout the site.

[1] https://www.lifewire.com/best-vpn-enabling-devices-4140254

Devices exist to filter out modbus data that doesn't belong [0], but I have yet to see one in the wild. Whenever I have offered plant owners additional security such as a VPN endpoint at the satellite ISPs base station, they have opted against it due to cost.

[0] https://www.tofinosecurity.com/products/Tofino-Modbus-TCP-En...

True to a point.

The difficulty arises when trying to implement something in ladder that is too complex for simple Boolean operations, and would probably be much clearer if written in a few dozen lines of Structured Text (assuming an IEC 61131-3 PLC). Unfortunately, company policies (especially in North America) mean that engineers are often forced to use ladder, resulting in a ladder logic program which is inscrutable for both the electrician and the engineer.

What would be sensible is using the right tool for the job: ladder for simple Boolean operations, and ST for more complicated stuff. Tech schools should also teach electricians and technologists the basics of how to read structured text - they do this in Europe, but I don't think it's common in North America yet.

The company I work for is at the inscrutable ladder logic point. If our PLC engineer dropped dead, nobody would be able to pickup and go. I would be at least a month or two of software archeology for someone to come in and take over.

I use ladder only on Allen Bradley PLCs and then only for primarily boolean logic. the AB ladder editor is much better than their function block editor.

Generally I prefer function block, especially in Schneider's Unity, since I find it is easy to watch a process operate the way function block diagrams are animated. The function block also promotes code re-use by creating your own blocks.

I use structured text only when the problem can't be well represented in function block, or more like I haven't figured out how to well represent it in function block yet. It is much harder to observe the operation of ST as the tag values aren't animated. Of course you can't use breakpoints on a process without stopping it, which usually you don't want to.

I will be the anti-ladder logic person here. It is a good language for simple systems. But it breaks down as the system gets more complex. Its really only good as a glue layer between modules.

Also the inherently binary and visual nature of ladder (and HMIs) makes version control impossible or expensive. Many PLC workflows I have seen are basically just a directory of files that you hope someone has been keeping up to date. Maybe there are previous versions. You will also have a hard time getting a code change history. Orchestration and configuration are also pretty primitive.

Mostly I have dealt with Rockwell, maybe Siemens is different.

I agree. Fortunately this tends to lead to PLCs driving relatively small systems. Like a small number of stations on a small rotating assembly line. That's probably a plus, actually.

Yes, PLCs do work well, especially since one typically just arranges to need relatively small PLC programs. But it is a lot like working with VHDL or assembly, and the only way in which you get to organize/abstract things is by organizing the actual hardware controlled by the PLCs into manageable units (e.g., a station on an assembly line, a small number of stations on a rotating assembly line, ...).

I've done this too. It's neat to watch a 'read-only' register on a PLC change because of a packet you sent it. Overflowing a buffer into an adjacent buffer remotely. It's an art that should have died long ago. PLCs need to be upgraded from C to C++ and devs should use standard C++ containers and remove all the fixed size C arrays.

> PLCs need to be upgraded from C to C++

You are implying that C++ code is generally safer than C code, even in the hands of developers who introduce buffer overruns in the first place.

This is so wrong I don't even know where to start. In C++, I doubt they will ever get it out of door. Which would solve the security problem, incidentally.

Modern C++ should be safer. If you keep dynamic memory allocation to a minimum, use the STL heavily and are OK with lots of copying then it's harder to screw up because there's not much manual freeing of stuff and buffer sizes are systematically checked.

Of course if you're going to leave C, why not use real time Java or some other safer language.... C++ doesn't seem like a very ambitious sort of upgrade.

software for setting parameters for hardware systems might not require complex logic. a language with formal verification would be a good idea in that case.

std::vector<whatever> in C++ is safe and is nothing at all like a fixed size array of whatever in C. People who don't understand this are ignorant of the facts. They are totally different languages. C++ is much safer than C. Go ahead and try to overflow std::vector<whatever>. You can't. It's not possible.

Also, it is relatively straight-forward to compile C code with a C++ compiler and adopt C++ constructs while throwing out all the old insecure C constructs. Doing this would vastly improve security and is achievable in a short period of time with little or no performance loss (it will probably run even faster).

Yet, all I read about on web forums such as HN are unattainable suggestions such as re-write X in the newest, non-standard, corporate backed/controlled language that isn't nearly as tested nor as performant as ISO Standard C++. Maybe it's just me, but I find these suggestions to be totally out of touch with reality and really uniformed about the vast differences between C and C++.

> std::vector<whatever> in C++ is safe

I may be remembering this incorrectly, but vec.at(x) is safe (i.e. does bounds checking), but vec[x] is not safe.

I'm going to try some test code here and check, but I'm pretty sure vec[x] will happily overflow.

Edit: https://gist.github.com/tonyarkles/e702e4d5b530a9b7cd3fc9837...

Definitely overflows and segfaults while unwinding the stack (presumably while deallocating the stack-allocated vector)

You can't overflow a vector. It's not a fixed size array. It grows automatically as needed.

What your example shows is an attempt to access a vector element that does not exist (std::out of range). And, that is undefined behavior and is documented.

Also, that code is 95% C (not C++). Even the includes and prints are C. Pure, idiomatic C++ would use iostream rather than stdio.h, and a vector iterator to access the elements rather than looping over the vector (in C like fashion) using an integer (which is unrelated to the vector) in an effort to access elements via index.

> that is undefined behavior and is documented

In C, buffer overruns are undefined behavior and are documented. Back to square 1.

> You can't overflow a vector. It's not a fixed size array. It grows automatically as needed.

His example shows the precise opposite: the vector didn't grow automatically. Instead, memory outside the vector bounds was accessed, i.e., the vector overflowed.

Yes, idiomatic modern C++ makes that less likely, by reducing the use of explicit indexes. Even then, you still can overflow a vector with innocent-looking code.

But can the compiler warn about such undefined behavior? Because if it can't, it's not much better than C.

I just tried:

    g++ -Wall -pedantic vec.cpp

And got no errors or warnings.

<Rant> Worser still, they have even trouble adopting object orientation. The whole area is stagnant in a Win95 Programming kind of way. Its truly horrifying.

Worser still, even the people producing the newer tools to allow for OO, do not really test these tools, so you develop in a environment where you are basically a beta tester on a budget (TC3 - looking at you).

Nobody dares to move, to not break precious backwards compatability and devalue the experience of "experts" who for several generations simply did fall into every trap state-chart based design can provide. Means, every project runs into complexity walls at the end, there are no automatic proofers to check the state charts, there are no unit-tests (If i here the poor excuse -"But there is no hardware to test against" one more time).

In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you.

Half of the projects in this area are not ond budget and not on time. There is actually not even planning for that- just a factor, which usually is bigger then 2.0.

Im so sick of the excuses. You cant compare conventional coding and PLC programming. Yeah, you cant, one moved forward and the other got stuck with the lowest common denominator - due to lots of non-software guys trying to get a food in. All these fb lego-players, with no clue what they do...

</Rant>

Did you ever have the pleasure of working with Siemens' SCL? I found it very frustrating to use but overall a better experience than just using the lego blocks programming you usually see. Combined with your data blocks you could at least get some structure into your programs.

I did, i did and i used the TC synonym.

The problem is- as with all state-charts fast grows in complexity to a level where nobody has a overview. Are we the first industry to encounter this. No- chip designers have this all the time, game industry has this all the time, every fucking industry using state-charts has this - all the time. So how comes, we are the only industry taking a lot of overtime all the time?

I have people whos VMs literally collapse under page-long state-charts. So how about breaking them up, just have small state-charts in FBs and small state charts in FBs marshalling them.

Not happening. And they use assembler, not for a final tweak but as base language.

And usually, when the whole mess is collapsing in on itself, like a black hole of bad design, the project manager call in some external consultants, who should be happy to work a project that is "that far along -its allmost done".

A castle made from dinosaur bollocks.

I think the big problem in us being that slow is that you can't properly test changes. The initial programs get written under immense time pressure to put it productive and what after that? Do you want to touch your productive system and refactor it and risk causing hundreds of thousands in damages for production loss? You pretty much need to nail your initial design and can't afford to do the usual "we will take care of our technical debt later".

I hear you saying "but you have debuggers/simulators!", for anyone actually having worked in the field you will know they are pretty much useless for big changes in machines that speak to hundreds of other systems, sensors, motors, etc. At least that's what my experience was and nobody has a backup factory to test changes on. In the tight schedules we had during production stops (Sunday nights, mainly) we were busy enough maintaining everything else than having fun on a PLC.

I completely agree that this is all a big mess and people don't tend to write maintainable programs, they just want their machines running and this is where everyone needs to be trained and improve.

Manual simulators don't give much benefit, manual testing in simulation is only marginally better than manual test on hardware.

Need automated tests in simulation, preferably based on data collected in real systems. Ideally also hardware in the loop testing.

> So how comes, we are the only industry taking a lot of overtime all the time?

Have you never met a game developer?

Don't know why the downvotes. I was once asked to work on the original Gears of War. After talking to the developers at Microsoft game studios, I noped on out of there. Most game devs I've met have horror stories of death marches followed by massive layoffs. It's... not a great sector of the tech industry, if you value work-life balance.

I would hope that the cycle of crunch -> ship -> layoffs is in the past now. From memory it used to be that way because the studio needed to crunch to get the product out the door to sell. Then once that happened they had no need for all of the people since there wasn't another product to work on.

These days there are a lot of studios working on so called perpetual games (MMORPG, etc) which have constant maintenance and improvements so they don't need to let go of people. Also the other AAA game companies are bigger and can shift people between projects as needed.

Working in embedded systems, also with a control background and software affinity, I find we have the same mindset problems in non-manufacturing contexts as well (any kind of industry that needs embedded systems).

>In my opinion, this industry is as ripe for disruption as it gets- if you can generate and update code from a high lvl object and process description here, and proof reliability and speed + allow for user-code - the industry would throw money at you.

They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place. For them it's just this nasty thing you need to throw a lot of money at until it everything works. The mindset would need to shift to software being the main space in which problems are solved, because in 2018 it is. I guess this also turned into a </rant>

> They won't, because they don't see the value in doing it right. An industry managed by mechanical engineers is not going to see how they're spending 100x as much as they need to for a worse product. Because they don't see the value in software in the first place.

This has largely changed in the automotive industry over the last few years. We still aren't quite there yet on getting all managers to spend on security preemptively, but at least we now have had the big honchos hand down the gospel that software is important.

I hear ya. The issue, though, is that the end product has to be at least readable, if not maintainable, by any half competent electrician, so lowest common denominator (ie. usually ladder logic) is as abstract as you can go. Also for most industrial controls, the system is "wide" rather than "deep", in the sense that almost everything is pretty trivial ('read inputs X and Y, set output Z if X and not Y') but there are thousands of X and Y. And finally, X etc. are tied so intimately with the hardware, which by the time the site is actually built and all the vendor packages integrated is only loosely related to the original design. So while I absolutely agree that it could be improved on significantly, the end result will still look (on the surface) quite like what we have now.

The systems where wide, and not deep. But by now they want industry 4.0 - means they want intelligent behaving machines. As in - material flow on fail rerouting, process controlling and self-adjusting, as in self checking for upcoming flaws. All parts are tracked over network, all guides to the machine have to be electronic.

And they want this with the conditions you described.

Is this in factory automation? I work in mining and everything is still very simple there, nothing more complex than a PID controller or speed ramps on some variable speed drives. I wish we could do some more advanced tuning, but the key criteria for anything we do is reliability, robustness (even in the face of sensor failures, everything is monitored and has manual overrides) and simplicity.

> the end product has to be at least readable, if not maintainable, by any half competent electrician

You can get that by using Python or similar languages; you can get that by programming in something like a Haskell DSL and hiding the underlining libraries; you can get that by creating an actual DSL and interpreting it on the fly.

But C or C++ state machines manually encoded into structured code is one of the things that won't get you there.

The people I've worked with who are operating power plants can't read code. Perhaps they can read a schematic and could understand ladder logic. They are usually quite skilled mechanically though.

Even the automation professionals I work with whose jobs are to write and maintain PLC programs look at the 'structured text' PLC logic I've written (a constraint solving algorithm) and can't be bothered trying to understand it.

Controlling a physical system is different from programming computers. The simpler the logic is to read and understand the better. Direct and verbose is better than layers of indirection and abstractions.

Absolutely not. Most industrial automation engineers, let alone electricians, are unable to handle Structured Text (the standard IEC text-based programming language for PLCs), let alone Python. Haskell is not even on the same planet.

And a DSL is the exact opposite of what we're trying to accomplish: An electrician with no prior knowledge of the system needs to be able to read the code and understand which switches and sensors need to be turned on in order for that motor over there to start. Having to learn an entire new custom language at 3AM when you're losing $700,000 an hour in lost production is not an option.

(Edit: I should clarify that I'm talking about mining here. High end factory automation engineers may very well be more familiar with 'traditional' languages.)

OOP isn't always needed. Most of the PCLs are very simple state machines where there really is no actual use for inheritance, or even encapsulation.

It it true though that the mindset is decades behind the web-oriented world.

Its simple at the start. And sorry, but with the stuff i work, its usually not enough.

OO would allow for easier code reuse, by composing objects out of pre-existing objects, something that is usually done by copy paste today. OO would enforce encapsulation and prevent global periphery fondling by serveral free running state-charts, where its really tough to find out, who in what order did flip the switch and cause a crash in 1 of 1000 runs of machinery.

List goes on... Sorry, but for my use cases the complexity is usually not completely avoidable.

Then again the web oriented world all too often come across as architecture astronauts...

Reply on news.ycombinator.com