Hacker News Reader

Comment by Kim_Bruning | Hacker News Reader

Oh I can tell stories too! Meanwhile these guys often have huge amounts of practical experience as electricians or control technicians/engineers to go with their PLC skills, so you can't just say we should get rid of them either.

Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?".

Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant.

Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!

View on HN

Yes, I don't mean to poke fun solely on the integrators. I've seen the "software first" type land in the same role and flail just as hard, if not harder. I recall one individual, who was quite proud of his "beautiful" ladder logic code, after about 3 months of being at the company ended up destroying a $100k+ chiller plant by cavitating all of the pumps because they didn't understand head pressure.

I think that's one of the core difficulties with PLC programming. You have to have strong knowledge on traditional science fields like thermal dynamics, material sciences, fluid mechanics, etc., while also understanding the limitations of a 16 bit floating point integer and why overflowing that can be catastrophic.

Sometimes I wonder whether one could design PLC-style software by first expressing the system’s constraints and then writing the actual code and having either (preferably) formal verification that the code will not violate the constraints or (less preferably) enforced assertions that will send the machine into a safe state if the constraints do not hold.

The system I mentioned in my other post controlled some VFDs, and one would occasionally get stuck running at minimum speed forever instead of turning all the off when it should have. Fortunately the only harm done was a stupid waste of power and and nothing was physically damaged. If it had gotten stuck at maximum speed it might have been a different story.

A lot of PLC controllers can certainly be unforgiving. I mentioned overflowing a 16 bit float point because that's exactly what happened at a waste water treatment plant I was called out to and the resulting regulatory fiasco that ensued probably cost the company tens if not hundreds of thousands of dollars.

I'd love to see more formal methods come to the field. Part of me says I'll return one day, maybe if AI kicks me out of my software field, but I'd really want to come back at a position I could healthily influence towards safety and correctness.

Seems like, here in the 21st century, one could reasonably have a model of the plant that tests could be executed against. Anything like that exist?

The industry/consulting term is "digital twin". Nothing off the shelf, since every plant is different. Even for a single process it gets complicated really fast. You probably want to track the environment itself too like temperature, humidity, leaks, etc. All the machinery and if they don't have necessary sensors, gotta retrofit them. Oh and human workers too. Stuff like "John Doe pressed a manual override button at 9:25 AM" or "it takes 12 minutes for a forklift operator to move a pallet from zone A to C".

I was involved in a project that made such a twin for a warehouse. It was simpler in theory because there was no manufacturing process. Just moving objects from one place to another. It was still expensive and took a long time. We tracked traffic heatmap but also collected data on and modelled luminosity for safety. On top of that there was a standard warehouse space optimization.

You're going to run into similar problems as with testing and documenting code. Eventually something will change - a process, a machine, a sensor. You have to stay on top of things and update your model too. Otherwise they will drift apart and not be as useful.

It can become really complex. As an example, try to predict resonant frequencies on an entire plant full of rotating equipment. It can be done, but it often requires very expensive equipment and skilled technicians capable of using it to profile a plant. And that's just one potential problem you can run into.

We're often talking about custom plants here, or plants made in very low volume. Making a simulator for that can be prohibitively expensive, since you can't amortize the cost. For particularly important/expensive plants it still sometimes gets done. Though note the simulator might not be able to capture all the nuances of the real world deployment, so it might not be sufficient.

Meanwhile having a separate copy of the plant is also not often viable, once the plants get to any kind of size.

edit: After having 'someone' fact check and remind me: there are quite a number of simulation tools available that can sometimes partially do the work.

The trouble is the simulation tools are either proprietary and thus have only a small mind share among software developers, or academic, and either way, PLC programmers have not gone near them yet.

But a decent LLM should be able to translate the description of a plant to a TLA+ model (or Alloy, or something similar.)

This is starting to be done for very high value plants (think chip fabs, large datacenters, that kind of thing. When the infrastructure is a $10b expense it's worth it).

Small plants in the middle of nowhere, especially municipal ones? You're lucky if it was designed this century and not 50-100 years ago. Everyone's just trying to keep old stuff running at absolute minimum cost.

Reply on news.ycombinator.com