Debugging Humidity: Lessons from deploying software in the physical world | Hacker News Reader

Debugging Humidity: Lessons from deploying software in the physical world

physical-ai.ghost.io

16 pointsboulevard14 comments

Some of my code gets deployed to a PLC aboard a wave power generator hundreds of metres offshore, with a cellular link that might go down in a storm. If something gets unrecoverably wedged, retrieving the device starts at $10K to hire a ship.

I feel this blog post hard.

Wow, that's a fantastic and terrifying example. "Retrieving the device starts at $10K" is about as high-stakes as it gets for software reliability. It perfectly crystallizes the difference between rebooting a cloud instance for pennies and the extreme costs of failure in the physical world.

> This is why so many “IoT platforms” die in pilot purgatory. They are built by cloud engineers who underestimate the friction of the real world and overestimate the availability of bandwidth.

Indeed. That's why it's important to send your engineers along with the sales folks to these sites. If anything just to get a perspective on things like that.

> The first time I deployed code to an actual factory floor, I learned that "edge compute" doesn’t live in climate-controlled racks. It lives next to dust, grease, and forklifts.

And bugs, real ones not just nice abstract software ones. So you may find yourself debugging spider webs and ants crawling around, which always makes for great puns and stories.

There is absolutely no substitute for an engineer seeing, smelling, and hearing the environment where their code will actually run. The number of times a mysterious software glitch on a remote device has been traced back to insect nests, rodent chewed cables, or just a thick layer of industrial grime is not zero. It gives a whole new meaning to flushing the cache :)

Okay, but what about humidity? I was excited to read about a failure mode where the moisture content of air mattered, or at least get mildly clickbaited into learning about a tool called Humidity. Instead there are no other references to humidity apart from the title

The cloud is 100% humidity, I suppose.

Too much humidity makes it rain, and rain can make the Wi-Fi work: https://predr.ag/blog/wifi-only-works-when-its-raining/

I chose "Debugging Humidity" as a metaphor for all the invisible, pervasive environmental factors that you have to deal with in the physical world. Latency, power flicker, interference, etc. It's the "stuff in the air" that messes up clean logic.

The title was actually inspired by a real incident where a device kept failing every afternoon. We eventually realized that condensation from the facility's massive air conditioning unit was dripping onto the enclosure right above the SoC. We were, quite literally, debugging the effects of humidity. I should have included that story in the post itself.

Isn't

> Now, imagine your request is actuator.rotate(90).

a good example of something that is not idempotent? As it is based on its current position. Actually idempotent would be: `actuator.rotateTo(Degrees(90))` with a predefined frame of reference, or a frame of reference that you can include in the request.

Like the difference between a servomotor vs stepper motor.

Your suggestion of actuator.rotateTo(Degrees(90)) is precisely the right way to engineer a robust, idempotent command for the physical world.

I used the simpler, non-idempotent rotate(90) example intentionally to illustrate the default trap. How a pure software mindset can dangerously oversimplify a physical action.

When it rains, it cores

Haha! That's brilliant. You have summarized my entire blog in four words :P

Obi-Wan, is that you? :)

Hello there.

Reply on news.ycombinator.com