Embedded programming is like web development | Hacker News Reader

Embedded programming is like web development

the.scapegoat.dev

33 pointslarve23 comments

I've written about the same comparison, but came to a somewhat different conclusion. In my experience Embedded Software development was poor preparation for my later distributed-systems work (I wrote Embedded Software for 10 years, then distributed databases and logging analysis systems for the next 10). Embedded software is so deterministic compared to large-scale systems.

https://www.philipotoole.com/a-prayer-for-distributed-system...

"The idea of monitoring the responsiveness of a read-modify-write cycle is obviously nonsense from a programmer’s point of view (though not so for the microprocessor designer!)."

You can turn even a small embedded system into a distributed system if you try. It's a common fault of designers to sprinkle a lot of different processors throughout a system without realising they're creating a problem for themselves or the software developers who have to use the hardware. I've seen a single PCB with 8 processors on it (for no good reason other than the electronics engineer thought it was convenient and liked PICs), it made it so much harder to maintain the state of the system than it needed to be. (and even though the response times may be far more consistent than what a distributed web application may need to deal with, at least the underlying tools like TCP generally work pretty well, unlike the half-measures embedded developers tend to wind up implementing)

A system made of multiple distinct components, while it can be complicated, is not a distributed system. A SoC is not a distributed system.

For the system you describe to have the practical challenges of a distributed system it would also need experience some of the following:

- some of the "different processors" would need to fail at seemingly random times, only to come back again in a few minutes.

- the speed at which those "different processors" respond would need to vary widely, and different times of the day.

Isn't that the case though? I often have the case of some cheap-ass i2c peripheral getting locked up because of noise on the lines or what not. While this hopefully gets caught and we get the timing right to make things work in production, I pretend it's an async device that can fail and lock up at any point. It makes debugging easier.

What the author fails to comment upon is the sheer breadth of embedded applications and focuses almost exclusively on IOT to prove a point about distributed systems (which he does quite well at a medium-high level). The analogy falls apart pretty quickly when you compare webdev to a safety critical system with hard real-time guarantees. Such a system may not have a viable logging framework due to deployment location or security reasons. From a 10,000 foot level I agree with the premise, but I think the title fails to explain the real topic: "similarities between embedded IoT and web development"

As you say, web dev is not like hard realtime development. It's not like resource-constrained development, either. Not many web servers are trying to run on 64K of RAM, or on an 8051.

Note that I'm not saying "easier" or "harder". It is different, though.

(author here)

This is a real difference, and one I want to expand on in my next article. You can fuck around much more in a web context than embedded, but you still often find out.

Doing web "well" requires thinking in terms of realtime and memory constraints. I don't want the latency of my microservice to jitter, nor do I want it to use random amounts of memory. Not only will I be able to provision smaller machines, but I will also guarantee better SLAs. Building your microservice just like you would a hard realtime constraint (guaranteed memory usage per request, preferably no allocations at all, time guarantees (response within 50 ms or I fail) helps make a robust, performant and resilient web application.

(author here) Actually I think that applies very similarly. The same way you can't just log willy nilly in a web system because you can easily overwhelm your logging infrastructure or introduce unwanted dependencies, you have to be mindful of instrumentation in low-level code. The same way you might not want to write to flash (either you have none, you don't want to wear it out, security, no way to read it back out anyway) has parallels to how you containerize and deploy your app (as a lambda? as a docker without persistent storage?)

The real-time aspect and hard resource constraint are indeed the fundamental difference. Some of these (static memory allocation) make a lot of sense when building microservices, for example, but it's much more squishy. I personally like building my microservices very similarly to my embedded systems: event driven, with different priorities to ensure time constraints, and bounded statically allocated memory and queues. Even in languages like golang, my allocations are usually done up front, memory ownership is very clearly delineated.

While you might not have realtime in web systems, you very much want to avoid GC pauses and similar behaviour to avoid spiking your latency, as these things compound. The underlying concepts are a bit similar, at least in my head. I think "this needs to be O(1) in runtime and memory to be repeatable. If the constant factor is bad, we can work on that. But I don't need fast and elastic).

As for distributed system, I actually consider a SPI / i2c / CAN system to be the distributed system, and a lot of patterns (retry mailboxes, timeouts, promises, bounded queues, circuit breaker) make sense in both.

I definitely plan to write more about these lower level details, and provide some code examples to make the parallels more evident.

I am an embedded dev and use web technologies for visualization. I don't think anyone said one is more hardcore than the other. 15 years ago web developers were laughed at, but that changed when it got far more complex. Saying it today is just a signature for your ignorance.

Still, there is a huge difference in my opinion. C and Javascript are quite different to write, test, debug and deploy. I have not a deep understanding of web development and don't care much about the latest frameworks or the latest approach to solve something. I just use it for visualization purposes, which are fairly restricted on embedded systems most of the time. Quite the contrary to web solutions.

There is a contrast to your average C spin-loop and some exaggerated JS promise chain. Also I think some parts of embedded development are easier, especially bare metal applications. Only static memory allocation for example, predictable common access to shared memory between main loop and interrupt subroutines. On the other hand you don't just execute some unit tests on your machine. You probably need specialized software. The complexity of the system as a whole is far lower than all the monkeys involved in displaying a website. The hardware to use is most often far less general than a webbrowser and you really need to read the manual and system specification of your CPU of choice.

Of course embedded systems also include the usual linux box. This kind of embedded development is far more comparable to classical application development. Getting desktop OS to do I/O beyond keyboard/mouse/display comes with its own perils. In practice these systems mostly interface lower level systems that do specific tasks while providing interfaces to networking. All in all these are pretty distinct worlds in my opinion.

(author here)

Thanks for reading. Fully agree on web being both more forgiving, and ultimately more complex than embedded development.

I do think however that a C-spin loop can benefit from knowing JS promise style. One of my proudest libraries is a monadic task system based on futures, that handles cancellation, introspection, error handling and user interaction, in a static memory constrained system. It makes some more complex interactions like: - move the motor until the limit switch is hit - if the limit switch is not hit within 15 seconds, activate the fan for 5 seconds - if the motor controller fails to respond, then activate the fan but show a red LED - if at any point the temperature sensor goes above 30C, interrupt the motor moves by slowing down the motor first - if at any point the user cancels, turn on the fan before slowing down the motor to a stop

This makes for a mad state machine, but is actually not that complex in a task kind of write up.

I plan to write much more about this topic, including the fact that embedded is a lot of "library" work (managing datasheets and different architectures and sensor versions and register locations), while web work can be a lot of "keeping up with the joneses" (in terms of UX, frameworks, cloud tech, etc...).

Oh, I agree. It would be nice if state was encapsulated for the respective tasks. Other routines don't need some global state to read the limit switch of similar non-related sensor inputs.

Do you have that library published somewhere?

I am not convinced. The strongest point to me seems that in a high level views these are both distributed systems, which have to react and interact in real time.

On some very rough level significant similarities exist. But I think it is much rather the case that a certain kind of web development is like a certain kind of embedded development. Web development is a very wide field and so is embedded development. And for very many parts they are very different. If you are writing your own OS and you have your own scheduler, carefully saving registers before jumping to the new stack pointer, you are doing something no web developer would ever do.

The systems are similar on an abstract level, but in the realities of the details they heavily diverge. And this is not very surprising, I think. Afterall the only reason software is useful is because allows abstraction, which allows for generalization.

Basically agree, yeah.

(author here)

Good points, I wonder if at this point I'm just saying "software development is software development!"

I do think there are a lot of similarities in building the run-of-the-mill "sensors + actors + orchestrator + UI" embedded system, deploying and monitoring it. Which I must admit is most of the system I have built over the years, some had an expansive IoT/Connectivity component, some of them didn't.

After handling the low-level peripherals, I find that a lot of time is spent on say, properly handing UI workflows, be it in react or with an ad-hoc push-button / encoder / character LCD setup. Or say, doing a RPC + streaming data exchange over USB. The point I'm trying to make, and that I might expand on in more details, is that for people outside the field (mostly on the web side, of course), it might not be apparent that: "you know that USB is kind of websockets + polling HTTP?". Hopefully that leads more people to do embedded development beyond arduino and micropython (which are great, and honestly my first prototyping choice).

Another common task in embedded systems: writing a driver. Most drivers I wrote are "look up the kernel API, look up the device datasheet, write the glue", which I find very close to "look up $service1 API, lookup $service2 API, write the glue". There's subtlety on the one side in how to say, deal with priorities and deferred tasks and userland locking, but that subtlety is very much present on the other side too, with retries, idempotency, latency, event loss, etc... To me again, a lot of similarities, and I will use the same kind of techniques (mocking so I can mostly develop in userland / localhost, instrumenting tricks, etc...)

Writing an OS (or, in general, building the underlying frameworks and tooling, like say writing a database engine) is a pretty rare thing.

As someone who cut their teeth in embedded before going into web development, I've seen a significant amount of similarities between the two. The most obvious to me is that now we spend a lot of time connecting prebuilt libraries together - back in the day we wrote everything from scratch. The skills required to interface to a bluetooth library are not any different to say connecting to a map API on a web page... if it all works and does what it says in the docs. At some point your core skills and experience are required in the specific domains to identify and fix bad behaviour (i.e. your bluetooth library isn't working because you have physical antenna issues etc).

There is also a large amount of crossover as embedded gains more resources. Often you're working with javascript because your device needs to have an embedded web server with custom functionality. Sometimes you see SQlite on embedded flash, although debatable if they really needed it. Running Linux is far more common and it works pretty much the same regardless if it's on an MCU or a VM in the cloud.

The main difference I see is to do with the ease of fixing mistakes in the field. With web development I can usually git push and the issue is fixed for all users. With embedded, I almost never have an internet connection so it's a case of physically sending people to site or product recall to fix the issue if it's serious. Internet connections are becoming more common in embedded so we can remote update, but still the minority in general.

Lots of common ground, but just like any discipline it's that last 20% that makes the difference.

I have been thinking on these lines or more specifically; a generalized application of distributed architecture techniques ever since i came across Roman Obermaisser's book; Event-Triggered and Time-Triggered Control Paradigms. and exposure to ideas from the Erlang language/system. From the pov of architecting the structure of the system there are some similarities between Web and Embedded systems but it is at a high abstract level rather than specifics. The key is to understand the difference between Federated (most Web systems) and Integrated (most Embedded systems) and how they directly affect Reliability, Fault-Tolerance, Latency etc. system parameters. It is also the case that due to a low-barrier to entry for Web Development there is a lot of Incidental/Accidental complexity in it which is not the case for Embedded Development (i.e. less forgiving). It would be interesting/educational to do a Commonality-Variability Analysis between the two types of Systems and learn lessons which are more generally applicable.

First, Thank you for this article.

I think embedded system engineers should be more open to learn from other programming domain and try to catch the good things to apply it in the embedded world. For sure there is some specific constrainsts in the embedded world limiting the usage of web tools. But I agree with you at a system point of view there is a lot of similarities and we could solve a lot of problems by applying the same strategies.

I'm an embedded system engineer specialist on robotics. Each robot I worked on have multiple boards (my personal record is 35 boards). Scaling a system like that is an absolute nightmare. That's why I'm now working on an open-source microservice orchestrator dedicated to critical real-time embedded systems. The concept is to follow the metodologies used to scale complex web applications thanks to a tool able to deal with embedded system specificities. You can find my Luos lib here : https://github.com/Luos-io/luos_engine Let me know what you think about it.

Depends a bit what is meant by embedded.

On the higher levels of embedded Linux there can be a grain of truth in this (especially wrt asynchronous model).

But the lower levels of embedded (both on bare metal MCUs) and on the lower level parts of embedded Linux require hardware skills without a web equivalent. While embedded software developers aren't normally expected to actually design boards they do have to be able read a schematic and review it concerning the software aspects (eg are the components used already well supported in software, are the right debug ports available etc). And quite often you have to debug with an osciloscape or logic analyser. Basic soldering to build cables etc is also very useful.

Also a lot of embedded requires understanding and modifying large codebases (eg the Linux kernel) whereas in web development its more about using existing libraries and frameworks more or less as is.

I remember doing a short contract to do some embedded programming.. a change of pace from web dev work. Getting simple things to work in C was sooooo much more difficult than in JS. Debugging also a hell of a lot more difficult. Code change feedback required loading the code files onto the device. Mutexs... And yes it is partially a reflection of my I adequacy with C but I think it's still a valid point

I've been in both fields and I'd dispute some of this post, but I'm not sure what it's trying to say.

web is the default entry point for the vast majority of developers these days, and has a much more welcoming leaning curve. Of course it will have a less-hardcore reputation. Look at the ways in which even the developers specializing in webapps attempt to distinguish themselves among the crowd.

Thanks for reading, author here.

You are right that this doesn't say much (at least for experience embedded developers).

The follow-up points I want to make:

- there is a lot of value for embedded developers to get acquainted with what is going on in the web world. Not only is there a lot out there that's directly applicable to iot, but the tooling is miles ahead of what I encounter in the embedded world, and a lot can just be scaffolded out.

- there is a lot that carries over from senior web development to embedded development, and jumping from one to the other is more about "incidental" complexity (learning new tools, learning to read datasheets, learning some abstractions), but a lot of the more holistic thinking carries over. I see a lot of people scared of even opening the source code to a kernel module and I want to show that it's basically the same as glueing some S3 stuff to an SQS queue, both in complexity and in boringness.

I have been working with more junior web people, and shown them that I actually prototype my embedded systems in plain javascript, or using tools like XState, just because it's easier. Once they see how my javascript can be ported almost 1:1 to a C state machine on an arduino, big lights go on.

(editing: formatting)

I've really only worked in embedded (aerospace mainly), aside from an internship. While I enjoy working on satellites with all the entailed constraints, the relatively low salaries and slow pace of progress have been getting to me. I'm near completing a master's in CS, which should help a bit in elevating myself as a candidate without real webdev experience.

For those of you who have transitioned from embedded to web or another high-demand area (ML, distributed systems), how did you find that transition? And for those who've stayed in embedded, what is it about the work that keeps you from leaving?

Like many reader of this site, who are not "front end developers," and hence have watched the constant churn of fads and ever-changing cleverly-named frameworks and tools for churning CSS and HTML and attempting to shore up JS and browser support into something representing a stable and maintainable platformm,

and as a reader who worked in embedded for a decade,

my instantaneous reaction to this headline was "JFC NO!" prior to even clicking o see if it is e.g. about domain priors or what.

Half in rejection of the contention that the actual ecosystem is similar (it's not),

mostly in rejection of that idea that it should be ("never again").

Reply on news.ycombinator.com