Ask HN: What do embedded engineers do? | Hacker News Reader

Ask HN: What do embedded engineers do?

22 pointsmeekasan34 comments

I would very much like to transition to doing more embedded type software that is closer to interfacing directly with hardware, but my only experience in this has been writing code for Arduino, different Raspberry Pi flavors, and sometimes the ESP8266.

I really enjoy doing these kinds of projects but through all the effort of making it as easy as possible for people to write, compile, and use code on these embedded chips I've never gotten the sense that people would be willing to pay me to do similar type of work.

I'm not under the impression that writing firmware or doing lower level programming is all like this, so I would like to hear what kinds of projects and challenges people who may have actually been employed doing work like this have come across. Was it interesting work? Is this type of software engineering in demand?

View on HN

The bulk of the work is mundane and can be done anywhere in the world, so pay is rather low. The work consists, much like pure software programming, of taking functional requirements and translating them into smaller chunks. Those become hardware and software components; the hardware you buy and assemble, the software you copy and glue together or write yourself. You know those disks at take out restaurants that buzz when your order is ready to pick up? Good example of a mundane design for an electronics engineer.

You can pick up work like that on upwork or fiverr, but you have to have a bit of business sense to know which projects are feasible and practical.

Combined with another specialty, it can be challenging and lucrative. High-speed/FPGA specialists for example earn more, as do instrumentation electronics engineers (physics/signal processing) and optical electronics engineers.

I’ll add to this excellent comment by mentioning that embedded+RF is another area that’s very lucrative and specialized. Lots of wearable communications stuff from low-end IOT to cars, industrial machinery, to high end phones use embedded and RF.

People used to mention medical devices as another specialization but I think nowadays the embedded design side of things is not the “hard part” and so it’s less lucrative.

Slightly off-topic, but: Sadly, and for reasons I don't entirely understand, embedded software developers tend to get paid far below similarly-senior desktop, mobile, or web developers. I wouldn't recommend a career switch unless you're really interested in the space and willing to accept the compensation difference.

It's the companies we work for (worked, in my case). They tend to be hardware companies, and software is not what they see as a critical, or even top ten, concern. It's not "real" engineering (not that most software engineering is real engineering, but in embedded, safety-critical systems it is, or at least should be). Programmers are pegged at a salary somewhere between technicians and lower level engineers.

It's gotten better over the years, but there are still huge misunderstandings and misvaluations of the different work. Obviously a hardware system needs hardware engineers, but nearly every hardware system these days involves software systems that do the actual work of coordination, communication, and control. It can't be treated as an afterthought but it often is.

Hardware companies have 10x the overhead of a software house. You can't have $200K+ prima donnas calling all the shots when you need a whole cadre of additional labor to handle development, compliance, production, test, packaging, and distribution processes.

Software companies have focused on efficiency, which is why the cost of building software has steadily decreased over the last 30-40 years.

From my perspective a lot of that is because the tooling and mindset for hardware is stuck in 1980s/90s. Compared to modern software development, there is very little to no automation in hardware design.

Things like CI for hardware designs and even automated version control are just starting to become a thing in the embedded space. Automation like building Bill of Materials, build pdfs, gerbers are not in any way optimized.

Scaling embedded stuff (design, manufacture, etc.) is very much a linear thing today, which is why salaries are low.

Hardware development has gotten much more efficient in the last 20 years across the board - both electronics and mechanics - but salaries haven't kept up.

On the electronics side, better modules like RPi's compute module, open source footprint libraries, and reference designs have significantly sped up development since I started twenty years ago. What used to take weeks or months in Altium can now be measured in days, especially if companies publish their reference designs in the application format instead of PDF. The turn around time and cost on PCBs have also dropped precipitously, to speak nothing of the turn key assembly that the fabs offer. Pretty much the only thing that has stayed the same is that MetCal induction soldering irons are still the best.

On the mechanics side, it's hard to appreciate just how much better vendors' CAD libraries have become, to the point where you can drag and drop Misumi/Mcmaster/etc parts from a Solidworks extension onto your assembly like it was Gary's Mod. 3D printing alone has made everything more efficient during development and has enabled many interesting production designs from rocket nozzles to turbojet engines.

this is a weird tangent; you don't need to be a prima donna or call the shots for your work to be fairly valued at $200k

> You can't have $200K+ prima donnas calling all the shots

Prima donnas? What a twit.

I wrote nothing about having software "calling the shots" (I assume that's who you mean as "prima donnas"), but when software is essential to your system, you should have that team at the table in your planning and budgeting efforts. If not, you'll get poor quality because it's low priority. And you'll get poor quality if you reliably underpay people, the good ones leave.

>> but when software is essential to your system, you should have that team at the table in your planning and budgeting efforts.

I like to tell them that software should be on the BoM even though the marginal cost in $0.00. Having it noted as a component helps get people on board with proper versioning, and also raises visibility for something otherwise unseen.

Visibility. Places that tend to build the kind of hardware that needs custom firmware are so preoccupied with the application space and hardware that the embedded software is almost an afterthought. You finally get the electronics, packaging, and other parts together and then just pass it off to the wizard to infuse it with some spells before it is ready to ship.

Pulling together working hardware, packaging, and logistics is non-trivial, often involves multiple people, many moving pieces, extended chunks of time, and heroic efforts to resolve the inevitable problems. When the hardware comes together, it is only after a valiant team effort and at some cost. Everyone knows how hard it was. By comparison, firmware is invisible, usually written by fewer (or one) people, and usually carries lower risk. If your firmware has a bug two weeks before shipping, it's a recompile and we're back on track. If your hardware has a bug two weeks before shipping, it might be the end of the organization.

With hardware, it's the physical 'thing' that is the product. With applications, it's the software itself that is the valuable thing.

Yup, switched from embedded (on high-speed trains), to modern backend, x4 my salary. Unfortunate since embedded was super interesting.

>> Yup, switched from embedded (on high-speed trains), to modern backend, x4 my salary.

These comments leave me confused. My Michigan based understanding would have you both poorly paid for embedded work and obscenely paid for backend work. I know some folks in CA make 200K and up, I'm just not sure how common that is. And anyone doing embedded should be at least 100K, so I don't see an easy x4.

Ah, the explanation is quite simple and I should've included it, sorry. I'm working in France, so I was at more or less 25k in embedded (not poor by french standard for my age, and had a lot of extra perks (like free train) compared to other companies, but not great either. I doubled when I started working for a french startup, and doubled again when I started working for an US-based startup.

Oh you changed countries in there too! That explains it ;-)

Off-topic but, could you share a transition plan with me?

I am a pure embedded sw. guy (3 YOE) and have already started to learn DB management and higher-level languages (Java and C#). How was your transition?

Any tips on moving from embedded?

Not sure I'm the best to give tips because I just had many great opportunities at work, moved from C/C++ embedded to Go embedded (same company) and then Go web backend at another. I wouldn't do anything different, because money is great, but tbh it's less interesting than C on an 8086 or embedded Go with audio/video, lua interpreter, GPS pathing and stuff like that.

what do you mean by 'modern backend'? backend is pretty legacy except for the SSR node.js movements these days as far as I can tell, maybe some Go-lang microservices too.

Maybe in my head 'modern' is the wrong word, I just meant web backend on modern stack (yes, in go), or at least backend for recent startups with everything in the cloud etc. While I don't really agree with using a shiton of cloud services (especially coming from embedded without even dynamic allocations), it does make for interesting problems related to distributed messaging and things like that.

> I don't entirely understand, embedded software developers tend to get paid far below similarly-senior desktop, mobile, or web developers

I disagree. At my workplace(regulated industry), Embedded software engineers are paid 2x-5x of other engineering groups. Infact, some of our embedded engineers have so much leverage that they have negotiated 3-4 day work-weeks. Second highest paid group are DevOps(though they don't actually do much for us, as there is separate IT department). And our generic software engineers for web and custom-devices also make decent salary. What you mention could be a US centric thing.

Just to make some sense, our embedded software engineers not only write good code, they also maintain their own CI/CD, extensive test coverage, harsh code reviews, very well maintained codebase(unlike on other places I have worked where arcane header files are thrown around and used, no questions asked), automated deployment and update system, A/B testing, rollback and other interesting things typically practiced by higher level(web/desktop/mobile etc.) engineers.

I have also worked at a big name AG where they make specialized hardware, the engineers there also made very decent pay but quality of software and development practices were more like script kiddies copy pasting things and calling it a day if the emulator showed correct operation.

This is why I left embedded unfortunately. The other commenter speaking of specializations as key to a more lucrative career is spot on though. I would have loved to have specialized in a subarea involving DSP but didn’t see much of an opening without even more school. And even then the open positions are few and required a lot of digging, and of course prior experience. Watching my colleagues with much easier software gigs get paid more and not be forced to fly to China on a moments notice made me realize I need to switch focus. I did really enjoy some projects I had the opportunity to work on and have no regrets. But I do miss looking at schematics as part debugging.

Profit margins are low when you ship physical stuff.

That depends on your market. In the aviation field margins can be pretty high for certain components, but the money is in the longtail (repairs and replacements), not in the initial sales. This leads to bidding low and skimping on budgets in the initial development portion. Once the system is deployed, they'll make plenty of money but there's less that needs to be done to sustain it on the software side.

Same in semiconductors.

Almost all of my ex-colleagues went to web dev/fin tech :(

When I was writing firmware, I worked for a product development contractor; clients would hire us to prototype some piece of hardware they wanted to sell, then we'd pass the design off to a manufacturer. I worked on medical devices, fitness trackers, even a little renewable energy stuff.

It was interesting work, and sometimes I miss it. There were immovable resource constraints which required careful thinking, and that felt like doing real engineering in a way that everyday software development usually doesn't. Debugging could be a significant challenge. Sometimes you can plug in a JTAG header and use a source-level debugger, or you can print to a serial port, but sometimes you have to get creative with spare I/O pins and and LEDs or an oscilloscope. I had opportunities to feel proud of my cleverness. One has to think carefully about time and power use and sleep modes and interrupts; your world is a state machine. It can be delightful to have complete control; often there's no operating system at all, except to whatever degree you end up writing one in the course of your project.

On the down side, the work came to feel repetitive. Here's a project, here's a dev board; bring up the devices, write the application, test it, ship it, repeat. Also, the firmware is kind of an afterthought compared to the mechanical & electrical engineering. You live on the hardware's timetable, which is slow and frustrating when you are used to just recompiling & running tests at the drop of a hat. Also slow and frustrating were the weird proprietary IDEs and generally old-fashioned engineering practices (though this was over a decade ago, and perhaps things have improved since).

If by "in demand" you mean "high-paying", I'm afraid not; there's a significant pay gap. You can make a comfortable living, but it's not a path to wealth like the tech bigcos offer.

Well, I work at a startup and I'm more or less the engineer. But mostly my work is electronics and embedded programming.

As a generalist I find the work extremely satisfying. You end up covering a wide array of subjects from assembly-level optimization up to designing communication protocols. You also have to be quite flexible as a programmer. While all embedded processors speak C, each has its own subtly different quirks. You have to be aware of where your framework's abstractions end and where you need to start manually poking bits into magic registers.

As others have said, the pay is not great. Despite being at a startup, I'm still being paid at about average market rate. It's good enough for my lifestyle.

But practically I'm not sure if one could be a great embedded programmer without also being an EE. The code is very tightly coupled to the physical hardware in ways that you don't expect, and there's really nothing in normal software development that prepares you for that. I mean, I'm sure some people do it, but I can't imagine how they could be at all efficient.

Do it if you think it's satisfying work, but do understand that your code is quite often butting up against physics. You sometimes have to code around the physical and electromagnetic properties of your widget, and it is really not at all straightforward if you haven't experienced it before.

Alternatively, you could just specialize in "Linux, but on a single board computer". Which is technically embedded, but not in the same class as what I'm describing here.

wages above 150K is very rare no matter how many years of experience you have, the potential on making money in this field is capped to say the least.

on the good side, it's less competitive comparing to apps or web etc, it's more specialized, so you can still find jobs in your 60s if you want to.

and yes you're supposed to know both software and hardware well.

Im in a PhD program at MIT where I mainly build embedded systems and its not something I would recommend unless you are passionate about it. Hardware is "hard" and frustrating to debug since you are navigating both the hardware, embedded software, and sometimes the external software the device talks to. Many of my peers just do software-focused projects and can iterate much quicker. On the flip side, showing people physical artifacts that I've built is rewarding. Having the skillset to build a physical product from the ground up is nice and I would argue there are less people around that can do that. I get at least one person a week that comes to me to help them debug a hardware project since they mainly learned through Arduino sketches but when shit hits the fan, its a daunting task to debug beyond the sketch.

The other thing to note is if trying to spin out, finding VCs to fund a hardware startup is also tricky since the time horizon is much longer than a software startup. That being said, I've worked at a hardware startup before and if you have a compelling enough story, you can raise money without needing an initial prototype. In fact, I would recommend this since I also know of a lot of engineers who've spent a lot of time building something first and then there is no market/funding for it.

Now getting back to me not recommending this route, especially on the PhD level. Stable jobs for this line of work are hard to come by and are often underpaid for reasons I don't entirely understand besides that pure software deployments can reach larger audiences in a shorter amount of time which equals more ROI. The space is a bit fragmented and the level of skill varies. Many undergrads aren't really taught embedded-c either and often gravitate to higher level languages which are more domain flexible. Unless you land a job at a tech company building rapid internal prototypes which pays fairly well, the more common route I see in this space is for highly skilled people working contract jobs which can be too unstable for some. If you really wanted to go this route, I would recommend coupling embedded engineering with another domain (e.g., edge machine learning, technology applied for conservation/ecology). I did this about 2 years ago in my PhD and I can say that it has brought me significantly more attention and interest from funders and potential employers.

Power controls, controls in general, communications devices, stuff like that. In c. None of it on Arduino, Pis, or ESP8266s. STM and NXP are big names in embedded. It's interesting work.

>> so I would like to hear what kinds of projects and challenges people who may have actually been employed doing work like this have come across.

I've been in and out of embedded for over 25 years now. Mostly in. Some things I've worked on over the years: Battery monitoring for a Ford EV back in 1997, control software for the Ibot (Dean Kamen's stair climbing wheelchair), a few small prototype gizmos, electric power steering systems (due to my early motor control experience on the Ibot), engine controller test code, airbag controllers (very boring, it's mostly endless diagnostics), ac/dc converters mostly for EV charging but also for other uses (up to 900KW), Electric water pumps, and something really fun that I can't tell you about yet. I've focused on motors because I accidentally became really good at Field Oriented Control and having that specialization pays well while being applicable to a bunch of different things - different products, but also the power conversion stuff was very closely related.

That's just the embedded side. I've bounced back to PC apps a few times in completely unrelated areas. It probably helps to be an engineer first, and software person second. This avoids the notion of being a "coder" and gets me involved more at product level design and development.

I know a guy who has done a lot of embedded audio DSP work. He's an older guy too and just had a few months out of work. It was fairly easy for him to find a job though and he's happy doing something new. Audio is an area where embedded and AI are actually coming together. If that's your thing, try diving into speech recognition on the Pi or something and then scale DOWN to smaller hardware. IMHO on-device voice control is going to become a mildly big deal in certain areas.

Be flexible and always try to work on interesting things!

My embedded portfolio for the last two years:

Battery Management Systems: Turn on and off 324 single cells in a 200 kWh array, measure their voltages and temperatures and ensure they're cells are not engaged above threshold temperatures (asymmetric for charging and discharging), or below threshold voltages.

Electric Vehicle Chargers: Communicate with car, communicate with charge point operator.

LoRa transceivers: Configure transceiver, send and receive encrypted messages over LoRa.

It was all Rust, and all very interesting.

But as other commenters have pointed out, there is a pay loft compared to other industries.

My boss is hiring interns/people without formal education to do embedded software. There is nothing interesting there - glueing (sometimes buggy) libraries together. Downloading Freertos’s port and writing some tasks is the most complex development. Nobody really cares if one knows all the processor registers inside out. It’s not sustainable career choice. Keep it as the hobby if you want some fun.

The job for an embedded engineer can vary wildly and it gets hard to define what embedded software even is. I’ve worked on microcontrollers in elevators and battery management systems for battery packs on the low end and I’ve worked on application processors, many-core processors DSPs and soft cores in FPGAs in telecom on the high end. Sometimes you don’t even notice the hardware. All depends on the job and the size of the company (do they have a platform team abstracting all the hardware away?).

As others say, many companies in the embedded space have had a very hard time realizing they are software companies and their practices are very old school and frustrating.

Talking salaries (Sweden), yeah it’s a bit higher in the cloud but not wildly so.

My recommendation is to start working in a not tiny company and on an existing product. Then it’s more about adding logic rather than knowing everything about RTOS and bootloaders. Them you will pick these things up as you go.

Reply on news.ycombinator.com