Hacker News Reader

Comment by msadowski | Hacker News Reader

My journey is a little bit opposite to the author’s: I studied robotics and started learning programming by myself to create software for robots. There are two observations I have that can be useful to people making the software->robotics journey.

* Agile for software-hardware is hard if not impossible. For software it can be reasonable, but it’s really hard to iterate on both hardware and software at the same time. It’s easier if the hardware designed is locked or the process involves waterfall on the hardware side. * I often found that people that come from pure software to robotics don’t have control experience and something that can be easily solved with a PID controller end up being a custom code that ends up being way more complicated than needed.

View on HN

I'd go one further and say that basically agile in hardware is fundamentally impossible in any way people commonly use agile.

If you have any custom hardware, you are basically stuck with a turn time of at minimum a day or two for any changes (often more, weeks + for new PCBs is common if you're not throwing huge amounts of money at people).

In this context, any process that depends on rapid small iterations is basically impossible, because each iteration just takes too much time.

Ive done firmware for a long time. Platforms vary considerably and most have surprises. Debugging facilities are limited. Reference examples are sparse and chatgpt hasnt got much info on niche OSes like zephyr, and safertos except some FreeRTOS.

Pieces of code interact more heavily than a linux machine. Testing requires more hand-holding/baby-sitting. Cross-platform architectures don't scale down well. There are many types of comms busses with no/few standard embeddings.

Some teams know how to make it a lot easier. Some CTOs know this, but most find out the hard way. Embedded practices lag webdev by 5-10 or more years bc they were good enough for a long time or for small projects. Expectations are rising, but there is less leverage than adtech so salaries are ok but not explosive.

> Reference examples are sparse and chatgpt hasnt got much info

Chicken, meet egg. If a web crawler can’t find plenty of reference examples, then LLMs trained on web crawled data aren’t going to be very useful.

I wonder (as someone who's basically always been in the pure software land) if the way to get around this is to overbuild your hardware prototype. Throw on more sensors, actuators, and motors than you actually need, and parameterize the physical properties of the hardware (like mass, power, and center of gravity). Then do your iterations in software. You can always ignore a signal coming from a sensor, but it takes weeks to add a new sensor and get a new prototype. So work out all the behavior in software, where you can iterate on a minutes -> hours timetable.

Then once you know how it works and have done most of your optimizations, you can trim down the BOM and get rid of hardware that proved useless. You'd probably want another round of tuning at this point, but at least you know basically how it works, and have tried out many different combinations of hardware in your software iterations.

That's what dev kits are in electronics. They usually even come with schematics/PCB layouts so an engineer can quickly copy a working design and remove the stuff they don't need. There's still a huge gulf between those prototypes and production and there's plenty of mistakes to make requiring multiple revisions.

It was the same end-position prior to internet publishing of software.

No updates. No bugfixes. What you pressed to CD (as much as a buck per CD qty 1000), was what your customers got. Period.

And it hurt to republish, because those updates were via mail. The box/sleeve, the labelling, the labour, the shipping.

DEVs today have no idea how easy their push is.

Get it right or you fail.

It's unfortunately not that simple. For most parts you can get breakout boards or dev kits that implement the chip and all its support circuitry. You can drop those into a simple PCB or just a breadboard and get going pretty quick. This is appallingly expensive for more than a handful of prototypes, but it does work. IME, this is how software people approach electronics.

The real trouble comes during the next step. You have to integrate all these disparate boards into one unit. You can copy schematics and layouts from the dev boards, but that takes a lot of time. Once you've done that, you need to re-validate your entire stack, electronics, firmware, and software. You will inevitably find that you've missed a connection or the routing introduced some noise or you got this component value wrong. So you spin a new board, wait a week, and do it all over again several more times.

Debugging electronics is a lot like debugging software. Except that the code is physical objects and the bugs are emergent properties of the whole system and are inherently chaotic. You (generally) can't step through the system like you would with code, you can only observe a small number of nodes in the system at any one time.

It's harder than you expect because you're dealing with invisible forces and effects of the fundamental laws of physics. It requires either very deep knowledge or a lot of time in iterations to solve some of these problems.

To expand on this, if software was like hardware:

    - when your function calls don't have enough white space between them, they'll some times mix up their return value (crosstalk)
    - the more deeply your if/else statements are nested, the more random their results end up being (voltage/IR drop)
    - when the user makes a selection, your bound function is called somewhere between once and a dozen times. Each time with a different value (bounce)
    - their is no `true` or `false`. You just are given a float between one and zero, where one is true and zero is false. Sometimes `true` is as low as 0.3 and sometimes `false` is as high as 0.7
    - the farther apart your variable declaration is from it's use, the less it represents what you actually declared. If it's two lines apart, a 'foo' string is still 'foo'. Hundred lines apart though, and 'foo' might become 5.00035 (attenuation)

If you want your program to execute as fast as possible, you have to worry about the speed of light! A trace a few millimeters longer than its partner can have nanoseconds of delay which can easily corrupt your data!

And don't forget that you have to balance the physical shape and arrangement of components and traces with frequency and surrounding components otherwise you've created a transmitter spewing out noise at tens of MHz.

Or the corollary: if you aren't careful you can receive radio signals that will corrupt your data.

Oh, you think your wire can handle the 0.5A your widget needs? Let me tell you about transients that spike to tens of amps for a few hundred nanoseconds. But it's okay, that problem can be solved with a bit of trigonometry.

On the plus side, if you forget to connect your ADC to something, you now have a surprisingly decent random number source.

I love the absolute chaotic insanity of electronics. On the surface things make sense, but one level deeper and nothing makes sense. If you go further than that, at the bottom you'll find beautiful and pure physics and everything makes sense again.

I feel the same way about software. It's a hot mess, but under everything there's this little clockwork machine that simply reads some bits, then flips some other bits based on a comparison to another set of bits. There's no magic, just pure logic. I find it a very beautiful concept.

Not so fast, some alpha particles from a distant galaxy strike your memory chips and some bits flip. If the CPU gets too hot or too cold it starts misinterpreting opcodes, branches, etc.

The reality is that computers are comprised of several PCBs running with thousands of multi-GHz signals. So all of the foregoing engineering design principles had to be observed to make our systems as reliable as they are.

I came here to say what you just did. Hardware modularity is incredibly useful in the face of unstable product requirements. Focusing on integration, optimizing form factor can come once the requirements are locked in (if such a condition ever exists lol).

It's common to design PC boards that have holes and traces for components that aren't installed. If you need three motor controllers now, design a board with space for six, plus a prototyping area of plain holes. Allow for extra sensors, inputs, and outputs. It's easy to take the extras out of the design later when you make the board for the production product.

When hardware design can be software emulated then iteration can be more agile. There's a story how NVIDIA was the first one to do this for GPUs- this was done out of desperation because they were out of money and had to ship quickly. They didn't have time and money to do any revisions so they just shipped what they had done in software emulation even though some of the features were defective. https://www.acquired.fm/episodes/nvidia-the-gpu-company-1993...

Absolutely all chip designs get software emulated/simulated for rigorous testing before being sent to the fab for production.

It seems like the publication turned what is industry standard into a sensationalistic article.

The only thing Nvidia did different was rolling directly to tapeout with their simulated design without any intermediate prototypes which was indeed a risky move but not unheard of for cash strapped semi startups.

I'd go one step further and say that Agile might as well be tossed away entirely since it's common enough for companies to treat it like waterfall anyway, making the practice of many small iterations or iteration at all unwelcome. If your software team embraces iteration and incremental improvement, any presence of agile is probably redundant; if they aren't, or the nature of the work doesn't facilitate it, then Agile gets in the way regardless of the domain.

Reply on news.ycombinator.com