The Rise of Python for Embedded Systems
zerynth.com
zerynth.com
> "Expert and skilled C programmers could justify this stating that “C generates a faster, more compact, and more reliable code”… BUT … if you replace C with “assembly” in that statement, you’d have exactly what an extinct generation of programmers said about 20 years ago!"
Yes, the differences between C and Assembly are directly comparable to the differences between Python and C.
> "There’s an enormous crowd of professionals skilled in using Python “potentially” able to develop the software for the “next big thing” in IoT and to ship new amazing embedded applications in a short time."
Python doesn't even appear on the IEEE rankings shown at the start of the article, It's amazing how this "enormous crowd" can stay so silent.
Are there any examples of products with an MCU running Python being manufactured at scale? Even if Python provides benefits for rapid prototyping, I'm really struggling to think of a business case where you wouldn't replace it with C or C++ before going into production.
I hear good things about MicroPython as well.
I'm not familiar with MicroPython, but checking out their website, I expected a testimonial section heralding how it's used by * insert IoT company here *. I'm still not convinced that Python is being used for anything other than hobby work and prototyping on MCU-based designs, but maybe I'm just not looking hard enough.
This was linked in the article and at the top of the page so.. https://www.zerynth.com/built-with-zerynth/
You're not wrong though, almost no one is using it right now
To unpack that for PC programmers: the best documentation, API design, and tooling for a microprocessor often happens at the memory map level. Sad, but true. Everything above that is typically cat shit wrapped in dog shit. The C ecosystem natively supports memory and addresses as abstractions, letting you directly consume those APIs, documentation, and tools. Note that this is one of those cases where "best" emphatically does not imply "good," and that it lies in stark contrast to the situation on "real" processors where you'll almost always find solid abstractions much further up the stack.
I once worked on a project where some poor soul took the time to provide a zero-cost abstraction, described in a Scott Meyers presentation[1], for every register on an MCU. This resulted in thousands of lines of C++ template classes, which, while hurting compile times, resulted in a really strong, safe and performant API that drivers could be built upon. It was wonderful to use, but all that upfront work isn't realistic for most projects.
I'd definitely be interested in a language that supports this type of abstraction "natively" (i.e. doesn't require lots of boilerplate code) while providing C-like performance on bare-metal platforms. I know that some people are trying to make Rust fit this hole but I'm not sure how that's coming along.
> "There’s an enormous crowd of professionals skilled in using Python “potentially” able to develop the software for the “next big thing” in IoT and to ship new amazing embedded applications in a short time."
Potentially is the key word. People aren't currently doing this, but they will if Zerynth succeeds in what they're doing.
i just can't understand why someone would want python for embedded applications. i primarily use labview for systems development, and there is not a single thing that would be gained by moving to python while many things would be lost. and no one can convince me that using python over labview would generate more robust code. i have used python before, and it was slower to develop in and more error prone than labview for system development.
personally, i want to see more lisps/schemes and ML-based languages on embedded systems. i know there are some, but they aren't quite there. that's the direction i think we should move. python isn't the best at any one thing other than having lots of libraries. there are far more productive dynamic languages (lisps/schemes) and far better languages for developing safe code (ML-based). i am currently learning idris, which seems like it could be amazing for embedded code development. having provable state machines would be a huge plus.
Exact opposite in my experience. I've been there, tried that. Re-wrote it in python for a more than 50x productivity gain.
At my first job out of school, I inherited maintenance for a couple of projects. All were extremely expensive custom test equipment, written in Labview. One was the central project, the whole point of this group's existence. It had been in development for more than two years. It routinely destroyed thousands of dollars worth of hand-made sensors during its power-on self-test. It was a far cry from being finished. It couldn't even be considered a prototype.
In a two-week period, I taught myself python, quickly cloned the existing calibration procedures, and finished the entire thing. My boss thought I was a genius rockstar, but no, I'm not. I just had the sense to evaluate every available tool and came to the conclusion that Python (mostly due to the ctypes module and a robust Windows ecosystem), was the best option.
Personally I'll use C whenever possible for embedded work. (And later I did port the entire project to a DSP running C, because Labview & an AD DAcq board was too expensive.) But if latency doesn't matter, Python is fine.
edit: Side note, I was already conversational in Labview. Admittedly only for school projects, but at that point I knew more Labview than Python.
What's the right platform for "embedded systems" is debatable. A true bare-metal app has different requirements from something running on RPi or Beagleboard.
but in general, without proper architecture, any project will become a mess in any language. this isn't special to labview. all of my systems and code in labview are extremely extensible and maintenance. why? i pay attention to design and architecture and use labview's OOP functionality. people regular comment how great it is that i can add full features in mere minutes, including UI behavior, and they are regularly surprised by the end result i have with labview. is that possible in other languages? yes. but labview's dataflow makes it very easy to reason about local changes.
Labview is great for data acquisition and GUIs to visualize this data. Drop a ready made UART/TCP/VISA block, some array manipulation in a loop, connect a waveform block with zoom/cursors/etc. and you have a working real-time app in a couple of hours.
Matlab is perfect for heavy matrix operations on large datasets, fantastic for plotting your data and it's very good for prototyping algorithms.
I tried python as an alternative for both. Tons of boilerplate code, a pain for GUIs and don't get me started on matplotlib/scipy/numpy...
The biggest drawback of both LV/matlab is their proprietary nature and the expensive licenses which make them unsuitable for hobbyists (at least there's octave as a matlab alternative). But they're great tools for the appropriate tasks
With LabVIEW you can compile and redistribute, but good luck writing an efficient binary search tree when your data gets bigger than what can be searched for with brute force. Also, at least last time I used it, the version control was horrendous.
For MATLAB, just forget redistribution in general. Nobody is going to buy a MATLAB licence to run your code.
Do you know of any Labview -> Python conversion tools?
I would think that would be a pretty ideal solution for the right problem domains. You could get a quick first version up, and then when you need more maintainability or flexibility, convert to Python.
Obviously, this would require some kind of Labview library in Python, but that seems like it should be relatively easy to make as a wrapper around the Labview included library.
That's true, but it's not their strong selling point. There's tons of problems to solve at the r&d stage when dealing with custom hardware, a dozen different instruments and algorithm exploration. That's where these tools shine and the reason they're ubiquitous in industrial/military/aerospace environments.
For a full product with drivers, libraries and fast, customer friendly UI sure there are better solutions, but there's no law that dictates to use the same tools at the development stage
That's a bit far fetched. People already having a Matlab license (and it's not like there aren't many of those) already made that cost, so buying your Matlab product is not really different. Alternatively you could compile your code into a C API and distribute that instead: the Matlab runtime itself is free of charge.
since labview is dataflow and has static typing, it shares a lot in common with functional languages. this means you can trace your data and execution in a very deterministic manner. i often write code that just works. there are almost always no bugs in the final software. that's almost impossible in python.
labview also has OOP, which any respectable labview developer uses heavily on any project. for larger projects and systems with concurrent processing needs, it's multithreaded features and actor framework are necessary.
labview makes it easy to code simple projects, and so people don't treat it with respect. labview code is often written by non-software engineers, who will write poor code in ANY language. people seem to forget that. if one treats labview as any other language and with respect from the perspective of a software engineer, then it has a lot of functionality that is very difficult to replicate in other languages. for example, async is all the rage these days, including in python. labview has had that functionality for many years now. and the dataflow paradigm pays off big time.
labview is different and people don't respect it. that's okay, because i do which makes me pretty valuable and quite efficient.
and i do think labview could be improved. it's just i wouldn't put any effort into python, especially for real embedded systems. many, many languages out perform it in many vectors.
I've always been really repulsed by labview for a few reasons: expensive, proprietary, vendor lock in to national instruments. Also prefer text only rather than graphical click and drag programming.
For these reasons I'd choose python over labview in a hypothetical project where the hardware could be chosen to support either
I feel that more options for embedded development are always better than fewer options. Personally I'd prefer Rust or Ada over Python, but other people like dynamic languages better.
in general, i am looking for a lisp, scheme, or ML-derived language. there are places where these can be ran on embedded controllers, but they aren't complete systems. it's great that people are putting together these alternatives, but it just feels python is being force fed in places it doesn't belong. i don't see how this article can say python is better for safe code.
I work on an embedded soc, and it's seriously easy for an experienced systems guy to make a python interface to the huge C++ high level driver, without knowing very much python (let alone some kind of "productive" language).
On the other hand, my experience using labview was like amputating my fingers with a spoon. So I probably didn't spend enough time with it to 'get' it. But it did crash a lot. A friend of mine uses it in the research lab he works in, and he hates it with a passion, and complains about buggy closed source modules and hidden accrual of state.
Need to quickly have a basic graph for some signals in 'real' time? I'll pick LV over anything else: it will be faster to develop, more robust, and just work (until you want to customize the graph). In Python I'd first have to go look for the appropriate libraries to use, start fiddling with gui layouts, etc. And after a couple of hours it probably still wouldn't even look remotely as good as LV and still lack basic zoom/cursor functionality.
But for even basic if-then-else and loop programming (not even talking 'true' algorithms) I'm, after about 10 years of Labview experience, pretty sick of using Labview for it. You see in Python I can just do [func(x) for x in someRange if x > 0 and x in someOtherRange]. In the time I typed that, in Labview I would have maybe managed to draw the outer loop and drag the func block into it. Then I'd still have to the conditional logic, all the wriring, etc. So even though you say for some cases Python might be slower, there is no way you can do this real life example faster in LV. And that actually goes for a major part of the program logic.
edit Also a note on Python vs C for embedded and the likes: we're currently running an application where the whole backend is C++ and does things like signal processing, but the frontend is written in MicroPython and with usability in mind. As such, I can just give it to very inexperienced programmers and they can actually build something useful with it. Yet they don't even have to touch a fiddly build system or learn how to shoot yourself in the foot a million times and the backend still happily runs speedy code. So again: heavily depends on the application.
It's not a silver bullet but it's nice to have it as a solution.
labview has some great libraries for developing complex systems. thiugh i a, the first to say it isn't close to end game.
I think Python has _everything_ a language needs to be the future of Embedded Systems - except a compiler.
You start writing you program using MicroPython as interpreter. You add type hints because it's the right thing to do anyway. Once your prototype is done, you compile to get the speed (cutting execution time, increasing sleep time, increasing battery live).
My background is Aerospace Engineering and maybe that's why I can't understand why nobody does a real, AOT, Python compiler. There is Nuitka, but it's a transpiler that extensively uses libpython and isn't fit for microcontrollers (still awesome project, tough). Cython also uses libpython and a different type-annotation system.
Though I suppose if you were writing a compiler, you could require and enforce type hints... but then any calls into the standard library will just have to be trusted. I dunno. It seems tricky.
The whole story with these critical systems is tricky. At the one hand, they want to have formal verification. Academia really loves stuff like Haskell for that. There is Ada which was designed with critical systems in mind.
On the other hand, there is a lot of C and C++ everywhere. On microcontrollers, no matter how much you love Haskell, Ada or Rust, there might not even be a compiler and probably no HAL.
And in the end, all of those languages focus on the machine and the math. A mathematical prove of the correctness of your software is nice, but readable code is too. And the formal verification doesn't save you from making wrong design choices. It doesn't check your logic. No language or tool can do that. But what I believe a language can do for you is help you express your thoughts in such an simple, structured and readable way that the cognitive load is reduced to a minimum, freeing up thought-power to check your logic.
A lot of the failures in space that were caused by software weren't actually programming errors like the overflows that allow us to hack everything. There was a Mars Orbiter that crashed because the data was in the wrong format - still the types matched. A lander crashed because they forgot to put in logic to handle invalid sensor data: a broken sensor in the landing led triggered the 'touch-down' signal, the lander believed it landed and cut the chute. Except it really wasn't even close to ground.
Using a bit of hyperbole, there is formal verification in Academia on the one side, and people forgetting to design robust logic while trying to figure out pointers on a microcontroller that can only be programmed in C on the other.
Static Python. That's my proposal to save the next mission :)
In fact you seem to defeat your own argument when you state:
>A lot of the failures in space that were caused by software weren't actually programming errors...
Indeed this is true. I work in aerospace (though not in a software development role) and I can tell you a lot of shops are using C/C++ for embedded applications with strict style guides to attempt to mitigate the flaws in those languages.
I think the fact that many aerospace-related problems aren't due to programming errors says that current practices are working.
and your dismissal of the power of type checking systems seems strange. seems weird to say that it doesn't really help when python has nothing of the sort. even a "static python" wouldn't reach the power of some existing systems. even for dynamic languages, python is behind the times for things like that. for example, look at racket versus python.
and then there's languages like idris, which go well beyond what even haskell can do. something like idris would be wonderful for critical embedded systems. and if you really want flexibility and a dynamic language, python seems well below lisp/scheme. i would love to see racket and ML-based languages to move towards embedded development over something like python.
finally, both lisps/schemes and ML-based languages are very readable. many would argue they are more readable than a language like python. so your implication that type systems incur a readability penalty isn't generally accurate.
> But what I believe a language can do for you is help you express your thoughts in such an simple, structured and readable way that the cognitive load is reduced to a minimum, freeing up thought-power to check your logic.
basically what i am getting at is that a language like racket or idris fulfill this requirement very strongly, even moreso than python. and additionally, they come with advanced language features that no version of python will ever likely touch on.
Python's a very dynamic language; a lot of things are inherently determined at run-time. You could build a less-dynamic language (and enforce type annotations and all that), but then that's hard to call "Python", right? At least we'd be free of things like this:
#Scans the module for methods starting with "do_", and puts their names in a list available_actions = [ method[3:].lower() for method in dir(this_module) if callable(getattr(this_module, method)) and method[:3] == 'do_' ]
My biggest beefs in this space are (not limited to Python):
1) So, far, every scripting language I have used is far too resource hungry for deployment (2-4K RAM/16K Flash or less).
2) No scripting language gives me good concurrency primitives out of the box.
3) Very few embedded languages are familiar enough to coders that they won't scream if they have to code in it. Forth is right out. Lua is a loser because of 1-based indexing (sorry, your environment is C--it's unfair, but I'm not going to bend my brain to undo 0-based indexing habits for anybody nowadays). Lisp and Tcl are marginal but are probably acceptable nowadays since enough people have experience (and Tcl is still one of the standard EE tool embedded languages).
4) No embedded language I know lets me connect an RS-232 terminal on the embedded device, inspect registers and call some functions, write some code, and then compile it to bytecode in flash when I'm done. I would kill for this in order to inspect state and find Heinsenbugs in the field.
Julia(2012) appears to be the only language created in the last 20 years that is 1-based. The next one is Lua in 1993(!).
For better or worse, at this point, the indexing debate is over and Lua is sadly on the wrong side.
* First stage initialization
* Interrupt prologue/epilogue
* Bitbanging where you want deterministic cycle counts
* For using the FIQ for very high priority interrupts. It has it's own registers partially banked, so if you stay in r8-r13 you don't have to save and restore state at all.
So we don't use assembly for magic go faster juice, but instead when there's a coding constraint that we can't easily explain to the compiler.
We use it for different things depending on the board, but my favorite is a really sweet profiler that's integrated with our watchdog logic that'll give us ~1000 PC+SP samples in the 100msec or so before our watchdog timer resets our board. That's pretty invaluable for debugging.
[0] https://github.com/micropython/micropython/issues/658
[1] https://github.com/micropython/micropython/issues/658#issuec...
https://community.zerynth.com/discussion/382/zerynth-vs-micr...
There's something satisfying about toggling an LED directly from the REPL.
You can also inline assembler:https://docs.micropython.org/en/latest/pyboard/pyboard/tutor...
... I'm just speculating. I know nothing about embedded systems
I wonder why everyone in this thread thinks that Cython works with CPython-incompatible implementations (like MicroPython or Zerynth)?
If you're talking about machine performances, the opposite, even for a dedicated dialect like micropython. The benefits are in safety, readability and ease of development.
I'm pretty curious because I'm pretty fluent in Python and I have been learning C++ for more lower level stuff mainly for speed reasons. I still program most of uC through C++ so I'm curious if I should learn micropython if it provides substantial benefits.
Python is a GC'd (reference-counted in CPython and probably µpython) and memory-safe language so it should be less prone to memory corruption, although it is more likely to open you up to resource exhaustion.
> In terms of safety, is there better memory allocation and hardware control through Python?
These are not what is usually understood by "safety" in programming.
> I would imagine micropython would be more inefficient
Definitely, the overhead of micropython would be substantial (even more so as it can't leverage compiler optimisations) so it's a tradeoff between hardware requirements and ease of development/time to market. IIRC micropython also supports only a relatively small subset of all embedded systems.
See http://docs.micropython.org/en/v1.8.6/pyboard/reference/spee..., http://docs.micropython.org/en/v1.8.6/pyboard/reference/cons...
> As for the ease of development, wouldn't C/++ be better?
Generally speaking python is more expressive and more readable than C++ code, which makes it easier to write (develop) Python. C++ is more finicky, thus harder to develop.
In many embedded and RTOS contexts, safety and allocation and hardware control are very difficult to distinguish.
> > As for the ease of development, wouldn't C/++ be better? > Generally speaking python is more expressive and more readable than C++ code, which makes it easier to write (develop) Python. C++ is more finicky, thus harder to develop.
C++ is a huge pain with loose type casting behavior (e.g. is this an long or does this expression convert it to an unsigned long?). But then again Python has very _loose_ descriptions of int and float types at all. For me that makes doing bit manipulation for talking to peripherals more opaque rather than less so.
Not at all. Python has unbounded-size integers and doubles. There's nothing loose about it.
No, but you can rapidly develop applications in Python and when performance becomes an issue, you just rewrite certain modules in C/C++/Rust/Go/etc.
There is a module called ctypes [0] in the Python standard lib that lets you import modules written in C.
In any case, CPython has no business on embedded (non-Linux) systems.
[1]: http://www.edn.com/electronics-blogs/embedded-basics/4440447...
[1] https://luarocks.org/m/root [2] https://pypi.python.org/pypi
What you need is things like [1] or maybe [2]. Lua has them because it's been the dynamic language of choice for this domain for years (with Lisps and Forths as its main competitors).
Now we see new challengers like micropython, mruby and jerryscript / iotjs, it will be interesting, but they're clearly not the mainstream choices for embedded.
Given that cheap embedded systems are so powerful now it makes sense.
We start with Intro to C, then move on to CS1 and CS2 which both deal almost exclusively with C and dig pretty deep into it. We have an OOP class that is Java-based, and an operating system class that is C-based.
So yes, C is still taught.
https://foxdeploy.com/2017/07/19/buildyourowniotmonitoringto...
Search for "via a driver written in Python!".
The author tried to do an IoT setup for C#.
If you searched properly, you would have found the remark I've written, where the author explains that the chinese OEM of a touch screen that failed to work with his setup,told him that the respective device driver was written in Python.
(I'm a huge python fan, but not this way...)
I have developed embedded systems in the context of a real business manufacturing and supporting these products using everything from assembly language, through Forth, C, C++ and, yes, Python.
There was always a good reason to make an choice at the time. For example, an embedded system running a sophisticated control panel with dozens of knobs and buttons was done entirely in assembly language eons ago. It didn't take long for that to become a very bad idea. It was very difficult to maintain, improve and evolve the system. We ultimately decided to bite the bullet and port the code to C. The difference in the ability to express ideas was massive.
The only two languages I would avoid today are assembly language and Forth. The first because it is a nightmare to support, maintain and evolve outside of trivial systems. The second because it is impossible to find good Forth programmers these days and the ecosystem (libraries, etc.) is nonexistent. That said, I still think everyone should learn both.
I have yet to see an embedded system that can't be done well (and by that I mean the entire life cycle of a product) with plain-old C. Nothing wrong with the language. No magic. It works and it can produce clean, reliable fast code.
C++. Well. If you came up through assembler and Forth you learn to despise waste. And that's what forcing an object oriented approach tends to be when there is, in the end, no real need to take this route. Sometimes I see OO as a tool lazy programmers reach for. Some of the funniest code I've seen are things like erecting a complex OO solution to implement a state machine when a simple lookup table and minimal procedural code could do exactly the same, run faster and with less resources. I also do hardware in FPGA's so that's a formula for being keenly aware of resources and not wasting them.
As for Python. Well, frankly, it's about the libraries and nothing else. If you need to do an advanced embedded system with, for example, vision, you almost can't beat the Python ecosystem. Not because of the language. No. I couldn't care less. I care about time to market and the Python ecosystem offers a cornucopia of solutions covering just about any field one might care to name. You can hire Python programmers and anyone who knows C-like languages can learn it fast. That's important.
In the end we build products and all that matters is to build said products as cost-effectively as possible and the ability to support, maintain and evolve them just as effectively.
I like wrapping an object around a hardware resource, so that the only way it can be modified is a call that went through a public member of that object. You can enforce consistency that way.
And C++ has the philosophy that "you don't pay for what you don't use". (Of course, if you use what you don't need, you can still cause waste...)
If I write a function called "set_serial_baud_rate()" and another called "set_motor_rpm()" it would be alarming if someone tried to use it to set the set_motor_rpm() to change the serial port's baud rate.
I understand encapsulation very well but it simply isn't necessary in a lot of embedded cases. I can't remember a single instance in thirty years of product development when we had an issue in development because code was NOT object oriented. Product development never slowed down due to not using OO. More bugs were not introduced due to not using OO. Code was just as maintainable and extensible using C or C++. It truly does not matter at all as far as I am concerned.
What I look for is "done for you" code. Libraries that can produce power-of-two or order-of-magnitude gains in developing a product. Show me gains of that magnitude and I'll code in COBOL if necessary. I truly don't care about languages any more. Just like I don't care about code editors (counting milliseconds in vim). Give me a decent IDE like JetBrains products and life is good.
I know this puts me at odds with most of what they teach today. Kids come out of school and automatically reach for objects for even the simplest of problems. OO has it's place, of course, but it should not be the first tool one reaches for unless it is well justified.
A CPU couldn't care less if you wrote the code in machine language or Python. The facilities provided by these languages are for the programmer, not for the CPU.
One of my favorite interview techniques: Ask someone to solve a problem. If they are on the younger side they'll immediately define a class, members, properties, maybe multiple classes. When they are done, ask them to do the same in procedural form. I've seen people get completely tangled-up by this simple request.
EDIT: To be clear, I am NOT saying OO doesn't have it's place. Of course it does. I've written a bunch of iOS apps in Objective-C with a good mix between OO and procedural when justified. For example, a genetic solver done in procedural form on iOS is miles faster than the same solver implemented in OO. A lot of it has to do with using layers of data objects rather than the raw data types typically used in C. In other words, there's a cost associated with using an object to store and manipulate what would otherwise be a simple array. You might be surprised to see just how dramatic the performance improvement can be.
Now, you're still right, because in C, you just make set_immediate_rpm() static, so that nobody can call it from outside the file, and then the non-static functions become the public API.
In this context the "nobody" in your scenario is the only person on the project. Which, once again, brings us to the idea that it isn't the language or procedural vs. OO at the root of the problem at all if things are done wrong or misused.
I must also point out that hypothetical examples are rarely meant to cover all possible scenarios, much less corner cases. They are simplified examples to facilitate an explanation. As such it is always possible to find holes in the hypothetical.
A colleague taught me a very important lesson about twenty-five years ago. He was about ten years older than me and had far more experience. I had just finished coding an electronics component management system for our manufacturing department where we worked. It was really slick and fast at a time when text-only apps and menus reigned. I was proud that my six month effort had produced such a product.
I was showing it off. When it was his turn to be dazzled he let me do my "look at this" tour of the software and said virtually nothing. When I was done he said "looks good" and proceeded to plant his palm on the keyboard and run it back and forth about four times, pressing as many keys as he could at random. The program crashed and he walked away.
It took me about a month to make it robust enough to have it react gracefully to such an attack. Learned a lot in the process about dealing with the real world. This gave me the mindset to write code defensively.
A function such as "set_immediate_rpm()" should be bulletproof, and this has nothing to do with OO vs. procedural. A good implementation of "set_immediate_rpm()" would have a rate-of-change evaluation such that you could call it a million times and yet not violate the maximum desired rate of change (presumably set through a different function or defined as a constant somewhere).