Nerves – Craft and deploy bulletproof embedded software in Elixir
nerves-project.org
nerves-project.org
The keynote by the Nerves creator Justin Schneck was particularly good.
When the job calls for a heavy linux-running processor, this seems like it could be interesting, as long as I can dig out all the details of what it's actually doing. The chips I use usually have an errata document 20+ pages long, and we usually find some new bugs to add to it during development. It's hard to code around hardware bugs period, and that's in a language hardware guys know, like C or assembly. Debugging hardware code written in Erlang sounds... challenging.
ARM processors are so cheap these days that I think your idea of “embedded” is a bit stagnant or too focused in one field. I’ve built and shipped a product where it had an ARM processor and ample flash (among a bunch of other components) packaged the size of a key fob and the BOM cost is way less than I used to pay for Arduino or other micro controllers.
Rust is kind of interesting in this space, but a little bleeding edge, will likely look at adopting it later.
Ended up picking an Cortex-M3 and using FreeRTOS. With a C / Lua mix. Not because I particularly prefer them, but it requires the least compromises. Also writing robust embedded C code is pretty well understood with many common patterns. Still, it requires being more diligent and disciplined than many other languages.
one would hope so, arduino is a retail packing of other chips into a kit.
You'll always see the crunch at high volume it's just small to medium volume where your options open up. Back when I was in the smartphone business we'd easily see 10M+ volume on a single device and the associated BoM was tuned to a tenth of a penny.
That's one thing for creating a one off or prototype, but when a company creates millions of devices, the BoM becomes more expensive than developing a C version of the code to run in <1MB flash.
Depending on which part of the hardware code you need to debug, Erlang could make it easier; once you get the erlang shell running, you have hot code loading, some pretty extensive debug hooks, and nice bitmatching syntax. If the hardware code you need to debug is in the chain of getting to an erlang shell (getting the system running to the point where you can have a serial terminal and boot an OS), I don't think Erlang makes much of a difference -- projects big enough to run Nerves are likely going to run a full OS kernel, and Nerves is using a Linux kernel, once you can boot a Linux kernel, you've probably done a lot of the hard work.
My recommendation is to not do database reads and writes willy nilly all over the place, but to actually send sensible messages to a real server process, that does the interfacing with the database. Sometimes, it's useful to cheat and read ets directly, but it's really nice for writes to go through a server process, because the server process's mailbox provides an ordering for your writes, and you get to skip all the complexity of overlapping writes. In an mnesia system, that means writes should go to only one of the peers. (If you can divy up the writes to different servers based on the keys or something, then you can also divy up the servers onto different nodes).
All that said, distributed systems give rise to emergent behavior, but that's true if you program them in erlang, c, or assembly; it's just a little easier to get to crazy big systems in erlang. :)
Our main difference is as you say, that we combine Erlang with RTEMS to make the Erlang VM essentially be the kernel (no Linux running underneath). This let's us use Erlang's inherent soft real-time properties more reliably and allow us to do more hard real-time things.
Our low-level drivers for accessing hardware are written in C, and high level drivers are written in Erlang (for development speed, readability, testability, line count etc.) but can of course be written in C if you need the performance.
I'd be happy to take questions about the project.
Right now only the GRiSP board is officially supported, although the project is made in such a way that new platforms should be easy to add.
We're deliberately focusing on smaller platforms than the ones you mentioned. If a small Linux kernel can fit together with Erlang, you don't really need the RTEMS combo as much. That being said, we're happy to receive any contributions for other platforms, even larger ones!
We don’t use the MMU so we don’t need it, some embedded chips have even small MMUs that allows it running Linux size OS but since they have very small translation lookaside buffers it needs to traverse the in RAM page table very often and performance is quite poor then
If I were writing code to drop onto a small device, I’d want the capability to efficiently have a lot of things run at the same time with a small processor, to know that a single heavy unit of work wouldn’t negatively affect the responsiveness of everything else and to know that all of those tiny pieces were built to basically never go down.
Personally speaking, it would be difficult to imagine using anything else for that type of work.
However, for me the best feature is that I can remotely log into a running BEAM VM and interactively explore the live system [1]. Since Erlang has been used in "embedded" type systems for a long time there are a lot of useful libraries. For example a tiny bit of wrapper code let me setup a secure remote Elixir REPL using standard SSH keys for our embedded devices [2]. You can also run entire "applications" much as you would a system service, but which you can communicate with natively in Elixir/Erlang. Also the support for running sub-processes as ports is really nice.
In general it's really more like a live operating system which you can program and investigate. The primary thing lacking is a capabilities system to allow running non-privileged code in a sand-boxed manner.
1: https://tkowal.wordpress.com/2016/04/23/observer-in-erlangel... 2: https://github.com/elcritch/iex_ssh_shell
1: https://en.m.wikipedia.org/wiki/Triple_modular_redundancy
1. It has a good system to hot load the code onto the pi and immediately start running your new code making deploys more efficient than other compiled languages.
2. Erlang uses processes which are very resilient and a good fit for projects that require constant streams of data & lots of connections which can be relevant in IOT. Basically the jobs restart themselves in a robust fashion.
3. Erlang runs on beam which is a well established VM that has years of development behind it making it a robust core for your project compared to some other operating systems that you may load onto your RPI. I don't think this is the biggest reason, but it is one.
The hot reloading & concurrency model are my favorites & elixir is just fun to build in as well. I'm already using it for web so to bring it into IOT makes it natural for a web/IOT mixed project.
- How's the supply-chain side of things? For a while, vendors would only sell one Pi at a time, if they had stock at all. Have you had any problems with that?
- Why do your customers know or care that there's a Pi inside? It seems like an implementation detail that only a select few would ever think to ask, unless you're telling them. "COTS ARM Cortex A53" would likely be enough to make most people's eyes glaze over before digging enough to discover that there's a "hobbyist" board inside.
Edit: I looked at your website and get it :). You're selling hosting, not pre-packaged devices. That's really interesting!
How good are the Lua bindings for OpenGL ES? Are they as nice as Löve? I tried https://www.mztn.org/rpi/rpi_ljes.html once, but it was a bit fiddly.
Coroutines are also amazing for sequenced AI routes.
Anyway, thanks for the trip down memory lane.
Not sure. Of course, the BEAM VM is robust and good for the things it's good at, but I don't think it's much more "robust" than, say, Linux, which seems pretty good, stable and robust these days. I'm not talking about software that runs on Linux; though the coreutils etc are pretty stable at this point (understatement). Linux is a fine platform to run your project on, and I claim it's no less robust than BEAM is.
Nothing against your other points though.
Besides, why are you comparing BEAM to a full-fledged OS kernel?
Edit: sorry, i see you're not the GP commenter.
How is Linux fault tolerant. Say a kernel driver starts overwriting kernel memory how do the rest of the kernel subsystems isolate that fault and keep going without crashing?
And what happens when a core module that is part of BEAM starts overwriting critical BEAM VM data structures?
A more apt comparison would be software running on top of BEAM vs. a userspace process running on top of Linux.
Agreed.
> And what happens when a core module that is part of BEAM starts overwriting critical BEAM VM data structures?
Segfaults and other terrible things.
> A more apt comparison would be software running on top of BEAM vs. a userspace process running on top of Linux.
That's a better analogy of course. I've heard of BEAM VM described as an "OS for application code". Nobody would want to put their latest and greatest crown jewel production code on a Windows 3.1 platform, where one segfault in the calculator process takes down the word processor, but that is essentially what is happening when using shared memory and concurrency units (threads, goroutines, co-routines, green threads etc).
In fact in high integrity certified deployments, it is another actually robust and fault tolerant kernel running Linux kernel as yet another user process.
Kernels like INTEGRITY are robust and fault tolerant by design.
It is much more robust than Linux. If Linux kernel has a segfault and crashes, it takes everything with it when it panics. If one of the million Erlang processes which has an isolated heap crashes, it can probably safely restart (maybe with a few other ones it is linked with).
But I do suppose that Linux has a higher chance of having a kernel panic than BEAM having an internal segmentation fault, simply since Linux has more code and thus a larger surface area for bugs.
To extend the analogy: running with threads that share memory between themselves is a bit like running on a Windows 3.1 machine - were if the calculator is broken it can crash and overwrite the memory used by the word processor
BEAM is a beast when it's in its own runtime managing concurrency, but the bytecode interpreted language it implements is very "meh" when it comes to performance.
(Since people seem to take these sorts of assessments personally, full disclosure and cards on the table: I massively prefer the BEAM world to the Node world. Nevertheless, the facts are what they are. The interpreter for BEAM is just not where you go for performance.)
There is definitely a cost to using the dirty scheduler, and if your NIFs don't need it, you're going to be paying the overhead for nothing. But obviously there's a plethora of uses for them when integrating with libraries that don't play nice with chunked work.
What do you use BEAM for? Is rock solid supervision really its biggest boon?
And if that's the case, how does it compete with the proliferation of easy to use tech like Kubernetes that, more or less, solve the supervision problem in a simpler, easier to scale, and more abstractable way?
There are parts of Elixir I really quite enjoy, but I've long felt that BEAM is holding it back as much as it benefits it, similar to how the JVM is both a boon and a "modern curse" to Java. The 90s dream of having these VMs executing platform independent bytecode seems dated in the face of infinitely customizable VMs on cloud hardware. And process supervision at the application level also seems dated in the face of modern devops HTTP-level liveness and containerization.
Honestly, I'd say it isn't a good fit for any particular reason. jbhatab isn't wrong, but there's other environments that will have other advantages over Elixir/Erlang/BEAM, depending on what you want to do.
What is the case here is that the Raspberry Pi is basically a little computer and can run pretty much anything, Erlang included. Erlang was built in a world where a Raspberry Pi's specs would have been mindblowing, so it works just fine, just like anything else first built in the 1990s.
Also, as other commenters have pointed out, embedded doesn't mean what it used to. Embedded now can range from a custom real time operating system written in embedded C with memory allocations from a static pool to Java running on an Arm processor.
I too met the Nerves dude, very nice and approachable. I was proud to have such a project come out of my area.