Embedded Rust in Production?
blog.lohr.dev
blog.lohr.dev
I recently finished a contract at a (very large game dev) company where some tools were written in Rust. The tools were a re-write of python scripts and added no new functionality but were slightly faster in Rust.
The reality was that these tools were unmaintainable by the rest of the company. Only the author "knew" Rust and it was hard to justify a new hire Rust developer to maintain this small set of tools.
The only reason these tools were written in Rust was because the dev wanted to learn Rust (a big but common mistake). I pointed out to the Technical Director that this was a big mistake and the teams had taken on a large amount of technical debt for no reason other than the ego of the wanna-be-rust-developer. Since I "knew" Rust he wanted me to maintain it. My advice was to go back to the Python scripts and I left.
Many such cases. Recipe for disaster.
The thing is that this is not a developer's decision, this is a management decision. The developer might pitch for Rust or even start developing in Rust but if that is not right for the org then their manager should say 'no'.
So ultimately this is a red flag about management.
Needing the managers permission is more of a "wartime" mode.
For peacetime the management should set a culture of peer review and data and logic driven discussion. The culture would ideally have a Rust lover sit there and think "would love to do this in Rust but...." because they know their peers will be affected by this decision.
The problem with management approval required is it usually meams no or bias towards management pressures. A management veto that is occasionally used may be better.
Disclosure: not a Rust programmer but pretty neutral on it. It is on my radar if I get a problem Node or Go can't solve.
I think the main issue here is having a single dev going for Rust. Ideally someone interested in a new technology would pitch it to the team, and they'd discuss if they want to have a shot a it and if they can justify the investment for the whole team. That's where a tech that doesn't fit would be discussed and rejected, potentially with a proposition for a better option that devs are also interested in.
Even if it starts with just a prototype, if it touches production the other members of the team should be able to review it and maintain if needed.
That's this whole bit missing, and the team looking at it as a thing of life, that makes me think they're not focused on pushing things forward anymore at an org level.
Rust is a complex but overall good language for writing solid software, C++ is making a deal with the devil in exchange for speed.
Typescript, Python, Go, Swift, and even Bash depending on the situation are all quicker to code in than Rust.
If any of those languages are inadequate for the requirements whether they be memory-bound, CPU-bound, or sensitive to gc pauses, Rust is an excellent option that is far superior to C++ in 2024 and beyond.
The notion of "one true language" has always been and will always remain a fool's errand.
Tbh I don't find myself to be "slow" in rust. Sure for quick exploratory stuff the notebook-environment and introspection capabilities of a dynamic language are definitely nice (e.g. when taking apart some unknown JSON data format), but especially when it comes to complex logic and refactorings Rusts type system is really making a big positive impact on productivity.
Sure TypeScripts type system is also powerful, but Rust is consistently better at infering types from closures and function calls, and the existing types and tooling story is miles ahead of TS. (and I've never encountered an easier to setup/use model checking language integration than Kani)
For one-offs where correctness doesn't matter, sure throw $SCRIPTING_LANG at it. But once you want correct software, I'd still choose Rust in a heartbeat.
A good rule of thumb would be: is correctness and code confidence more important than learning curve and compile times? If so Rust, otherwise JS or Python (with possibly parts in Rust).
ie: security is rarely a problem when selling software
That said, obviously all languages were in that state at some point. Twenty-eight years ago you might have said the same thing about Java. Twenty-four years ago, Perl might have looked like a better choice than Python; now it's clearly the opposite. XenServer took a gamble and wrote their main control stack in OCaml in 2008; on the whole that has had some benefits, but the number of OCaml developers has not significantly grown, and it's not easy to hire people.
That said, I think Rust is much more likely to follow Java's trajectory than OCaml's: My prediction is that it's only going to be easier and easier to find Rust developers.
If there's a killer open product out there where the primary supported language is Rust(Linux kernel might as well be it someday), or if a big company launches a successful product in Rust then adoption may accelerate. For now its mostly limited to cryptocurrencies, performance critical paths or rewrite of hidden core components but nothing flashy.
"Nobody ever gets fired for buying IBM". Unless teams can say the same about their choice of Rust, people are going to be risk averse and prefer using safest choice as long as its good enough.
Memory corruption bugs are no joke. If you've got staff that already knows C++, their ramp up time for Rust will be much shorter compared to typical devs working only in Python, JS, etc.
For the latter it’s still kinda of "everybody can write some code that will run everywhere". The former is more like "you can have performance, consistency and correctness in a shallowly-C-like syntax", which is more likely to tease the average HN reader, but probably not the average programmer who is in to earn some descent money or as a way to start before moving up in the businesses hierarchy.
And you see there are some people which enthusiast to push Rust into the Linux kernel, while I doubt that in Java projects of similar size there is that much ardor for even introducing it in some part of it, let alone propose a full rewrite.
I don't think the origin lame was that it would compete for the same segment, but that it would reach the same level of ubiquity. It could do that by opening up a new non-existent segment, eg smart contracts on the block chain. Or it could take a little from multiple domains.
When you want a decent paycheck, but instead get a descent into computing madness.
I'd rather compare Rust to Haskell and Kubernetes.
Haskell:
- Dozens of handfuls of experts open to remote work.
- Lots of aspiring juniors who see no way to enter the market.
- Most employers are R&D-heavy academic consulting environments.
Kubernetes:
- Wildly hyped and cherished among people who don't have the complexity to warrant the purchase.
- Eventually criticised for being overly complex and suitable only at a certain size and setup.
I imagine it's never been easier to find Haskell developers and Kubernetes developers, either.
Yet, one's wish to code Haskell professionally does not necessarily align with one's market value.
It seems like there's a lot more open for Kubernetes, so Rust could go that way.
Unlike Haskell, Rust seems to gain industry acceptance.
Had both of these languages' ecosystems and communities not dragged their feet and go full elitist on newcomers then Rust might not have ever been created.
So sure, Haskell and OCaml are great but after I spent half an afternoon trying to bring in a dependency in a little-more-than-hello-world Haskell program and then spending the other half on trying to do the same in OCaml I just threw my hands in the air and said "frak this, there has to be a better way".
Hence I landed on Rust. You and others can always say "skill issue!" but that's hugely missing the point; I want to use my skills on solving the actual problem, not on the logistics. We're not in the 1970s anymore and a weirdly huge number of people still haven't gotten that message.
That's certainly true of the last 10 years of Ocaml tooling evolution.
If we both start from that point then I think we might find a common ground.
For instance, coming from C Unix programming, it took me 2 hours to try OCaml 15 years ago or so, because Ocaml tooling, at that time, was the same as C. Basically, I made a generic rule for my makefiles to call ocamlopt instead of gcc and I was good to go. Installing a dependency was just the usual "./configure && make install"
This was simple for me, but not for students with no Unix background. In that context "opam install" was simpler, although it's objectively more complex as it adds a few additional layers on top of the same process.
I moved away from makefiles long ago and I don't regret it, for a multitude of reasons that are well explained in many places.
`cargo add ...` is great and I wish every language offered a tool like that.
I use `just` for project tasks, but there are many others.
You mean: I still need make
For example, build.rs replaces some uses of a makefile.
> build everything that's not just code
Have you heard of: Just, Nix, Bazel, Helm, Terraform?
Depending on the situation, there are many things you can build that aren't strictly code where you can use a bunch of other tools than make.
I came for the job opportunities.
I stayed for the tooling.
> Haskell and OCaml are great but after I spent half an afternoon trying to bring in a dependency in a little-more-than-hello-world Haskell program
I had a similar experience trying to make Haskell's LSP work in test files. After ten years of using Haskell for academic and hobby projects, I still couldn't make the red squigglies go away in test files. With Rust I struggle similarly when dealing with cross-compilation and a complex set of feature flags on Nix. But at least this is not "Hello World!" failing.
Sure, not the same usage patterns, I get that of course, and I love the idea of Nix but they could have went about it very differently with a much better UX / DX... and they refused.
Side topic though, sorry for the digression. Main point was that if you use Rust normally without any weird and not fully accounted for wrappers then everything works fine. I just fired Neovim in a personal project I haven't touched in a year -- 20-30 seconds of LSP compiling stuff and I saw my project-wide compiler warnings immediately after.
The python scripts of the same vintage have to be reworked because python versions and importantly supporting libraries have changed.
This is mainly because Perl hasn’t changed, with the failure of perl6 to launch. But it’s an interesting comparison.
I’ll agree Rust will be like like Java.
I don't argue with the portability and longevity of Perl but it too has its problems.
Go is in many ways leap years ahead of most AOT languages I've used in anger. Yes, sometimes you miss XYZ, but seldom is that an issue outside of personal preferences.
What you do get though is amazing tooling and little-to-no downtime. Builds basically don't take any time, tests run past, pipelines are quick.
go test ./... finishes faster than you can print Determining projects to restore... to the screen. It's a very liberating experience that is hard to go back from.
I have my very real pet peeves with Golang -- lack of sum types and thus no exhaustive pattern matching is the top one -- but I can't argue with results. Every time I wanted to whip up something quickly, Golang has gotten the job done and then some.
Unfortunately it does nothing in terms of exhaustiveness, but it effectively gives you sum types which IME are so, so much easier to work with than state pattern or other inject-an-object-abstractions. Not having to design an API (that later leaks) is rewarding enough.
It's strangely efficient as well. I've experimented with implementing sum types via tagged C-style unions and at every (ad hoc, micro) benchmark I've tried them against, type switching on an interface comes out on top.
Maybe Python, but it doesn't compile to a static bin.
I'm not talking primarily about Perl as a technology in and of itself, but as a long-term technology choice in terms of being able to find people to maintain and improve it.
The Xen Project has a bunch of support stuff (automated testing and tooling around doing security work) written Perl. It was just about a reasonable decision when it started to be written 10-15 years ago; Golang was too young, Rust wasn't written yet, and Python had its downsides. But when the author of the code left the project 2-3 years ago, and there was 1) nobody with a similar level of expertise to maintain it, 2) nobody particularly wanted to gain that expertise, and 3) it wasn't easy to find someone else to pay to do maintenance.
If you've managed to find people who can maintain those perl scripts, more power to you; but it unless perl5 can get some momentum back, it seems like it's going to be harder and harder to find replacements.
Rust is a different beast, much more complex from the outset, and not trivial to learn (as Java was). It will grow if that's where the best jobs are, but I don't think it is going to grow in the way Java did, because the difficulty and dynamics are different. There was a good stretch at the beginning where Java was both very helpful on your resume and really easy to learn.
I don’t think it’s reckless to have e.g. Haskell or OCaml as your go-to language for a decade, and switch it out with Rust.
The typical answer is "The jobs market will provide".
The issue with that method is that you end up competing with everyone and, usually, your company is not the top employer.
OTOH, some companies decide to use technology and invest in knowing the community, by hosting events or meetups, by frequently sending their devs to conferences on company time, etc. Usually, the people hired are above average, and are also able to train juniors if the market dries up. These companies typically are the safest in terms of ability to maintain their stack, but have to account for the fact that alienating their existing devs becomes a big risk.
The real harm comes for companies that choose niche tooling but believe the market will provide.
So XenServer embracing OCaml is nothing like Random Gamdev writing some scripts in Rust without the team deciding to embrace the change.
Rust may have an advantage in certain places (no segfaults is a definite win). But replacing working Python scripts is almost certainly not that place. Also, for a C++ developer, Rust isn't as easy to learn as Java.
Rust may get there. But don't use Java's success to predict Rust's trajectory.
This looks like pipe dream of those Rust devs who usually claim that Rust borrow checker is easiest thing since heating HotPocket in microwave.
Viability of a language is certainly a consideration, but the characteristics of the language also matter.
What you couldn't really predict about Java in, say, 2000, was how much the complexity of its ecosystem would grow. In 2000 (or 2005) it was fairly easy to move in and out of Java from project to project. That is not true today. My fear on the Rust front is that it's going to be another C++, where the complexity is considerable and requires more or less full-time expertise.
You can certainly make a case for that being inevitable in some ways:
Rust can be written in many ways to address different levels of abstraction, and when you stack those, you can require the reader to know about all of them.
If someone gives me a piece of Python code, it can only be hard to understand if the domain is hard (e.g physics) or it is poorly written.
In that way, Rust belongs with C++ and Haskell. The abstractions that Rust will introduce over time won’t grow in all directions like with C++ and Haskell (e.g. you won’t get higher-kinded types or GADTs in the next ten years).
The second one was Sun's Distributed Objects Everywhere project being canceled, and the Objective-C code being reborn into the first J2EE proposal.
There was nothing else like that for business software other than Smalltalk and later .NET, however most Java vendors were ex-Smalltalkers, and in 2001 .NET was a no-go for anyone that would care about supporting UNIX deployments as well.
In 2001, we were moving away from scripting + write extensions into C, into Java, and only pivoted into .NET because being a MSFT partner with access to the "partner eyes only" betas from Microsft before .NET was announced, additionally there was some ideas to move away from UNIX customers.
However by 2006, while at Nokia Networks, a strong UNIX shop back then, the large majority of their applications were going into the Rewrite into Java phase.
I would decide this on a case by case basis tho, Rust is different, but if e.g. what you wrote derives most of its business logic from its configuration, or the usecase is so generic changes aren't gonna be a thing for a while, why not. Rust is different, but unless you want a complete rewrite or new complex features any developer should be able to get into it if you give them a little time.
And the strictness of Rust makes it unlikely that they break things in a bad way, which is good.
there’s also the fairly easy cross compilation to bear in mind.
i built a container entrypoint binary via python to stick in all our containers at last job.
getting that to cross compile was going to be a bunch of pain i wasn’t willing to go down for a simple “download files from S3, run a sub process, upload files to S3” pre-execution wrapper.
digital ocean infra + self hosted gitlab —> no readily available windows instances.
so everyone had to use linux to dev these containers with wine+mono.
being able to cross compile would have been brilliant here. i made the decision to not do this in rust for future maintenance reasons. but then people avoided using this thing because they couldn’t run on windows :/
(i know there’s that paid for package but the license fee is mind boggling from what i remember).
unfortunately the same maintenance issue applied -- it was a python / R shop so no-one else would have been able to change the code once i left.
This is all it takes to build a linux arm64 binary:
env GOOS=linux GOARCH=arm64 go build your_program
This is how you list all possible targets:
go tool dist list
#!/usr/bin/bash
archs=(amd64 arm64 ppc64le ppc64 s390x)
for arch in ${archs[@]}
do
env GOOS=linux GOARCH=${arch} go build -o your_program_${arch}
donelooks on par with rust then. go is the next one for me to dive into when im doing fiddling with rust audio stuff.
regardless, wouldn’t have worked at last job as no-one knew go either. would have been easier probably to get someone in, but i spent a year trying to get one new hire to no avail, so doubtful.
I can be confident to teach any average developer about Go without any prior experience relatively quickly, I'm not so confident I could do the same with Rust.
C++ is better than it used to be, but it was truly atrocious in that regard for decades. It was simply the least worst option for those decades.
I do not consider it the least worst option anymore.
Unlike Go that only really requires experience in general programming to become rapidly productive, Rust basically requires that you read The Book in its entirety before you even think about applying that knowledge in production. There are just to many interdependent concepts to hand wave away the prep time. It all (mostly) makes sense once you've familiarized yourself with the basic concepts, but it's a training step you simply cannot skip without inflicting a fair amount of pain on yourself and others.
Don't get me wrong, Rust has a niche where it's the right choice. But being a popular language of the day, it's getting used a lot in the wrong places.
Web stuff is about developer speed. So familiarity, libraries, and tooling. There are plenty of good options.
Anything that needs to be performant can go in it's own service.
Then everyone will know, numerically, whether Python is worth it. Maybe it is!
Python, Ruby, Go, PHP, C#, Java, TypeScript, Elixir.
Performance benchmarks when writing APIs as 1st concern? Really? 99% of the cases this should be the last criteria.
Monthly reminder that a good part of instagram still runs on Django/Python.
There are two dialects of Rust. One can be written immediately by beginners, the other is for experts.
I make sure to write scripts in the first dialect. It uses `clone` all over the place, does not make use of any async runtime, and makes extensive use of "anyhow" for error creation and logging.
I advise all my peers to do the same, if they find a need for Rust in tiny scripts or applications.
I believe it's more like a spectrum.
I'm 3 years into my Rust career.
I started with .clone() everywhere.
Then I moved on to write highly async code with tokio and std.
Now I'm somewhere entirely unfamiliar, writing highly portable no-std Rust for bare-metal and wasm.
It takes some time to know the difference between std and core.
It takes some effort to write modular, allocation-free code (static allocations only).
It takes some rediscovery of what crates will work for you.
It's a steep curve, and I don't imagine everyone likes it.
I can believe this, but it's funny because I work on a project where the main product is written in Rust, and it's the build tools written in Python that are the technical debt that nobody really wants to touch! In our case I suspect rewriting them in Rust will end up being the right thing to do (if/when we can justify the switching costs).
Oh hi! You must be my teammate!
I joined a team like this and just started a personal build/dev/ops tool repo in Rust for things that I needed but were not automated. Over time it has become an "official" repo and renamed to be blessed with an official "org" package prefix.
When I started this project and people were interested, I was asked why I didn't just add to the Python repo; I said that others are free to rewrite them if they want to, but I can't justify spending days messing around with Python to achieve what I can in less than 30 minutes with Rust when I need to quickly automate a tedious task.
Edit: A lot of downvotes on this, but in case it wasn't clear - the team is happy that someone took the time to scaffold a repo in Rust that everyone feels more confident in contributing to, since the overwhelming majority of the codebase is written in Rust and the Python tools repo is full of annoying runtime errors that could be trivially caught at compile time.
The end result is more people feeling more empowered to automate repetitive and tedious tasks which can easily be fudged when executed manually instead of suffering with them because someone who isn't even here anymore decided to start a tools repo in Python for a team that almost exclusively writes code in Rust.
There is not a lot you can do to make your python code as good in terms of reliability or clarity of intent. I'm sorry to hear that you did not have a good experience with it, but frankly speaking, I cannot look at python code anymore because of the inherent "hidden" work that it creates, variables being passed trough without much care for their exact meaning, hidden "bugs" that make the overall job way more annoying to debug.
Python is like that person that tells you everything is fine but is mostly wrong and won't even tell about that. Whereas Rust will just straight up tell you what's wrong in the draft.
I prefer the honesty and the little additional overhead work to getting lost in debugs for weeks bc some person has inputed the wrong shape for some vals or whatnot.
The peace of mind is saving me a ton on maintenance as well. (project is a 2years on with distributed python pipeline with a bunch of IA processing - switching up to rust was not the easiest thing to do considering some parts had to stay in python because of code ownership being someone's else - the stack is a rust/python stack now with some parts being left out in python depending on the actual needs of update)
I understand very well that some people do not want to code in rust, specifically because they might not have the time or the will to do so. But objectively these people do not have to define the pipeline execution except for the actual order of execution, nor do they need to maintain every part of the code because it's just not their job.
Now if you are to compare the exact benefits of rust vs python for a company, the server pricing will just speak itself and the choice will be made depending on the cost of production, the lifetime of the code, and the actual benefits.
If I was to do some API today, I'd 50000% go for rust and I DO think of python as the `no-coder coder` thing. I initially learned C, switched up to Python for jobs, and the code quality of python is just downright bad. It does not push me in the right direction AT ALL.
The fact that we used a scripting language with no typing for tools that need any kind of guarantee for years does not mean it's the best way to do it.
Also to answer the problem concerning `rust` dev's being non-existant on the market, and no job being open is really simple : even tough rust has 10 years on the developer market on this is really young and need to learn.
You need 3 years of practice to get 3 years of experience with rust. There is no way around that. And there is just not enough experienced people with this language yet to make it a shift. I'm not saying there is none, i'm saying it's not yet democratized.
This is the major barrier. It's a difficult language to learn for people who've worked with a compiled, strictly typed language with manual memory management. That barrier is much higher for people who've only ever used a dynamic, interpreted language.
I'm on-the-fence about orchestration of pipelines with Python since I've experienced a lot of the issues you're talking about, but you can just abstract the Python algorithm parts in to APIs that your pipelines call and separate out those two concerns. Then it's clear who has maintenance responsibility, it separates out the algorithms from data collection + calling, and it doesn't introduce a big barrier next time someone wants to implement something that depends on the latest and greatest algorithm written in Python, which writing in Rust inevitably does.
To me though, I don't really understand the use of Rust in non-systems programming. For a lot of the use cases people talk about, Go seems like a better fit with a lot less of the pain since it's a much simpler language. When teaching people a language I'd much rather do that with Go than Rust, and that matters in an environment where developers who know either are not really that commonly available (at least here in the UK).
This was not what Google and others found when they studied it.
https://www.theregister.com/AMP/2024/03/31/rust_google_c/
> More significant, Bergstrom said, is the comparison of rewrites of C++ code into Rust.
> "In every case we've seen a decrease by more than 2x in the amount of effort required to both build the services in Rust as well as maintain and update those services written in Rust," he said.
You can add typing tools now, though? CI pipeline runs mypy and that gets you a long way.
The Rust codebase I’ve built on run faster, don’t crash, are more correct, and take only marginally longer than the first python version. They’re easier to maintain, they’re easier to upgrade their dependencies, I’m more confident about asserting guarantees from the codebase, and it’s an easier time onboarding people and having them meaningfully contribute. There’s no more death-by-a-thousand-cuts that I experienced with Python codebases. I don’t have to worry about my Rust codebases blowing up at 2am because of some random exception.
I am defs not going back.
Fixed it for you. You wouldn't believe the quality of the average applicant for a basic standard Python position if I told you.
Greasing the wheels of the corporate meatgrinder with Rust is almost impossible, actual work quality notwithstanding.
Huh? There's tons of good developers looking for jobs right now, more so in Rust.
Can I hire 1000 Rust devs at the drop of a hat off the street? Well, no. Have I ever struggled to find competent Rust devs, or high level people who were happy to learn? Nope.
I would absolutely not be afraid to use it because of “hiring concerns”. If your current devs can’t or won’t learn anything new, they’re not very good at their job. If you can’t or won’t hire because you can’t find a super-senior at the drop of a hat, well, your hiring process is broken and no language will really help there.
Case in point from the article:
> it is not viable (for us) to retrain a C developer to Rust
What? Why the hell not?
Not because I don't enjoy learning, but because it's a waste of time. I don't need to learn Rust, C# works fine. I don't need to learn Vue, React works fine. I don't see any value in learning yet another tool to do the same job, I'd much rather spend my time getting actual work done and getting better at using the tools we're already using.
I have had to touch things in languages I'm not familiar with and initially it's slow due to having to look things up, but plenty knowledge still does transfer. Opening a file is still opening a file, updating dependencies is still updating dependencies. Plus python devs should be used to changing tooling all the time ;P
While they should get wider buy-in first then if the choice is technically justified within reason, its perfectly appropriate (and normal) for devs to pick up new languages in this way when existing expertise is not available on staff. No competent dev will accept their skills atrophying due to overly rigid political/tech choices.
That's how skills are built, its always been thus in such environments, and thats ok.
And I say that as an employee.
There's some flexibility to choose tooling but with autonomy comes responsibility.
Replacing a bunch of working Python scripts with Rust is not just irresponsible, it's disrespectful and isolating towards coworkers.
rather different to my
While they should get wider buy-in first then if the choice is technically justified within reason, its perfectly appropriate
> No competent dev will accept their skills atrophying due to overly rigid political/tech choices.
Competent dev to me is one that delivers value. I say that as a dev but I'm pretty sure stakeholders would agree.
It's perfectly reasonable to hone your skills outside of paying hours if you chose to expand to new technology outside of your employer's stack.
I would probably concede that it depends on context - for devs paid at the top of the market to simply churn stuff out factory-style there is less justification for self-improvement on work time.
For the majority though there is always give and take between responsible employees and employers, and building skills as discussed is normal.
If anything, companies expecting employees to know their specific tech stack on day 1 of the job is much less common an expectation outside of Software engineering.
Tons of people would jump ship to be able to use Rust, it has a lot of love in the community.
"Being totally unwilling to accept anyone junior, or who is new to the stack" is a disease in this industry, and it's really apparent when some 200 person+ company is only hiring principal level rust devs.
Some memes just write themselves.
Only If I had a dollar for everytime this happens. Choosing a language for the company is a business decision. we have a similar issue with Elixir and now the entire team is spending a quarter just to get rid of it. For anyone building startup on LLMs , this is one killer application I would pay for. The generated code doesn't event have to 100% correct, just right enough for devs to tweek would go a long way.
The important part is evaluating the experiment and having the courage to say “well that didn’t work out, let’s go back to the original version”.
They should have evaluated what is “slightly faster”, how much money it saves and how much it will cost extra to maintain it.
I don't blame them. If they were able to convince their manager to re-write python scripts in rust, good for them. They got "on the job/real life" rust experience. They solved the "chicken and the egg" problem in rust hiring - for themselves.
when companies can fire engineers when they are in trouble, engineers are bound to "think for themselves"
I see very few job postings, and almost all of them are either cryptocurrencies (I don't want to waste my life on this), or "3 years professional Rust development in production" (disqualifies self-learners).
Given that nowadays most applications are not replied, it makes little sense to spend time even browsing the postings.
Just take a look at https://rustjobs.dev/. Most of them are well paid remote jobs, but they are asking for +3 years of "professional experience" with rust, "with a proven track record of building and deploying production-quality code", and more. Hell, iirc, i saw one asking for proof of contributions to the rust repo (eg, being a core maintainer).
edit: to be fair, i saw one position a while ago asking to be willing to learn rust
Isn’t this a fairly well-known phenomenon though?
- new language makes waves. The people who picked it up early do some impressive stuff.
- early early adopter companies either snap them, or internally make the choice to learn it too. In turn, they do some stuff with it.
- gets a reputation for being the hot new thing. Other companies “want in on it”- they want to be able to do the fancy cool things the other places did, but they don’t have the patience, time or culture to grow it themselves so they aim to hire out seniors and everyone with lots of prior experience. <——- many orgs are here.
- proliferates enough that it’s well and truly mainstream. Also known as the “hire 1000 Java devs and throw bodies at stuff” stage of hiring and availability. Python, Node, Java, PHP, etc are here.
Asking for proof of contribution to the Rust repo is insane though.
- Must have 5 years of Rust and 8 years of embedded experince
or
- Must have 5 years of Rust snd 10 years of writing SQL engines.
or
- Must have 2 years Rust experience and 5 years with Linux kernel development.
etc...
If you are looking to hire someone with specific domain experience, it makes more sense to compromise on the Rust language side of things. A developer can learn Rust on the job, but it is harder to learn to write a production SQL engine on the job. But that means you aren't hiring Rust devs.
Developers who just specialize in Rust are behind their peers who are domain experts when it comes to looking for a job.
IMO that means Rust will "win" not when a bunch of undifferentiated developers learn Rust, but when the domain experts learn Rust. e.g. Kernel maintainers.
It doesn't necessarily. Very few jobs postings that require "N years of experience in language A" actually require that. Most will accept N years of total experience and some smaller amount of experience (not necessarily professional) in language A.
Especially if there are other candidates who do have that much experience in a specific language. Even if you are a better candidate overall (whatever that means), you need to pass the resume screening first.
That's why should always try to avoid recruiters and apply direct if possible!
> Especially if there are other candidates who do have that much experience in a specific language.
True. But the context of this discussion is that there are not very many candidates with the (nominally) required experience (i.e. several years of professional Rust experience). And that's often the case as job specs often have unreasonable requirements.
Without this, using even very simple hardware interfaces (like an SPI/I2C bus) is gonna be a huge pain, because you'll have to comb through reference manuals and register descriptions for your processor and piece everything together yourself instead of just calling a few API functions (this is also very error-prone and using Rust is not really gonna help one bit).
The only chance to get even halfway decent rust integration is to pick one of the like 3 most popular hardware platforms among hobby enthusiasts (think ESP32, raspberry pico), which is simply not viable for a LOT of embedded applications.
So I still think its probably a really bad idea for a typical embedded-shop to fully go for rust right now-- the downsides from lacking tooling/libraries, reduced developer pool and the need to train extant devs appear very hard to overcome to me.
Also consider that calling those FFI APIs from Rust is not gonna do anything for you by itself-- you are basically just calling C functions with extra steps... To actually get anything out of it, you'll then likely have to wrap the whole hardware interface in a "rustified" API (which, again, you'll probably have to write and maintain yourself...)
Once it was written the code was stable, but you bring back not so pleasant memories of building / testing. It might just be our make files were unnecessarily complex or we were building libraries not executables…
If you have a reference implementation in C it's not such a big job to set up something like an I2C or SPI peripheral. At it's core, you're making a sequence of writes to some memory locations. Most of the manufacturer libraries are quite low quality, and very easy to misuse. In my experience trying to do anything that pushes the hardware, you will end up combing through the reference manuals anyway.
I've done a bit of embedded rust, and the huge benefit - just like on desktop - is that code which compiles tends to do what you intended. Obviously there's still scope for logic errors, but that takes away a huge amount of debugging time.
Interfacing with the peripheral registers themselves is also a bit simpler with Rust in my experience, as a proper .svd already contains bitfield setups with enums and R/RW/W1 information to at least set the bits you want instead of the Macro-Hell in C some manufacturers throw at you.
There's a number of micros these days where the same libraries are provided using SVD[1] that will generate interfaces which is handy.
Better than any C(++) embedded hal I've used
Basically, this is a company who uses ESP32 to read serial data from batteries using UART and retransmits it in json over MQTT. They apparently had buggy C code to do that (for unspecified reasons) and successfully rewrote that in Rust.
Conclusion, you can write ESP32 code in Rust. No information on what that actually entails sadly.
I would guess that the more expressive type system is very nice but I don't expect memory safery features to be that useful in this context. Would like to hear from someone with real world experience on that.
Definitely much easier than in C, where I've resorted to using Ragel for safely parsing serial protocols (and also in Go!) since it generates a single small dependency free source file containing the state machine and your "actions", but nom is much more expressive.
Parser combinators are notoriously poor performance and memory-wise. They are somewhat nice to write if you like the style and don’t mind the tax but we are talking about ESP32 here.
Unless they are inexperienced, the issue is unlikely to be the state machine anyway and more likely to be in how they manage buffering of long message to avoid running out of RAM.
Another nice resource for Rust embedded is https://embassy.dev/ which also supports STM32, Nordic chips, and RP2040.
IMO, the coolest feature about Rust embedded is that Async Rust works! This makes building a HTTP server for your embedded chip easy.
Having said that, I agree with one of the commenters in this thread: Rust is essentially a solution looking for a problem. It is a great language, but it fails to find its niche. Rust developers are nowhere to be found, companies are not hiring Rust developers (except if you want to work in crypto).
[0]https://yieldcode.blog/post/one-year-of-rust-in-production/
Nowadays, Linux kernel also has some drivers written in rust so I'm hopeful.
Rust has some nice high-level features and tools, so it gets used in other areas like web dev too, but in the end, it is a systems programming language. It is a low-level language, even though it may not feel like that a lot of the time.
Let's say I want to use a real type-safe language to write web applications. So rails/python/JavaScript/php is out of the window. I'm left with two big options: Java/Kotlin and C#. If I want to avoid Microsoft, I'm left with Java. And in fact, this is the industry standard for "real" web development: financial, enterprise, etc.
But Java is cumbersome. It's very `FizBazBarFooAbstractInterface` type of cumbersome. Kotlin makes it nicer, but Kotlin didn't get much adoption in the web industry. C/C++ is too low level for web development. Rust is the perfect language, in my opinion of course, to fill this niche.
Oh, there is also Go. Which is, by the way, also a system programming language. But nobody argues that Go shouldn't be used to write web services.
If you come from proper languages, like C, the borrow checker is not that bad. The problem is that most developers don’t even know when variables go out of scope. Sure, there are more to that in the borrow checker, and I’m far from understanding every single use case. But for fay to day web development, it’s fine.
Java on the other hand is straight up annoying. It’s super bloated, there are 101 ways to do the same thing (recent example from swing: do I do new Broder(), do I use BorderFactory? What’s the difference? When should I use each one?). And it’s extremely wordy ‘Employee foo = new Employee()’
The thing I have to say in the context of the article is this: There is no way to know whether a complete rewrite in C would have yielded similar results to Rust. The phrase "C prototype" made me squirm, even more so when reading that in the context of critical infrastructure. It is known by now (or should be) that such prototypes live on like zombies, so unless it is really some throwaway (from the point of architecture!), these things tend to live on for longer than most feel comfortable with. And, being so critical in function, maintainability is one of the primary concerns.
Yes, Rust will, eventually, at some point, maybe? the go-to language we use in the embedded field, but we are talking not just about a language replacement, we are talking ecosystem replacement. That is not going to happen overnight.
That said, as some mentioned Java, Perl, and such things: I revived a personal Perl 5 project not too long ago that was more than 20 years old by that point. Needed a small change because the latest installment of Perl 5 is a bit more restrictive with some borderline syntax things (good), but other than that it just worked. In the larger context of the project there is also some C code for binary file processing, also >20 years old. Needed a renaming of a POSIX function (arguments and functionality all the same, though), and then it worked, even compiled as native 64 bit code. Granted, there are not that many dependencies beside POSIX, and the code was even back then written to a level of quality that allowed it to run on all sorts of (POSIX) platforms already.
Which is why "C prototype" sounds to me like "we cobbled something together", and all sorts of bugs are no surprise then. You can cut only so many corners before it becomes an issue, especially in software that is used all the time and in a critical place of a system. This needs to be done right, else you will waste a lot of time (and money!) afterwards.
Regarding C prototype. I wrote it. 6000 lines of code, works nicely, 4200 lines of code were Xilinx driver calls to move data between the hardware blocks. So changing the language will not really bring any benefit. Maybe even opposite - one must study the register calls and read data sheets to create equivalent functions in other language. The code wasn’t beautiful, was created as “prototype” and was at the end the version shipped to a client.
The embedded rust ecosystem has invanced a pretty insane degree since then. I have an ongoing embedded project and I have had to use C FFI calls a total of zero times in a ~100kloc codebase. There are native rust HAL libraries autogenerated from manufacturer published device specs that are insanely good, and take advantage of rust type system features to offer vastly superior APIs compared to mfgr provided C libs
If your Rust project leans heavily on unsafe code and/or many libraries that use lots of unsafe, then aren’t you fooling yourself to some degree; i.e. trusting that the unsafe code you write or that written by the 10 other people who wrote the unsafe libs you’re using is ok? Seems like that tosses some cold water on the warm afterglow.
The power of unsafe is that it‘s opt-in, making the surface area of „dangerous“ code smaller, more visible and easier to reason about.
As long as the unsafe parts are safe, you can rest assured that the safe parts will be safe too.
So, yes the less numerous they are, the more you gain from it.
Not really. Safety is non-local. It is possible to break unsafe code by feeding inputs from safe Rust that don't uphold the invariants that make the unsafe code safe. So it's not enough to look in the unsafe blocks--you have to consider the all the contexts that invoke the unsafe code.
See https://doc.rust-lang.org/nomicon/working-with-unsafe.html, and https://notgull.net/cautionary-unsafe-tale/ for a practical example.
That is not true. It is possible to have two pieces of validated unsafe code that are "safe" in isolation but when you use them in the same codebase, create something unsafe. This is especially true in embedded contexts, where you are often writing code that touches fixed memory offsets, and other shared globals like peripherals.
But in general this is specifically a bug in the unsafe code. The Rustonomican is very clear that it's not the safe code's fault that your unsafe code doesn't work. In the scenarios with conflicting libraries I guess it's the fault of somebody who linked conflicting libraries, but it's definitely never the safe code.
That's why every unsafe block needs a SAFETY block.
Is using vec.get_unchecked(6) safe? No. Is it safe for a vector that will under all circumstances (i.e. invariant) have exactly 64 element. Yes.
As long as for all possible inputs in safe function your SAFETY block holds, that code is considered safe.
It's true is that you have to trust your dependencies (unsafe or not). Not needing to trust at all that developers know what they are doing was never a thing a programming language could provide. We can only carve out some specific properties that we can machine-check in a limited way.
There are limits on what a type system can do (Rice's theorem, Gödel's incompleteness theorem), and in addition there are limits on what a non-dependent type system can do.
Therefore, you either need unsafe (something that adds operations that the type system doesn't model) or you can't write some perfectly OK programs.
Basically, the Rust type system is a toy model of your computer's abilities and the domain you want to model. And so is any other type system. The type systems of systems languages at least have some inkling of the actual machine--which is not necessarily the case in non-systems languages.
Ask a computer engineer what he thinks about this toy model's misconceptions, like that reading and writing from the same location via the memory bus affect the same thing, or reading the same memory location twice in a row when there's only one cpu is guaranteed to give you the same value, or that reading a memory location can't change it, or that writing to some memory location can't automatically change some other aliased memory location, or that writing some memory location from cpu 1 means cpu 2 can immediately read that new value out etcetc. I could go on (memory barriers, cache coherency, paging, ...).
This is not specific to Rust.
I'm not sure why we are having new "unsafe" discussions lately. Java and .NET have unsafe as well. Didn't we have that discussion already in ca. year 2000 and everyone arranged themselves with it? What changed? Are there new arguments?
If you want to have some empirical tests if the unsafe blocks are broken, run your program under miri.
Now you could say that you could just make better and better type systems that encompass everything as it really is. To that I say (1) you can't do that in principle and (2) if you could, humans wouldn't be able to practically use it anymore and (3) It would be too much effort for something that only a tiny minority of programs need in some places. The toy model is pretty good 95% of the time!
I wonder how much of it is low-level experienced developers only ever using C fail to see that C is not the universally best tool (or, 'if all you have is a hammer, everything looks like a nail' question).
I'd say that's it. From my own experience with software developers, convincing some of them to learn something new is practically impossible.
If the person in question has a true passion for the craft, you can regardless of the age/seniority of the developer (at least in my experience). In fact, learning something like a new programing language is a big undertaking and if your work doesn't offer incentives/rewards the will has to come from the person him/herself and so that's why the passion bit I mentioned above.
In my experience I also notice that more senior/older devs are more reluctant to learn new things, but I am unsure if that's due having their passion destroyed by many years of bullshit companies politics, pointless meetings/trainings, and adherence to the latest flavor of agile development every quarter or simply an age thing (I'm not there yet and so I can't tell first hand).
Sounds like a way to replace yourself by 2 low pay students who also have 1 year of Rust experience.
Edit: My point with the replacement is not that juniors don't/won't have C on their CV but that a manager would be more willing to replace an experienced dev with 1 year exp in the current tool with 2 worse devs also with 1 year exp in the current tool compared to replacing an experienced dev with say 10 years of experience in the current tool with 2 worse devs with 1 year experience in the current tool.
The more you know the better software engineer you are with everything else being equal (IMHO).
Also, if you need to gradually add rust to an existing codebase the C knowledge is extremely valuable.
Suck here is more of an emotional thing. They can output high quality code fairly quickly. The issue is not knowing all of the quirks of the language and all development being slower and more difficult.
Dropping down to simpler tasks while learning the language is a huge ego blow.
It was a good argument literally a couple years ago less a few weeks before the chatgpt first release. Nowadays it's basically a moot point.
Well, with some exceptions...
Each value in Rust has an owner.
There can only be one owner at a time.
When the owner goes out of scope, the value will be dropped.
[1] https://doc.rust-lang.org/book/ch04-01-what-is-ownership.htm...
Or, as the old tale goes, your GC might be the software literally exploding (it's in an air to air missile)
A lot of C(++) devs just like that they can mostly get away with writing broken code
Anyhow, on the other side of the coin, it's good to see newer languages getting a proper outing in real world situations. Proving stuff is 'up to it' can be a bit of a long haul, so every data point is useful.
> but spent basically zero time debugging
that's the difference to rewriting it in C
Rust moves more bugs to compile time, so you will technically spend more time getting the code to compile, but in my experience in 99% of cases it's a time saving. And it lets you be more confident about a program correct by construction, rather than merely fuzzed and not observed to crash. The 1% of counter-examples is trying to be too clever with generic interfaces and hitting Rust's limits.
2024-02-20 (3 comments): https://news.ycombinator.com/item?id=39446699
2024-02-04 (0 comments): https://news.ycombinator.com/item?id=39251131
Rust does not need a phd in quantum physics. Anybody can learn it with a bit of patience.
Once upon a time a specialist in Computational Chemistry or Physics or whatever would be expected to learn Fortran, and it makes lots of sense to use Rust for the same purpose in the interim (eventually a WUFFS-like special purpose language seems like a better fit to be able to deliver absolute safety and better performance by not needing to care about generality)
This kind of attitude only works in an exceedingly small part of the software world that just happens to be disproportionately represented on sites like HN. Elsewhere, it's not a good luck to be using words like "legacy" on a resume without a lot of explanatory text about why exactly it really was legacy and deserved a full rewrite.
I have the same sense as you: when someone tells me they rewrote their company’s “legacy” code, my spidey sense perks up. Now I’m wondering if there was really a good reason or if he just refuses to learn an existing code base.
The safety of references, no raw malloc, no null pointers, compiler-checked thread-safety of types, consistent and enforced error handling help a lot to make robust programs, and allow making bigger refactorings without fear of screwing something up.
The Turing Tarpit means that theoretically everything you can write in Rust you could have written in C, but in practice Rust enables things that wouldn't be worth the risk/effort in C, even when doing a ground-up rewrite.
If only Oxide found a need for esp32 modules, perhaps they would be up for the task?
However we need to work with a lot of the hardware apis like twai, ble, lte, ota, etc. It seems like support is spotty or it’s DIY.
It’s also worth keeping mind the remark about developer availability. I can generally answer questions about the C to someone who’s not experienced in C, while Rust might be another story.
I can somewhat attest to this! I have a pretty extensive prototype of using Rust on ESP32 in production. I wrote a reference implementation for ESP32 platforms deployed via BRSKI, a secure remote key bootstrapping protocol which builds upon attestation keys and public key infrastructure.
I'm largely using `esp-idf-svc` for most things, which is a Rust-native wrapper around ESP-IDF code. For anything closer to the hardware, I'm using `esp-idf-sys` which is just a thin wrapper around the the `ESP-IDF` SDK.
You can see the project here: https://github.com/hm-seclab/open-brski
Hope this helps! I'm by no means an expert but I sometimes help contribute to the Rust ESP32 ecosystem, so ask away if you want :)
I guess your takeaway is that despite spotty support, the benefits of rust are net positive? I guess learning to call C from Rust is gonna be part of it too right ?
> I guess learning to call C from Rust is gonna be part of it too right
Depends on the platform, but for ESP32, all native C SDK functions have thin Rust wrappers.
Rewriting something is not the same as the first effort. Try green fielding see how long it can she develop it once you already had the first basic C code it's pretty trivial to convert it to rust most of the time especially just using other libraries.
And imagine that. You fixed all the bugs in the prototype and the second rewrite didn't have as many bugs. That's his nothing about the second versions language it just says you fixed all the bugs in the prototype.
So why do we need Rust at all? What's the use case for it?
Anything that I'm missing?
C# is garbage collected. This is a no-go in many/most embedded software applications.
C# also grants you poor explicit control over heap/stack allocation: this is essential for embedded development.
I think that most modern embedded systems are nowadays more powerful than my first desktop computer — is it really still worth for the majority of embedded projects to count every byte at the expense of developers' productivity (and overall project success, as a result)?
Yes, you can run a VM on one of those processors in which you run a language like C# or Java. Look at MicroPython as a successful example. But the code which runs that VM has to be written in something. Typically that has been C and C++ - but Rust can also do it.
You're right though - a lot of what we call "embedded computing" could be done using a modern VM interpreted language. There are some languages out there - MicroPython and Squirrel come to mind - which can run on the memory and storage constrained environment of a microcontroller. The mainstream implementations of Java and C# use way to much memory - they've been optimised for desktop environments. Microcontrollers often have 64 - 1024 kB of RAM.
You could ship products with application processors like the Raspberry Pi, running an OS, and write your application in whatever language you like. But that costs 1-2 orders of magnitude more money. A cheap microcontroller is $1. By the time you've added the RAM, flash and supporting power rails, a cheap application processor is $10+. Then you have to maintain an OS - that's a lot of engineering time. Sometimes it's worth it, sometimes it's not - the tradeoff depends on the product.
If it’s embedded in a coffee machine, maybe all cost-effective taken into account it’s ok to have an over-bloated software stack maintained by the cheapest folk the manager could found.
Now if you are working on embedded software for some vehicle like a critical part of a car, a train or a spaceship, considerations of safety for both ethic and legal reasons might lead to different tradeoffs and conclusions.
The actual issue is existing selection of runtimes for embedded platforms is limited: https://nanoframework.net/ and https://www.wildernesslabs.co/device (to be fair, a friend of mine uses the second one for automating his lab for his microfluidics devices research project, so it is useful).
It's the best, but if I can't say: "never heap allocate under any circumstances," it's a problem.
I'm not saying it's an optimal choice for embedded, only pointing out that you can write allocation-free code with confidence. Patterns that allocate are known - predominantly boxing and closures. You can easily avoid both, you can also declare your structs as 'ref struct' which completely prohibits placing them on the heap, either as a box or as a part of heap-allocated memory inside some other object/struct.
When you compile rust for an embedded target it's running "bare metal" - there's no operating system, the microcontroller literally just starts running the instructions your compiler put at memory address 0. If you want anything an operating system offers - task scheduling, memory allocation, file system access, network access etc - you have to compile it into the binary that's loaded into the microcontroller.
Only languages which compile directly to machine code are really suitable for that - C, C++, Rust (plus a long list of less common ones - this list of languages which can compile using LLVM is a good place to start https://github.com/learn-llvm/awesome-llvm?tab=readme-ov-fil...)
- https://www.wildernesslabs.co/device
Here are examples where C# is successfully used as a language for an embedded target.
In addition to that, compiling to bytecode is just one way to execute it out of many, and the statement does not correspond to reality. I'm not arguing C# is a good language for IoT. I'd personally use Rust for that in almost every situation, but the amount of false claims in this discussion is disheartening.
Nano Framework is a framework to run C# apps on microcontrollers - but when you get to the hardware interfacing, it's done in C++. Look at https://github.com/nanoframework/nf-Community-Targets/tree/m... for an example. It appears to be using the ChibiOS RTOS at the low level.
Wilderness Labs have built a C# runtime on a microcontroller. But if you got access to their source code, I can guarantee that somewhere there's C++ or C code setting everything up, kicking off the virtual machine etc.
Nobody is claiming that you can't run bytecode-compiled scripting languages on embedded targets. Hell, you can even emulate other CPUs on embedded targets. But ultimately to set up a microcontroller and use its peripherals you need to be able to read and write registers on the memory bus, which means you need to be able to create assembly that the CPU understands.
There are projects to compile C# directly to machine code, bypassing the need for C++, C or Rust to set everything up. But those aren't at all mainstream. C# code expects a memory allocator and garbage collector. You'd have to explicitly run those somehow.
If you're already going to have to write some sort of board support package in a language that compiles directly to ARM machine code, then Rust is a great candidate. And if you're already writing your board support package in Rust, it might make sense to write your business logic in Rust too.
In all C# codebases I've seen, you have threads and tasks, and you often run into multiple threads or tasks holding a mutable reference to the same data. That's not legal in Rust without synchronization/locking.
If you don't believe me, just spin up a new main.rs file and write code that has a data race.
While it is true that Rust is a strictly superior option for highly concurrent systems code, it still leaves areas where you can make a mistake regarding lock management and other advanced forms of synchronization.
In addition to that, .NET as platform is fairly tolerant to misuse and calling the code that is not thread-safe from multiple threads concurrently usually leads to logic bugs or "stop modifying this collection concurrently, please" exceptions but not to catastrophic memory safety issues like it happens in C/C++.
You can read more on its low-level memory model here: https://github.com/dotnet/runtime/blob/main/docs/design/spec...
> The same is true for async, which in C# is also a problem.
Now, this one is strictly not true. Async primitives are thread-safe. In Rust, you must synchronize because at the very least you must deterministically deallocate memory used by shared state between the tasks. In C#, this complexity is handled for you by a GC (ironically, you get negative sentiment towards async from people having experienced Python's async or Rust's async complexity, assuming the same applies to C#). In some scenarios, it is also a throughput optimization since it reduces memory contention by not modifying the cachelines shared between the cores, lending itself into better performance on many-core systems - the memory is modified/reclaimed when it's no longer in use, while the actively shared data is placed elsewhere.
IoT is roughly equivalent to a Raspberry Pi - the thing will usually have an operating system that you're running on top of, and most of your existing knowledge about computers will port over.
Embedded is the chip in a happy meal toy, or your microwave in 1995. There is no "operating system." Your code is the only code running on the machine.
Typical IoT SoCs are a step up from that but it’s not huge. The ESP32 is pretty representative but there are significant other options - see what Amazon supports in FreeRTOS as a starter.
It’s a big leap from small IoT SoCs to SBCs like the Pi.
Also there's a video on writing keyboard driver in Nim [2].
[1] - https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...