Rust advocacy at a medium-sized startup
briankung.dev
briankung.dev
> Yet, it made me sad to look up articles on the language just to, in effect, bash it. I’m not generally in the business of discouraging people from coding, regardless of what their favorite programming language was.
While this level of self reflection shows a lot of maturity, if your judgement is clouded by excitement and you're searching for evidence to support your plan, this is _exactly the wrong_ time to attempt to shift your organization's direction. You're on tilt.
Worse, others are likely to recognize that you're on tilt, and discount otherwise rational arguments that you're making. It won't be clear to you why you're not making progress.
I've introduced Rust to an organization for a specific project, and it wasn't a decision taken lightly. It fit the problem domain well. I'll likely advocate against Rust for the next greenfield project that's on the horizon - as it's not nearly as much of a fit.
These are tools and there is no silver bullet.
This doesn't apply (as other commenters have said) to quitting and getting a job elsewhere that already uses Rust. Indulge in your passion without engaging in a sisyphean battle that'll burn you out and harm the business.
If the contender language is Go, think carefully before using Rust.
If the contender language is Python, think very carefully before using Rust.
Rust does an excellent job of providing fairly comprehensive opt-out safety, compared to C++'s hodgepodge of incomplete, edge case laden, compiler and tool specific, opt-in static analysis and annotations. Even a C++ skeptic like myself can learn a lot about how terrible the language is, by trying to write the same code in Rust and learning exactly why the borrow checker is complaining.
But it's also a very front-loaded knowledge bomb, with a nasty learning curve. It took me a good month or two to stop fighting the borrow checker and embrace clones and RefCells. Perhaps a less painful learning curve than debugging C++ gone wrong, but certainly more painful than a lot of other non-systems languages - I doubt it'd make a good 1st programming language. Arrogance means you'll shoot yourself in the foot with `unsafe { ... }` escape hatches as well, and end up debugging memory issues anyways. Web and gamedev ecosystems are still weak for my tastes, although improving. I want a nice existing successful fun game to contribute to, that's written in Rust, so I don't have to find the motivation to RIIR an existing one or write my own.
Also little point in statically proving to the compiler that your code is totally fine and safe if your bugs are already shallow through testing. I feel little desire to rewrite a typical C# codebase in Rust - what the borrow checker can catch, so too can proper unit tests, for the most part. Maybe invest in some code coverage analysis for CI instead. Unless those GC pauses are actually causing problems... or you have enough data races from missing mutexes in your multithreading code...
- infra projects.
- Resource constrained environment (depends on the hardware)
- Command line program
- WebAssembly for a complicated webapp (like Figma)
- Anywhere security is a concern
Whereas a web service that reads from a database and supports some CRUD operations? Many languages specialise in this problem. If the business domain is complicated and needs a good type system to express that domain - yeah maybe Rust is a good fit. But you're probably better off with Python or Ruby or Go. I could revisit this opinion in a couple of years if the Rust ergonomics initiatives succeed.
Video games - it's very early days for Rust. There's a lot of ongoing work (like Bevy) that could bear fruit in a couple of years but C++ is much better today.
- Rust is strongest where you need high performance, low latency, or low memory usage. If you're spinning up 20 machines to run a single data-crunching job, or if you care about your 99th percentile tail latency, Rust is great. This is where Rust shines compared to GCed languages.
- Rust also shines where programs need to transform large, complex data structures correctly. It's fantastic for anything resembling a compiler, even if the output isn't executable code. This is where Rust shines compared to languages without Rust's fancy "enum" and "match" (aka union types) and without Rust's strict null handling.
- Rust is strongest for CLI tools and for microservices with a small API.
- Rust is weakest for rapidly-changing business logic. This is because most companies have few Rust programmers, but they have many people familiar with scripting languages. In these circumstances, putting the business logic in a typed scripting language will make it accessible to more developers, and easier to change.
- Rust's closest competitor is C++. Rust has better tooling out of the box, it's arguably easier to master, and it has many fewer ways to shoot yourself in the foot. But in many problem domains, including games and native desktop GUIs, C++ will have far more mature libraries.
In my experience, the best way to introduce Rust to a company is to pick a project which is exceptionally well-suited to Rust's strengths, ask for permission to build a prototype in Rust, and do a really compelling job.
- A dramatic reduction in runtime crashes, and a ~70% reduction in security vulnerabilities (based on Microsoft's analysis of security bugs, but also personal fuzz-testing experience).
- Much better out-of-the-box tooling. You can set up cargo, VS Code and rust-analyzer in 5 minutes. "Maintaining the build system" will typically become a thing of the past. Rust also provides many verification features for free that would require setting up static analyzers for C++.
- A much easier time integrating third-party libraries. But you may loose access to certain domain-specific libraries, depending on your problem domain.
- Safer use of complex threading and/or async, if you need them for performance. Look at the Mozilla experience converting complex concurrent C++ modules to Rust. They've written a lot about it and it has almost always been a giant win.
- Hiring is a tradeoff: Depending on where your company is located, you may have more local C++ programmers. But if you do remote hiring, the Rust talent pool can be fantastic.
- It's easier to train people from scratch. Rust is not a small language, but it's a lot smaller than knowing the entire history of C++ gotchas and idioms. Also, Rust provides much better guardrails for novice Rust developers. It's feasible to take a strong junior developer, train 'em up, and turn 'em loose.
A migration like the one you describe makes the most sense if 3 things are true:
1. Your C++ codebase is small enough to incrementally rewrite (and your team knows how to manage an incremental rewrite without too much loss of velocity).
2. Your problem domain does not rely on major C++ libraries with no Rust equivalents.
3. Your developers are enthusiastic about Rust.
If any of those three factors were missing, however, then I would normally advise against a rewrite.
I haven't worked on any bigger production projects in Rust, so how to say if it would be a common experience, but sounds interesting
If the main value proposition of Rust is bug fixes, error handling and nice-to-have (?) platform support it sounds like you're really scraping the bottom of the barrel. Unless there is an extremely obvious and large benefit to using Rust over Go I would not do it.
I have not written any Go, but Rust seems to have a far steeper learning curve and there would also likely be high organizational costs associated with transitioning from Go to Rust (E.g. coordinating the transition, ramp up time, hiring, etc).
When Go is working fine and you only have 40 engineers this sounds like a massive distraction from building the product. I wonder if the author considered the possible negative impact of this as well as the positive.
Migrating a team of PHP, Javascript and Go programmers to Rust for no technical reason seems just hopeless.
It's putting the cart before the horse.
I think you do that by using the “weird” language starting from team size of one, not by convincing 40 people to re-write 500k lines of already working code base.
Does obsessing about programming languages really correlate with being an obsessive problem solver in general? I'm not so sure about that. My observation is that people can be deeply interested in one thing while being completely indifferent towards other things. Obsessing about everything is a mental illness.
Maybe it depends on the problem space. If the technology part of the problem is simple and the main issue is how to increase productivity then PG's argument makes sense. But if you have a difficult problem to solve that isn't developer productivity, then I think it doesn't make sense to hire people motivated by the choice of programming language.
I didn't find it particularly hard to pick up the basics, but compared to say Python or Ruby there is a lot more upfront complexity to do things.
If you're coming from C/C++ it's probably easier to pick up, but in terms of relative difficulty across all languages I definitely think it's up there.
You don't really need Rust for server-side Web stuff. That's what Go was designed for. Go comes with well-exercised libraries for most of the things people do web server side.
As I've mentioned before, I'm writing a metaverse client in Rust. Now that's something for which Rust is worth the trouble. You need concurrency. A lot of tightly interlocked concurrency. If you want a large high-detail virtual world, you need to keep a half dozen CPUs, a multi-threadable GPU, and the network all quite busy. There's a huge payoff in being able to do all that threading without spending time in a debugger. I've opened a debugger once in the last two years, and that was for a bug in someone else's "unsafe" code.
The price of using Rust is that you hit bleeding edge problems. I just finished writing a basic internationalization package to translate menu items and such. That should be unnecessary at this late date. But it turns out that the basic "internationalization" crate is fundamentally flawed, and even its creator now says so. The crate most commonly used, "fluent", is not suitable for use with the immediate-mode GUI used for games, "egui", because you need lookup memoization for strings retrieved on every frame. A lot of time is wasted on side issues like this.
Many widely used Rust crates are still at version 0.x.x, and not really stable yet. The language and core tools are pretty solid now, but the crates are all volunteer and not well-curated. The graphics systems I'm using, Rend3 and WGPU, are under heavy development, and not yet ready for prime time. I'm in regular contact with those dev groups.
There's not much off the shelf code for metaverses yet. Unity and Unreal Engine don't support metaverses yet, where the content comes from hundreds or thousands of people not working under common management. It's not like picking one of a dozen widely used payment rail systems. So you're going to have to do some serious development from a cold start. Rust is good for that.
For routine web development with short deadlines, Rust may be overkill or unsuitable.
And then, less dramatically but still significant, there’s the runtime errors and all the hand-written JSON (de-)serialization code. It adds up.
Recently, I ported one of my Python projects to Rust using PyO3 for exactly the reasons you mention. I haven't battle-tested this Rust code yet, so I can't tell you whether this was a good time investment. What I can tell you, though, is that it was much easier than I thought.
However, the productivity gains mentioned are an effect of any statically typed language, not just of of Rust, and the problem is that Rust's cognitive overhead offsets them. I definitely would not bring Rust to the table at my company because of that; I've recently had an experience with an engineer who thought it was fun to use, but couldn't write anything with it; I'm sure that with a simpler language, they could have.
I think there are new languages that are a valid middle ground (between Golang and Rust), although they don't have yet enough traction and/or maturity (most importantly, in terms of libraries available) - in other words, they's still hobby languages as of now. I have the suspicion that if a new language will become an established one, it will still take "not few" years, not less than 5/10, unfortunately. I like to use Lambda languages support as an indicator :)
I really enjoy programming in Rust, but it's undeniable it makes tradeoffs in terms of productivity for the unique value proposition it presents. The idea that Rust should be used to implement every CRUD webservice, or that Rust -> wasm should be used to implement every web app just seems like irrational exuberance to me.
While still having nulls everywhere. Rust doesn't have the concept of null at all - Option is the only way to represent the non-existence of something.
I've used Rust for about 6 years now, and I don't think I've ever used ptr::null(), mainly because I don't really write low-level wrappers around C libraries. I'd argue Rust does not have null as a concept or language feature - ptr::null() is just a convenience function for creating a pointer with the value of zero. The point of not having null is that in most languages null can appear basically in any variable or value, while in Rust null pointers can only exist when writing very low level potentially memory unsafe code.
> If the issue is representing non-existence, Kotlin, C# and TypeScript (configured properly) all have nullable types.
They do indeed have nullable types, but at least in in C# and TypeScript (and probably in Kotlin because of JVM interop) non-nullability is basically a lint with zero runtime guarantees. That is often good enough, but it's still a far cry from an absolute 100% percent guarantee provided by not having nulls even as a concept. Having nulls is a problem you can't fix by adding features - you have to design the language and/or the runtime from scratch without them.
> There's a good chance you'll be more productive with those instead of Rust for regular backends
I'm not arguing against the productivity of these languages or demanding anyone writes anything in Rust. I mostly write TypeScript in my day job, and it is a practical and productive choice for the work we do (web frontends, backends and serverless services). I just want to inform people about the capabilities of Rust and correct misunderstandings.
Heh? What else is it in Rust or any other language? Like, the only way I could imagine a null pointer exception leak into C#/JVM-language program that got statically analysed to not contain null is through some Reflection-based unsafe operation - at which point we can also talk about unsafe operations in Rust that just as well can cause “null-pointer exceptions”, except those will be hard memory failings (undefined behavior basically), potentially corrupting the whole execution.
https://docs.microsoft.com/en-us/dotnet/csharp/nullable-refe...
The language has to do this for two reasons. Firstly, this is a retro-fit, so if you don't have this you need to throw away all the old C# code, why not call the new language something else ? Secondly though, and perhaps more important to the C# ethos is you must allow null everywhere to be compatible with the Common Language Runtime. C# has to accept anything the CLR does at any function boundary. This means what can be promised is the lowest common denominator of all CLR languages which exist or are going to exist.
Suppose your C# code checks a string value called username is actually some string, by calling IsNullOrEmpty(). It'd be annoying if after doing that C# just insists username might be null anyway - you just checked! So, that function marks the string reference you called it on as definitely-not-null after having checked. However, you can just make a new function LOLWhatIsSafety() which just always returns 69 and marks the reference as definitely-not-null. In both cases the compiler just takes their work for it.
You can do this in Rust too. Unsafe Rust is still Rust.
In contrast all the routes I suggested for C# aren't unsafe, aren't even flagged as "Be super careful here, dragons ahead" because the C# attitude is that every single C# function is responsible for explicitly doing its own null checks if they're needed.
You can't do much with a pointer unless you use unsafe Rust. So this provides an easy to see red line for inexperienced developers. "Huh, this is unsafe, I need a grown up".
The affordances for Option<Thing> are - in my biased but experienced opinion after similar time with both languages - superior to those on a nullable type like Thing? in C#. Partly that's historical, Rust has always had Option<Thing> while C# originally had nothing like this, grew nullable value types in a subsequent version and then much more recently got nullable reference types.
The technical mechanism is also much weaker in C#. Suppose a junior dev pastes a Stack Overflow answer in to fix the weird compile error they have. In Rust, if the pasted code is doing something awful with pointer mangling that's unsafe and hopefully your review is like "WTF is this code unsafe?" but in C# maybe there's a null-forgiving operator smuggled in somewhere or some tricks with casting, and now your "Never use null parameters" function is being called with a null parameter. If you complain, Microsoft's C# support will comfort you by explaining that all C# functions are responsible for null checking, even if their signature explicitly says they don't accept null parameters, they need to actually write the checks explicitly, yes this costs at runtime as well as making the code messier with redundant checks everywhere, too bad.
Maybe Kotlin is better? But my guess is that in the JVM ecosystem it's not able to actually deliver better.
Rust takes more effort to constrain the solution more tightly, but I would lean toward it for any problem where the JVM is clearly too expensive. I don’t see a lot of design space between them.
That being said, I think Rust is cool so I'll give you a hint. Go has a kryptonite: binary size. This didn't matter until recently when WASM runtimes became the new hotness. It is very very difficult to deploy Go on e.g. Cloudflare Workers. Even with Tinygo, it's a struggle getting your binary less than 1MB. Some features that make Go ergonomic like reflection (encoding/json package is a good example) ruin "tree-shaking" and bloat the binary. If you're dead-set on using Rust, I'd try framing it in terms of edge computing. Rust is absolutely in a position to become the best non-JS language to use for these kind of 'workers' environments. And they have a lot of benefits: super fast, super cheap, effortless deploys, etc. so it's totally possible to make an argument for them. "Rust is the only compiled language viable in these environments"[1] is certainly a compelling position.
[1]. Hyperbole, I'm sure there's good progress towards getting others to work. For example, I'm currently working on adding Prolog support for Cloudflare Workers :-)
many wasm subjects are referring rust as _the_ language, but I feel modern c++ can do it as well if not better on all aspects(maturity, size, speed, even safety).
How come, and what sort of use case(s) do you have in mind?
This assumes Go is even evaluated as an option, let alone Rust. There are many programming languages any organisation can pick from. The right choice is usually "the language most programmers are most productive in".
I don't think "let's teach an entire team this novel programming language" is something you'll see in many companies. It makes sense for new companies, startups, and divisions, I suppose. A company with 200 employees is hardly still a startup, though, that's just a regular old company, even if it relies solely on new investors not to go bankrupt.
As someone who picked the primary BE programming language for a software org in the last year, our considerations were:
1. What's a language that's reasonably easy to hire for
2. What's the language where we feel that the community attracts high quality SWE's
3. What's the language that's easy to teach and easy to pick up, so that if we find a quality SWE who's not experienced, we can still consider them as part of our hiring pool
4. What's the language that's productive, easy to read and easy to maintain so that if we have turnover it's reasonably easy for someone new to pick up where others have left off
For us, Go scored very highly on all of these metrics, and imo it's almost purpose built to avoid the types of problems a software organization can typically run into. That's at the tradeoff of not being the most expressive or conceptually exciting programming language out there.
Imo Rust is at the opposite end of the spectrum. I have a lot of fun programming with Rust on hobby projects, but to be honest it would be hard to recommend it to an organization as a primary programming language unless the problem domain really can benefit from Rust's value prop in terms of performance, safety, and low-level control. Because you're paying a lot for it in terms of steep learning curve, potentially complex and difficult to maintain codebase, and small talent pool.
It's a hill at this point and people are calling it Everest.
There are a few incredibly uncommon things that are still difficult (the ones you see "Rust is hard!" blog posts about), but you're most likely never going to hit those corners. Especially if you're writing CRUD web apps with a few threads here and there. That path is boring and vanilla at this point.
I think most people would agree async is much more demanding in Rust than other languages.
And I think you can find Rust intuitive in hindsight once you have picked it up, but I don’t understand how you can seriously argue that Rust doesn’t have entire categories of topics that just don’t exist in other languages which make it harder to learn.
You could say you are just going to stay away from the more esoteric bits, but the problem is if just one of your colleagues decides to incorporate them in their code, now everyone has to be able to deal with them.
That’s a risk you don’t have with a language which is designed for simplicity.
> Rust Evangelism Strike Force
I know these are in quotes or meant as a joke, but it still is very off-putting to me. It's a team of 40 engineers and they have different subgroups of people coordinating essentially political efforts to advocate for programming languages. I would not like to work at a company like that.
What is with Rust and this uniquely fanatical evangelism? For me the biggest reason for not using it is the rabidness of its advocates. While they state technical reasons it seems the reasons are not technical in nature, which leads to my gut reaction of not taking them seriously.
For example, Go's popularity is only because of Google. I'm sure that has to do with it, but I don't think many companies are going to be putting their bottom line at risk by using a language just because it may have come from Google. It has to actually do its job and do it well. In that case, the proof is in the pudding. Again, as more of an outsider to these languages, Go seems to be doing pretty well for a relatively young language and it's already taking over the cloud space.
The business didn't care about nothing but hiring basic js developers and throwing them at the issue.
Nowadays I don't waste my energy trying to change clients, I'm just collecting a paycheck and going along with whatever bs the leaders think is good.
My stress levels and happiness levels are way better. If a company has leaders who are open to change they'll hire you with that in mind, if not, they can just keep rotting with the other dinosaurs.
You could get most benefits of it by using Go in terms of tooling/speed, or something with a great type system if you‘re after the correctness checking compiler.
Personally I didn’t find a language yet that checks all the boxes I have, there is always some tradeoff involved that keeps bugging me. Recently stumbled upon OCaml and for the first time in years it looks like I found what I was looking for
Or the much more mainstream Java/C# choice with much bigger ecosystems.
High performance, embedded systems, or things that have some kind of criticality with respect to fail cases, and must be fast, sure.
That said, I think the design decisions of the Rust borrow checker leads most developers either to write borrow-checker-agnostic code by using indices and Vec<T> instead (see https://kyren.github.io/2018/09/14/rustconf-talk.html), or go ham with Rc<T> and Arc<T> until the language basically becomes similar to (but slower than) garbage-collected languages like Go. The former leads to performant code but makes you question why you are using Rust in the first place, and the latter leads to slow unwieldy code and also makes you question why you are using Rust in the first place.
Are you delivering your groceries in a Ferrari, or a Truck?
As Jeff Atwood wrote, "Performance is a Feature" https://blog.codinghorror.com/performance-is-a-feature/
So, if your answer to the question, "Do you want speed, trivially?" is "No, I do not." Ok.
It is not inevitable; it barely ever happens.
This single line shows the extreme bias of the author. Rust is new and exciting, but it's not the best at everything nor nearly as productive as Java or Go at general webapps. Engineering needs to serve the business and get things done, not vacillate on language decisions. This was a rather big waste of effort and I blame leadership for allowing it to continue as far as it did.
but the ending? Is this a "no good deeds go unpunished" story in sheep's clothing? perhaps young engineers are being conditioned to sudden and unexplained unemployment as if that is normal ? that's a different kind of sheep. Are there grownups in this story ? just kind of awful at the very end after all that positive detail.
When an infra investment makes sense (lang, DB, ...), especially for 40+ engineers and who knows how big a workload, the multiplier should make the benefit clear. GPU computing speedups, k8s for easier elasticity, something, and ideally obvious like some sort of 10X. We certainly have quite a queue internally like that ("platform X unlocks property Y that has lift Z"), as do many. Conversely, with time wasted, that's a person distracting themselves and everyone else. Getting aligned with the business isn't about pushing a language that might help but identifying what is the most thing you can do to solve the company's/customer's problems, and going for that one.
This is the company he is talking about: https://www.hologram.io/blog/why-we-dont-hate-php
They love PHP and Python. Good for them, I don't like those languages but why would I get a job there and try to get them to use a language with a pretty much completely opposite philosophy?
It would be like getting a job at a vegan restaurant and suggest they should have wagyu beef on their menu because it's so tasty and everyone loves it.
Often companies that are in need of more bodies will hire people and tell them they can certainly help change the culture and help adopt a new language. Everyone there wants to believe it but it'll probably never happen.
There is probably a fun math problem here: count up the number of engineers, sum up their time in the language. If the average is X or more, they will never switch away from the language they've earned their keep in. It's too hard once you get past a point. Maybe that is already a rule. But I'm sure it is stronger for certain languages, probably PHP especially.
I currently work in a Go shop. Why Go? Did it win some business value delivering contest? I don't think so, it's just the initial team lead liked Go the most of the languages they knew and made everyone else learn it.
If you read my comment history you'll see I think Go is a mediocre choice. But, it's here to stay in this company and it doesn't matter what new language comes along it won't have the inertia or acceptance of the status quo language. The expected improvement has to be large enough to overcome the associated costs to sell the business case.
This idea of the status quo being somewhat arbitrary, and difficult to change, applies to more than just languages. It applies to everything else, your architecture, your work processes, your culture. A blueprint is set early on by some founding members, and perpetuated potentially forever.
In this world, rather than trying to change a company you change employers. Innovation is change, and to achieve that the individual changes, but leaves their company to remain the same.
Personally, I seek to improve in what I do, to innovate. And I think what we do can be done better and that the tools and systems in-place play a part in that. But will I introduce them at my workplace?
Perhaps not, I think I'll find a new job, and both my old and next company can pay the rather high costs of someone leaving and someone joining. This makes me a little sad, since I like my company and leaving would be a significant amount of institutional knowledge walking out the door.
By this logic either language keep changing when new people join and desire some different language.
> This makes me a little sad, since I like my company and leaving would be a significant amount of institutional knowledge walking out the door.
You shouldn't be. If company you are leaving is any good they would have documented important stuff. If not, they'll get what they deserve. Better to join place where you are part / lead of initial team. Because anything less you would simply be following someone else's language choice or even when language is preferred still their product design decisions.
Changing is a big effort. Unless a company grows huge, they're probably stuck with their original choice of language for the whole lifecycle.
It seems obvious from the start this is never going to go anywhere. You already are using 4 languages at this company (PHP, Go, Python, JS/TS) ? Adding another is essentially a waste of time and a huge distraction away from the core function of a engineer which is delivering value to the customer.
In Go's language class of statically typed GC languages, Go is much worse than the best languages, which are Kotlin and C# (Go lets you forget to write error handling, had no generics until recently, uses duck typed interfaces (!)).
In general, Go's inferiority can be traced to its root cause, which is the Google Go designers having severe Dunning-Kruger syndrome because they think they are good because they invented C/Unix and work at Google and don't realize they have no clue about programming languages in general and were only successful because the limited hardware at the time didn't allow sophisticated programming languages and thus allowed a terrible language like C to succeed.
As a language class, non-GC memory safe languages are far better than statically typed GC languages since they don't have the overhead of a GC and can guarantee the correctness of concurrent programs since they are data-race free (in Go-like language two threads can share the same data with no checks whatsoever resulting in arbitrary racy behavior), can make other guarantees enabled by the linear type system and use composition instead of implementation inheritance.
So in general you should always use Rust unless existing framework or libraries for other languages make solving the problem much easier and the app code is small enough that the extra problems in using a subpar language aren't too bad (e.g. you probably want to use bash for system administration, Python for simple deep learning projects, Kotlin for simple Android apps, Swift for simple iOS apps, TypeScript for simple web frontends, etc.).
However, if you plan to write a lot of code, then just using the best language, i.e. Rust, is probably better since the time to write or correct dependencies will be small compared to the sheer amount of code you are planning to write for the main software.
If you are willing to give up good performance and a large library ecosystem for correctness, then dependently typed languages like Idris are also an option.