Writing a Kubernetes CRD Controller in Rust
technosophos.com
technosophos.com
Rust has a near-best-of-class type system, better abstractions, and runs on everything from embedded to the browser. Go has captured the hearts of sysadmin types and those looking for something just a step up from python (as evidenced by much of the devops code written in it -- which I'm very grateful for, I don't think I'd use kubernetes if it was in Java still), so I can see it growing in that field.
I don't see how Rust doesn't become the most important emerging systems language (C & C++ obviously aren't going anywhere just by virtue of being incumbents) of the next decade.
Rust is getting better all the time—non-lexical lifetimes and rust-analyzer have provided a surprising improvement in development velocity, but Go is still in another class. Personally I think Go is the best language for developing quickly—even better than Python for non-toy projects (I say that as one with 15 years of Python experience and almost a decade of ago experience).
You CAN think a lot about memory management if you care about performance.
If not, you can throw `clone()` around everywhere and still have a very fast program.
Compilation time is generally not the bounding factor for development speed these days. Theoretically, compile time is zero for scripting languages so that's at least one point against weighing compile time heavily. I think the key benefit of OCaml and other ML langauges over Rust right now is actually higher kinded types, dependent types, and things in that realm of abstraction -- which you're probably not going to be using unless they're built into some library you're making use of, so the benefit is kinda moot there. Feels like the best power/weight for abstractions in codebases is the typeclass (or Traits in rust), most of the stuff above that is not great for fast development unless it's already in place.
And if we're talking about ML languages, I'm firmly in the Haskell camp because I think it has a better ecosystem (more than any research language should), an incredibly good runtime system (they've got the best native+green thread implementation I've ever seen), and fantastic abstractions for working with shared memory (when you need that). The whole "if it compiles, it works" is generally true. No need for a holy war, but if OCaml is in the race for fast development (which means that ML is in the race), then I'd pick Haskell over it to start on ecosystem/adoption, production readiness and features -- if you can manage to keep your developers from climbing the ladder of research-level abstraction.
OCaml has a lot of features that other languages lack, but they aren’t enough to make up for the table stakes features that it lacks.
Multicore--Node, Ruby, Python etc. all seem to be doing fine without it. Nevertheless, it's coming with OCaml 5--this year.
Windows--support has gotten better over the last few years but still being worked on.
Hostility--don't know what to tell you about that one--it's not my everyday experience. Sounds like a one-off.
Syntax and style guidelines--they are the way they are for a reason (no pun intended), and after a while when you understand the reason for each quirk, it just falls into place.
Table stakes features--many other languages out there, but IMHO the really table stakes features are being able to actually model your domain properly in a language. Which IMHO very few languages outside of OCaml get right. E.g. here's someone trying to propose abstract data types for Go without much luck: https://github.com/golang/go/issues/43123
Ruby 3.0, released last month, has experimental parallel support via limited sharing “Ractors”.
I think this really depends on the the higher level abstraction you get to use in rust -- case in point being the original post. I'm not sure if this is a result of the rust code just being done that much later but the Rust in there is much simpler to read and understand than getting stated with a golang codebase (whether via kubebuilder, or others) outside of scaffolding your code base -- there's just less moving parts and less noise.
Golang shines when it comes to developer fungibility -- it's simpler to onboard, train, and hire go developers. It's also easier for those developers to make mistakes, but compile type typechecking will constrict the mistake "space" (more so than ruby/python/vanilla js), and performance will get a boost. If Java is the mouse trap because C/C++ were too hard, Go is a better mouse trap on almost every axis (though I think generics in Go 2 are sorely needed).
> Rust is getting better all the time—non-lexical lifetimes and rust-analyzer have provided a surprising improvement in development velocity, but Go is still in another class. Personally I think Go is the best language for developing quickly—even better than Python for non-toy projects (I say that as one with 15 years of Python experience and almost a decade of ago experience).
I agree here, though I personally favor Typescript for building quickly (at least it is for me) if you add Typescript to it. Performance is great because most simple things are not CPU bound at the API layer, ecosystem is enormous (finding value can be difficult), and it's relatively easy to deploy (though Go is easier). I largely stopped keeping track of Go once I'd written some stuff in rust, but I do still recognize it as an excellent choice for building systems quickly and with some (type) protection as opposed to other options, with easy deploys.
I'd love to get your thoughts on recent "modern" Python and it's shiny parts and warts though. I used it last year for a client and found the modern stack to still be pretty disappointing compared to what I knew was possible with server-side JS -- typing use being sort of incomplete still (mypy was a little awkward to use), async still being a little bit awkward, GIL still being a thing.
I think people over-index on abstraction, especially in a professional setting. I think humans have a harder time understanding abstractions than they do in dealing with concrete problems, and I think Rust (and many other languages) encourage us toward the most abstract code we can conceive of whether or not that abstraction is actually necessary. And not only are people bad at understanding abstractions, but making good abstractions is a skill, and I’ve seen way too many bad abstractions created when no abstractions are necessary at all. On the other hand, sometimes abstraction really is necessary or helpful (and contrary to Go’s critics, these cases are not to elide error handling boilerplate or to facilitate generic map functions or other “hyper-localized-abstractions”) and in these cases Go can be quite painful.
> I'd love to get your thoughts on recent "modern" Python and it's shiny parts and warts though. I used it last year for a client and found the modern stack to still be pretty disappointing compared to what I knew was possible with server-side JS -- typing use being sort of incomplete still (mypy was a little awkward to use), async still being a little bit awkward, GIL still being a thing.
My opinion is that using Python these days is painting yourself into a corner. Developers need more rails to keep them from writing shitty code and Mypy isn’t yet mature enough (and its pace of development seems glacial). Further the package management is still worst-in-class. I’ve never worked on a Python project that didn’t hit some major performance bottleneck that the Go/.Net/Java tier of languages wouldn’t have struggled with at all, and unlike that tier of languages, Python leaves you with no better options. Async makes a lot of applications faster, but it also allows for some pernicious bugs (“Guess Who Is Blocking The Event Loop And Bringing The Whole Application Down!” is not a fun game to play even with Python’s tooling). It’s particularly unpleasant if you don’t have a type checker because you’ll find yourself forgetting “await” a lot even in the kind of code that you would think is so simple that it doesn’t need tests. Mostly I think we have better options these days—my go-to is Go (pun unintended) but I’ve heard good things about TypeScript as well. If you really need Python for some data science stuff or something else, I would try to contain that bit as much as possible by making it its own tiny microservice or calling into it as a sub process call or similar. I would not write more than necessary in Python.
I love Rust. It's like a beautiful puzzle that keeps my mind sharp and lets me make things that are truly elegant and performant. But the borrow checker is omnipresent, inescapable, and complexity-inducing... far from straightforward.
In my experience type safety is one of the most factors in terms of being able to produce code quickly, at least once a project grows beyond a couple hundred lines.
I agree with your overall point, and I think the advantages of Go are especially relevant in a team setting: Go's categorical aversion to complexity makes it much more difficult to write code that someone else will not understand, where Rust projects can become relatively impenetrable when they wade too far into getting "clever" with the more esoteric aspects of the type system.
Also the learning curve is real: you can take a Javascript developer and get them writing Go code in a couple of weeks. If you hire someone who's not experienced with Rust for a Rust project, you may be paying them for months just to get up to speed.
Don't get me wrong, I'm a huge fan of Rust and use it for almost all of my personal projects currently. But I think the idea that Rust can become the main programming language in all settings comes from a place of heavy bias. Rust is definitely on one extreme when it comes to complexity, and it's a more relevant trade-off for some use-cases than others.
aside - I know it's a minority opinion, but I actually think that Swift is a sweet-spot language in terms being simple to program in like Python (by aggressively obviating away low-level details), but giving you many of the powerful tools for correctness which Rust has, like algebraic types. It's a shame that it's painful to use in all but a few "blessed" applications.
I think it depends on your domain. If you are working in a mission-critical space where every bit of static analysis saves hours or days of testing, then yeah, the more static analysis the better. If you’re building web systems software (things like Kubernetes) or even ordinary application software, then you get benefit from a little static analysis, but returns quickly diminish beyond a certain point. I think Go hits the sweet spot and Rust is generally excessive (development is slowed down much more than necessary). Swift might also hit that same sweet spot; I haven’t used it much. I have heard that it has some maturing to do before it’s ready for more than Apple GUI apps, but that might be dated information.
Speaking only in terms of type systems, I think Rust's features - like algebraic types and explicit nullability - are strictly better than Go and make it easier to write correct code. It just happens to be that this benefit is eclipsed by all the ceremony required to satisfy the borrow checker.
> Swift ... has some maturing to do before it’s ready for more than Apple GUI apps
I actually think Swift as a language is plenty ready for general purpose programming. The problem is the tooling. It's supposedly being worked on, but last time I checked it's possible but not at all simple to get it running on anything other than Apple platforms and a few blessed linux distros. Also I had the experience that every time I would uprgade the toolchain, it would pretty much break all my larger projects. Even though I like Swift as a language more than Rust in some ways, I've pretty much abandoned it on new projects, because with Rust I can just run `cargo run` on any system and be pretty confident it's going to work, and Swift is very far from this.
Also you always feel like the language is a bit at Apple's whim. Like when SwiftUI was released, they shoe-horned a number of features into the language which seemed pretty half-baked because of iOS's priorities. It makes one hesitant to invest too much in a language which is so heavily influenced by a single stakeholder who may have different interests than you.
Rust lacks a GC so it uses its type system (namely its borrow checker) to compensate. I don’t know that this can be considered “orthogonal to type safety” in a meaningful way.
> Speaking only in terms of type systems, I think Rust's features - like algebraic types and explicit nullability - are strictly better than Go and make it easier to write correct code. It just happens to be that this benefit is eclipsed by all the ceremony required to satisfy the borrow checker.
Yeah, I completely agree. Go with ADTs would be a significant improvement. I would really like to see someone write a language like this that is interoperable with Go (compiles to idiomatic Go to the extent possible).
> I actually think Swift as a language is plenty ready for general purpose programming. The problem is the tooling.
Hmm, well I hope it finds its feet. It certainly seems like it would be a welcome addition.
Ok fair enough. I guess mentally I separate the ownership/lifetime system from the type system, but I suppose it's true that borrows, mutable borrows etc. are actually part of the type system.
Rust complexity is embedded in the following: in order to get something to work, you effectively need to solve a puzzle.
Solving the puzzle is fun and feels very good once completed, and the prize is definitely worth it when performance and safety are critical.
I highly doubt that "solving the puzzle" makes sense in the context of ad-hoc automation or boring sysadmin/devops stuff. Last time I checked, creating a static binary (a-la go) was not easy. It may sound stupid... but at the end of the day go is the king of pragmatism.
I think exactly the opposite though about systems languages: Rust should take over C and C++.
Rust improves quickly so I don’t think we should discount it, but I wish Rust folks would address these concerns when talking about how Rust is going to take over $x domain.
Some refactorings in Rust has brought back memories of refactoring struggles with highly coupled code bases, where one seemingly small change ends up requiring touching many many areas of the code.
The omniscient need to define ownership across a code base seems to make it easy to introduce high coupling, which is unavoidable unless you start using shared_ptr/RC all over the place.
When was the last time you checked?
I recall it's truly static on MacOS though.
You give some Go and Rust code to some team that never used either language, I can tell you right away that the Rust code will never be properly understand.
Go is easy to read and to maintain, I can't say the same for some Rust code. Go itself has been stable for years, there is no new keyword, std lib is the same, almost no changes ( beside modules ).
When you look at Rust they're adding thing much faster and it feels like C++ at times, the mental overhead to keep up with that is not trivial.
I’ve been down the “developer speed above all else” rabbit hole with Python, and I don’t think it ever ended up working out meaningfully better than the “think the problem through first” approach.
Personally I don’t find go easy to read: the noise of the “mechanics” of the implementation gets in the way of the intent behind the code in my opinion. Also the error handling is far inferior to the more principled approach taken by Rust/Haskell.
> ...I can tell you right away that the Rust code will never be properly understand.
I think this is an unfair characterisation: if I did this with my c#-writing coworkers, I think they’d certainly get a cursory understanding of Go quicker, but it wouldn’t take that much longer to get a grip on Rust and they’d write better and more correct code in Rust.
> Personally I don’t find go easy to read: the noise of the “mechanics” of the implementation gets in the way of the intent behind the code in my opinion
I'd add another dimension: reading go is slower than reading Rust, but it's easy. Rust may be faster to read, but you can hit some very rough patch and serious head scratchers.
It's all a trade-off at the end of the day.
Certain languages give you better tools than others in terms of reaching a good design. Go for instance puts you in a bit of a "jail" in terms of complexity which prevents you from going too far off the rails. Rust uses a strict compiler to categorically prevent whole classes of issues. So yes in some sense bad code is a "people problem" but the right tools can go a long way to mitigate it.
> Rust may be faster to read, but you can hit some very rough patch and serious head scratchers.
I think this is where Rust's type system can often be a double-edged sword. I have run into more than a few libraries which try to make things "simple" to use by leveraging traits solve problems in a somewhat magical way, but in my experience this often makes it very difficult to trace what's actually going on. I find that Rust code is not all that self documenting, and it relies heavily on documentation in order to be understandable. And quite a few libraries are strong on per-item, per-function documentation but there's a gap in terms of high-level docs explaining the intended use of the library.
I've become convinced that any language without a strong type system will eventually lead to a codebase which collapses under its own weight. Lately I've been working on a bit of data analysis in Jupyter, and even in that limited setting I find it hilariously slow and error-prone working with python. I would rate developer speed in terms of the time it takes to imagine and implement a correct solution. The correct part is key, because solving issues at runtime so so much slower than doing it at compile time. The only place where Python has an advantage is maybe in terms of how long it takes to type the code.
> it wouldn’t take that much longer to get a grip on Rust
This just hasn't been my experience at all. I've been programming professionally for over a decade, and I've dabbled with probably over dozen programming languages, and done real work in several: C/C++, Java, Obj-C, JS, Ruby, Swift, Python and Rust. I would say that Rust is an extreme outlier in terms of learning curve. It really took months to get to the point where I really got to that comfort point where I can just type Rust code with confidence that it's going to compile without looking a lot of things up, and even then I run into situations where I have to ask for help, and I feel like I am just scratching the surface with some more esoteric aspects of the language. In most languages it has been possible to reach that point in weeks.
I say this as someone who actually loves programming in Rust and currently reaches for it often as my "main language" of choice, but it is a complex language which is not easy to learn, and this is a real point to consider in terms of using Rust in a team setting.
Recently the ability to do `exec` and `attach` was added. It should reach feature parity with the Go version soon once `port forward` is added.
Disclaimer: Co-maintainer of kube-rs
(Not to dunk on TechnoSophos, none of that existed back when the blog post was written.)
Disclaimer: Co-maintainer of kube-rs
[0]: https://docs.rs/kube-runtime/0.47.0/kube_runtime/controller/...
[1]: https://github.com/clux/kube-rs/blob/8611e5cdd23d4ff487fdcea...
Did the custom resource controllers pan out well?
It was never reactive, at least in the sense of the Reactive Manifesto. There is no system of backpressure.
> Did the custom resource controllers pan out well?
I think it will prove to be useful for a handful of uses, but that in general, it will wind up like Wordpress and Jenkins plugins. Powerful, popular, but a guaranteed mess.
Kubernetes was not originally designed to accommodate CRDs. There's no concept of tenancy, only things you can cobble together into tenancy-ish-alike-kinda shapes. There's RBAC, but it was designed before CRDs and means that poorly designed custom controllers can be attack vectors.
I think CRDs are useful and necessary[0], but that they are greatly overused and have expensive design flaws. Unfortunately, I don't have the wits to join into the multi-billion dollar market of "papering over things Kubernetes doesn't do very well but which can't now be changed".
[0] I mean, I wrote a book about Knative, which is a purely CRD-based architecture.
Could you elaborate on the "design flaws" part? Do you feel that the CRD model itself has design flaws or are you referring to specific project's CRDs?
——-
The usage patterns of native k8s types and the implications those patterns have on the scalability and reliability of etcd and the apiserver are relatively well-understood. CRDs can be a wild-card, though, and afaik testing efforts thus far have not investigated worst-case usage of CRD-based applications.
As commonly deployed, CRDs are served from the same apiservers and etcd cluster that serves the native types for a k8s cluster. That can result in contention between the CRDs supporting 3rd party additions to a cluster and the native types critical to the health of a cluster. This kind of contention has the potential to bring a cluster to its knees.
Efforts like priority and fairness seek to ensure that the apiserver can prioritize at the level of the API call. But that won’t prevent watch caches from OOM’ing the apiserver if excessive numbers of CRDs are present. The judicious use of quotas could head off the creation of an excessive number of objects, but it’s not just count that matters - the size of each resource is also a factor.
In theory, CRDs could be isolated from native types by serving them from an aggregated apiserver backed by a separate etcd cluster. afaik this not a supported configuration today, and even if it were the additional resources required to support it (especially the separate etcd cluster) may be prohibitive for many use cases.
You can actually nominate particular types be stored in particular etcd servers -- GKE does this to put Events into a separate etcd from everything else.
However, it still has problems. Firstly, you can only define it for inbuilt types. Secondly, it's common for different objects to cross reference each other through objectRefs and the like, which behave badly when you effectively perform a join in the API server over multiple etcds.
Interesting. Is this documented anywhere?
What I said was slightly wrong. It's not that you nominate Kinds, it's that you nominate which etcd servers get which etcd key paths. You can essentially work it out because the path structure is consistent.
>"But operating with CRDs at scale is a different story and suggests careful testing with the specific applications involved."
Do you mean the number of different CRDs deployed here or just the number of custom resources created? Or is it the same concern with either? I'd be curious what you are defining as "scale" as well?
Scalability is relative, and depends on many factors including but not limited to:
- the resources available on the hosts running apiservers and etcd members
- the number and size of resources (custom and native) that controllers will maintain
Relatively speaking, a cluster of a given size might be perfectly capable of handling on the order of many thousands of resources . Push that an order of magnitude and the overhead of serving LIST calls - marshaling json from etcd to golang structs for apimachinery and back again for sending over the wire - could exhaust an apiserver’s memory allocation. And since the impact of resources is cumulative, any one application relying on lots of CRDs might not destabilize a cluster on its own but might well contribute to an unhealthy cluster when running alongside similarly CRD-heavy applications.
The key takeaway is that the kube api is best thought of as a specialized operational store rather than a general-purpose database. Anyone wanting to rely on CRDs at non-trivial scale would be well-advised to test carefully.
Controllers can be scoped to a namespace without requiring cluster-level permissions, no?
It means every controller for a CRD winds up installing another webhook. And having to test a variety of orderings. It's hard to get right.
> Controllers can be scoped to a namespace without requiring cluster-level permissions, no?
The difficulty is that Kubernetes RBAC is good at expressing rules "Role 'foo' can perform operation 'list' on kind 'Deployment'". But it's less capable of saying something like "Role 'foo' can perform operation 'list' on kind 'Deployment' which were created from kind 'CoolerDeployment'". It's also hard to delegate something, along the lines of "Role 'foo' can delegate ('create' over kind 'Deployment') within namespace 'bar'".
I think dissatisfaction with PodSecurityPolicy will cause Rego to worm its way into the core architecture over the coming years. It'll then eventually crowd out RBAC because you can impose (sort of, more or less) arbitrary rules. But none will dare call it ABAC.
Custom resources had/have some growing pains but I think worked out pretty well. It can be very very hard to test and distribute them though. As someone who maintained a controller with paid support, your test matrix gets pretty large pretty fast accounting for different k8s versions, different hosted versions (GKE, AKS, etc), different distros (openshift, rancher, etc). And that's before you even get into specific configurations like pod security policies, can the control plane communicate with the data plane, is there a service mesh.
Resource versioning is hard to get right. Once a resource type is v1, it becomes difficult to extend it. You can't add a beta field to it easily. Revising schema can be hard since anything more than a no op conversion between versions requires a webhook, which requires a certificate chain, and while cert-manager is popular it is not ubiquitous and regularly has breaking changes. Webhook setup issues made up a large portion of our support requests.
As far as the general "reconciliation loop" architecture goes, you end up with something similar in most orchestration systems I've worked on, or you wish that you did. So overall I think that worked out well. Getting it right can be hard, but I think that's the nature of the beast.
Right ...
Because of Golang's anemic type system, massive code generation is the only viable way to deal with the dynamic blackboard that the API Server presents to clients.
It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keeping up with that.
Replying to your edit:
> It's just a bad idea to write something for Kubernetes that is not in Go, you have 0 support and k8s releases a new version every 6month, good luck keeping up with that.
Well, a few things.
First, using client-go doesn't save you from version shifts. It is explicitly unsupported, so if they break your downstream system, stiff shit.
Second, client-go is derived from code generators. The same generators also produce other clients. Unfortunately they are terrible, by virtue of OpenAPI generator being a hellish maze and because they leak Go-isms like a spiteful sieve. I've used that code generator to develop a proxy. It sucked.
Third, this library is derived independently. It uses the same OpenAPI definitions as client-go, client-java and so forth. It's as easy to keep up to date as any of the others.
Fourth, the Kubernetes API isn't fully tested. Not even close. Many libraries can rightfully claim to be just as conformant as client-go, despite being broken on ~66% of endpoints: https://apisnoop.cncf.io/
See https://github.com/Arnavion/k8s-openapi/blob/master/README.m... for a full description.
I also confirm that I keep it up-to-date with Kubernetes releases and have been doing so for the ~3 years that it's been around. Not just the minor ones every few months, but even the point ones; these days the latter usually only involves updating the test cases instead of code changes and they're done within a few hours of the upstream release.
The generated codebases like client-go provide a great deal more navigability of code rather than APIs. It is the code that I actually interface with. I have my complaints about the necessity of code generation, but I do like having some kind of in-advance typing to help me figure things out.