Fault tolerance and resilience patterns for Go
github.com
github.com
I might be slightly biased, because, not knowing about this project, I implemented half of the functionality in my own simpler module: https://git.sr.ht/~mariusor/ssm
I've spend a lot of time ripping them out of java projects abusing those patterns for internal services, where service interruption was caused more by the delays from the circuit breakers than actual problems with the internal end points. but everyone who added them got promoted in the past.
and on those companies, most java people are now go devs when they were retrained during the move to the clouds because then most tools were go or python.
I found that states are not really required to know anything about internals of the executed code. All that a policy needs to know if the next state is an error or not, that's all.
I might have simplified the problem too much for my library to be useful in real life applications, but I still hold hope that a
stateFn func(context.Context) stateFn
is expressive enough for implementing the whole functionality.
If I may make another suggestion, I found that for more complicated flows, it is useful to be able to build a graph. I have made a rough approximation of a toold for my project based on parsing the go sources and inferring a state diagram from it. You can check it here: https://git.sr.ht/~mariusor/ssm/tree/master/item/cmd (Apologies for the lack of documentation, my work is much more recent than yours :D)
Glad to see a Go version.
https://www.oreilly.com/library/view/release-it/978168050026...
https://learn.microsoft.com/en-us/azure/architecture/pattern...
Should have added that I read this book in 2016, and the first edition is even older, so there’s naturally been lots of new (and exciting) developments in this area!
“these libraries are just what otp does natively”.
“Here are some things in this library that otp doesn’t do natively.”
“You could build a erlang library for that”
Having enthusiasm for erlang/otp is fine. It’s a pretty great stack. But there are lots of reasonable reasons not to use it and lots of distributed systems knowledge we’ve picked up since it was designed. Some of which are encoded in libraries like these.
If you'd like, you can attribute that to lack of Erlang experience or skill on my part. But in practice, we instead replaced that system with Go running on a Kube cluster and it's now been rock solid.
I would choose Go every single time. It has a better story for teams working together, largely due to better typing. Lacking types (and specs don't cut it), I still found I had to go up multiple function calls to understand what the parameters were, how they could be used, etc. so much state has to stay in your head, which I did not expect from a functional language
OTP has either a usability issue or a marketing issue
Static typing: I've worked for years with dynamically typed languages and won't go back. Dynamic typing was a big mistake that have wasted years of developer time. Dialyzer isn't an option.
Syntax: Erlang's syntax is not great. Enough said. I think Elixir isn't better, or at least doesn't offer enough of a value proposition over straight Erlang.
Native compilation: Erlang works well for control systems. It's absolutely subpar for anything requiring compute performance or I/O. This means you may find yourself in a sticky position if your backend app has hotspots needing very low-latency, high-performant logic. You may have to write bits in C or some other natively compiled language.
Friction: Erlang is hardly used anywhere. It is sufficiently weird (functional, Prolog syntax, immutable, the process model, antique tooling, not to mention OTP itself) that it's not just something most developers are going to pick up in a day. Forcing Erlang on a team not consisting of a monoculture of Erlang developers is a huge ask in any company. It's hard to hire for. You can't just reassign a random "full stack" developer within a company to a team that uses Erlang, if all they know is JavaScript/TypeScript or Go for that matter.
I think fans of Erlang have to realize that no matter how much they post comments like yours, it's just not going to take off. Sometimes superior tech just doesn't pan out (Amiga, BeOS, QNX), but we move on.
I am optimistic about Gleam, though. A much better language that I would actually want to use, and there's interop with Erlang/OTP, which is a great way to bootstrap a language.
This is mostly accurate, but I think Erlang is a good fit for network I/O, especially network I/O with lots of concurrent clients.
> Friction: Erlang is hardly used anywhere. It is sufficiently weird (functional, Prolog syntax, immutable, the process model, antique tooling, not to mention OTP itself) that it's not just something most developers are going to pick up in a day.
At my Erlang job, almost nobody we hired had Erlang experience. It certainly took longer than a day, but IIRC, my first big Erlang server stuff deployed (and mostly worked) a month in, and I was focusing on a PHP service. I probably did some minor stuff earlier, but I no longer have changelogs to reference.
Yeah, some of the OTP philosophy takes a while to understand. As for antique tooling, we just used Make, and that's what I continue to use for my Erlang projects, and well Make is Make... once you learn it, it works. Figuring out how to get dist running and pop debug shells when and where you need them is work, but if you're dropped into a place that already has that, you don't need to figure it out yourself, either.
To some extent golang fills that void because of net, it's application in projects liks k8s, and it's concurrency dx through goroutines. But it's also had problems that I don't want to dive into here that I think make it more divisive than would be expected.
- https://medium.com/coryodaniel/from-erverless-to-elixir-4875...
- https://elixirforum.com/uploads/default/original/2X/f/fdd70a...
Perhaps you’ve heard of Discord, Pinterest, WhatsApp, Spotify, Moz, PepsiCo? I’m sure I’m missing a whole bunch of names as I’m in the bootstrapped world where you’re seeing a lot of Elixir (especially now that LiveView is stable).
Tell me, why would you pick something that needs 8 servers to do the job Go or Rust could do with only 1?
Allow me to introduce you to this language called PHP.
Usually the hardest thing for devs I've seen learning Elixir is just getting used to immutability. What were the severe problems with the learning curve your former startup encountered?
https://laravel.com/ (or Django or Rails)
because it brings most of the things you need to write in other frameworks by yourself. Too many reject it, because, PHP!
Can build for Linux, macOS, Windows, Android and iOS, either JIT (better for servers, sometimes GUI/Games) or AOT (better for phones, CLI, serverless).
Its build system and CLI tooling is very productive and easy to get started with.
In PHP:
return base64($myText);
In C#, it's much more involved to do these "easy" things. Even reflection is easy in PHP:
$class->$dynamicMethod(...$props);
The base64, in most languages, requires importing the right package or namespace.
Converting bytes to Base64 is just Convert.ToBase64String/CharArray(bytes).
I can't believe someone would seriously and unironically suggest PHP. It does not even compete at the only area it's applicable at being back-end or server-side rendering.
> It does not even compete at the only area it's applicable at being back-end or server-side rendering.
I'm not sure what you mean here. I've deleted my blog, but I once published benchmarks showing PHP was faster than C# in some cases. I'm currently working on a reimplementation of some C# stuff in PHP and competing with C# quite well.
PHP is written in C; it's quite fast.
To think that an interpreter, where each single instruction is, in the very best case, a computed goto jump with args evaluation, would execute the code faster or comparable to code that is compiled to CPU instructions, is insanity. Even when/if PHP gets JIT compiled, it still has the same language constraints where even something as sophisticated as V8 has to contend with language spec so it cannot optimize away e.g. property access.
It might be best to re-evaluate prior assumptions and knowledge which do not correspond to reality.
heh, right back at you dude. Opcache JIT is running on an actual CLR just like C#, it can even AOT compile.
From the article: Spotify is using Elixir for an internal ad debugging portal.
So no. Spotify backends are not written in Elixir but:
"The listener-facing backend services are done with Java."
But of course most large orgs have a little bit of everything.
Elixir is promoted by enthusiasts a lot but once you start writing a project in it, you’ll realize after a couple of months you are writing a lot of things yourself, popular tools don’t have bindigs, lot of libraries that do exist are abandoned etc.
I worked on a Phoenix project that had a Graphql API, we couldn’t integrate it with Apollo Graphql because the Elixir library didn’t support some features, had them in the backlog for years. The company had 20+ other services in different languages, none had this issue.
Though I am not buying the idea that lack of static typing would hinder adoption as many of the most popular backed languages are traditionally dynamic with gradual typing only becoming popular in recent years.
We need to drop this fight around tools. Discussions are fine, but "My tool is better than your tool" doesn't get us anywhere (not saying you're doing this).
In general there is not a lot of innovation (in the mainstream), the only recent thing that comes to mind is Rusts borrow checker.
It's been painful to see how much Java has squandered it's potential. Putting aside something like OTP, just the concurrency in Java has been confusing - project loom has been on the horizon for a long time, and it makes it difficult to invest in something reactive like Reactor/Webflux. Or, having quickly reviewed elixir, since it uses the actor model, something like Akka - which words don't even describe how much of a dumpster fire that whole thing became (which I feel like I dodged a bullet because my old boss was super gung-ho about how Akka was the next big thing a few years ago). And if we're talking about things on a platform level, then... spring is not it. Others have mentioned K8s, what takes a simple yaml config there takes like huge guides, a wing and a prayer on Baeldung articles, and a soul crushing soup of XML for config (yea I know Spring supports other formats, but it's roots are in XML and old guards just copy that stuff over).
It doesn't seem like Erlang/Elixir community have these problems. Idk if it's because I am not invested in it so the grass looks greener on the other side, but it really does seem like they've got a productive and focused solution.
I only have some experience with Scala/Akka. Not sure if Akka was bad or it's the actor model, I found it hard to debug and understand code written by others.
I'm now 50+ and I use Go. It's so simple it even works for old people.
And there's some Elixir based loggers: https://logflare.app
Also, we shot ourselves historically with Scala which had the same promises.
Recreating programing language stuff, after telling us we don't need them.
Just as adding more routing functionality recently.
While others nod and think, "I did choose Go for some valid reasons, great library."
Imitation is the sincerest form of flattery that mediocrity can pay to greatness...
Im guessing that pretty much sums up how you feel. "Why isn't my tool more popular" is really the subtext of your question here.
A lot of GO devs came from java,ruby,python,php ... For 3 out of 4 of those static typing is an upgrade. For all of them compile to executable is a massive change and a good one. We're not moving code and runtime to a box to execute...
Its a breath of fresh air that if I need a tool I can quickly write it in go, compile it, throw it on to the server its needed on and be done with it. I dont have to instal vm/runtime/server to make that code work.
Dont think that matters? I have a Rpi camera server that is just a go binary running... All I did was wget it and turn it into a service.
Furthermore every one who is coming from those languages has seen upgrades go all sorts of directions.. php 4-5-6-7 were somewhat smooth but evolved the language, there is a way. Python 2 v 3 you can evolve a language and there is a BAD way. Python venvs are a pain... and ruby gems have their issues... Go's no breaking promise and packaging choices speaks to all the things that these devs lived through.
And the laments of "go lacks" or "too much boiler plate" are features not defects. Write dead simple, easy to read code. Deal with your errors on the spot in a pattern that everyone will recognize. Im not sure that I qualify Erlang/Elixir easy to read or "simple" (effective I will give it).
So yes, people are stealing OTP... take the compliment and go out and invent something else cool that we can "borrow".
Pick a language, any one, and there's probably a post by someone complaining about a bad design decision or a wart on its fuzzy ass... Nothing is perfect.
Yes go has flaws... I find that first class, high speed, testing makes the fact that I screw these things up all the time moot. Test, fix move on to the next project.
I just see too many comments on HN from people that have decades more experience than me that say that the theory of these amazing languages simply does not survive contact with the real world most of the time. I'm sure there are great things built with them somewhere out in the world. I just don't think its worth it for the vast majority of developers to worry about.
(amongst other languages!)
The other people that comment on it say the same thing. They use it for personal projects but can never make it work at a corp job.
I was a little bit unclear though in my previous comment. I mean production as in at work with lots of people besides yourself.
I can't count how often I've seen this:
Go: panic: runtime error: invalid memory address or nil pointer dereference
It definitely wasn't due to lack of previous tested technologies with field experience since the 1970's.
If Golang starts to add more and more features, it will lose this advantage and become just "yet another language". But worse: because of how was Golang designed, it will then end up worse than, say, Java. In other words, it will lose its advantages without really gaining much.
The problem is that this is probably inevitable, because the people that start as inexperienced developers with Golang will grow and become unsatisfied with the language features. Some more, some less, but that will be the general trend. And I think they are already vocal enough to push the things you listed.
Instead, I think it would be better for everyone if those people would instead switch programming languages and leave Go as is. Otherwise the cycle will repeat with the next Golang...
That a language is easier to use, doesn't mean it can't add features. The features are relative to how it helps the language be more useful and solve issues that users are having, so keeps it easier to use. Languages are always going to have pressure to adapt.
And the more features a language has and that a codebase uses, the longer it takes for someone to learn it and be productive (unless they can transfer knowledge from similar concepts of other languages of course).
Do you have a citation for that? As far as I remember that’s as most a second order effect -- they wanted a language that felt as clean and straightforward as C, to benefit everybody including experienced developers, not primarily inexperienced ones.
I do indeed have a citation for that by Rob Pike - and no matter how downvoted I am, it stays a fact:
> The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.
(https://bravenewgeek.com/go-is-unapologetically-flawed-heres...)
and also:
> Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical.
(https://go.dev/talks/2012/splash.article)
> As far as I remember that’s as most a second order effect -- they wanted a language that felt as clean and straightforward as C, to benefit everybody including experienced developers, not primarily inexperienced ones.
No, it's not a second order effect. Or rather: both what you say (a language "as clean and straightforward as C") as well as a language for devs early in their career (i.e. inexperienced) are both second order effects from the underlying reason that the language is targeted at developers at google.
Says anyone that has never tried to write portable C code during the UNIX wars days, or deal with UB across compiler toolchains.
Perhaps you're thinking of sum types? They are often confused as enums.
Go read Algol 60, Pascal, Modula-2, C manuals to understand what enumerations are all about.
Not a comic dance between iota and manually written constants.
From the GNU C manual: "An enumeration type represents a limited set of integer values, each with a name."
Which is exactly what Go provides. Which isn't surprising as it is identical to C in that regard.
this hack
type Operation int
const (
Escalated Operation = iota
Archived
Deleted
Completed
)
typedef enum {Escalated, Archived, Deleted, Completed} Operation;
Literally "Go enums suck", https://news.ycombinator.com/item?id=39564737
Enums suck, period. That's why many modern languages don't offer enums at all, providing sum types instead, which more accurately model what most people are trying to use enumerations for. It is an antiquated pattern that may have served a useful purpose in the early days of computing, but we've learned a thing or two about programming since then.
Well, at least you've come around to recognizing that Go does have enums, even if you needed someone else's words to say it. After all, Go enums can't suck if it doesn't have them. But, of course, Go purposefully tries to be antiquated. That's kind of its whole deal. Naturally it is going to have enums to really drive home the antiquatedness.
Unfortunely Rust wasn't mature enough in 2013, otherwise the main programing language of most CNCF projects would not have been Go.
Rust started as a research language by a PL guy working on his free time, he wasn't even suspecting his work would end up becoming mainstream. Rust only became somewhat serious several years after Go was released and used in production at Google. If you count Graydon's first Rust publication, you may as well count all of plan9 as part of Go's history! (Or how about using Go's repository first commit[0] if we're being facetious)
Also, implementing proper generics should have been straightforward from the start for Go, given that it was already mainstream in multiple programming languages back then (including the super mainstream Java and C#, with different approach). But Go was made by people who believe syntax highlighting is useless, of course they decided against generics…
[0] https://github.com/golang/go/commit/7d7c6a97f815e9279d08cfae...
-- https://go.googlesource.com/proposal/+/master/design/go2draf...
CLU was invented in 1976, C++ concepts were proposed in 2005!
Could they have done this earlier, in principle? Sure. But at what cost? What other balls would have been dropped? I think they were absolutely right to focus on aspects of the language design and tooling that they were confident about. Go is all the better for it. It's the polar opposite of a language designed to appeal to programming language enthusiasts; and well, that shows in the reactions.
By all means use Rust. I use Rust sometimes too, and it has its advantages over Go. Personally, I find generics in Go an occasional convenience and missed them only a little before 1.18.
- Use factory functions for struct initialization where members are pointers.
- Use errcheck and friends to ensure errors are not ignored.
- Never dereference the left hand return value if the right hand is an error.
This itself is a very concise argument for exceptions. Automatically checks errors are not ignored, and don't allow access to the left hand return value.
Making bubbling up even easier because people find this too verbose is the original mistake:
if err != nil {
return err
}
I'm all for ensuring errors are not silently ignored. That's Go's mistake. Exceptions are not the answer.
Maybe you're not like that, but you're the exception that proves the rule!
This is exactly what I meant. It's "handling", when viewing the consuming function as failing system. It's not "handling" when viewing the application as failing system.
I don't even particularly like exceptions either, I just like having to implement them by hand even less. Meanwhile, all the Smug Lisp Weenies™ are chuckling and mumbling something about "restarts".
What really kills me is that Go has a structured exception mechanism with panic/recover/defer, and it's actually better than try/catch/finally at that. It just doesn't seem to have caught on as the standard exception handling idiom.
Not only that, but there is heaps of code which is correct in the presence of errors (after all, Go developers tend to obsess a lot about them), but leaks resources in the presence of panics. Those just fly under the radar of most Go programmers.
Good thing Go introduced defer. Because the panic-safe era (similar to the exception-safe era of C++) seems to be an era yet to come for the Go ecosystem.
Which is exactly what you should do if you are in the game to build reliable complex systems. See my lengthy post here https://news.ycombinator.com/item?id=39808367
And yet, this is exactly how you learn to design dependable systems. You try to stay in fail early territory as long as possible (bubbling up errors, failing every system level on the way), until you reach a layer where fail stop is not acceptable (or log and throw is adviseable since you reached a top level system layer).
It's at this point where you introduce a resilience layer, sometimes using a library like failsafe.
The most horrible designs and buggy systems I've seen in my career where the result of trying to mask errors in random places, making it impossible to reason about any of the code, and leaking resources left and right. The most stable systems I've seen started from a place where _everything_ that went off script resulted in the most horrible stacktraces, but taking proper care about cleaning up resources, and improving from there. Often, the top level resilience layer would only be configured for retries or failovers in (pre-) production systems, so developers and testing would keep getting battered with stacktraces, providing a chance to fix stuff.
Maybe you could argue Rust does because it refuses to compile a ton of invalid cases, but I consider the time it takes me to write usable code (whether that's compiling or passing some runtime tests) an important metric.
You'll notice the go stdlib uses pointers for things which absolutely should not be nullable types, like '(*big.Int).Add(*big.Int, *big.Int)', or 'time.NewTimer() *time.Timer'.
'time.NewTimer' will never return a nil. 'big.Int.Add(x, y)' requires all of those things to be non-nil.
However, those are all pointers for unrelated reasons, like mutability, and so clearly pointers are not _just_ "nullable types".
I even wrote a blog post about this very thing:
https://wakatime.com/blog/48-go-desperately-needs-nil-safe-t...
Nullable types allow devs to write safe functions (functions that don't allow nullable types as input) and spread that safety through the whole codebase. That safety isn't even an option without having nullable types. Nullable types also mean the compiler knows and warns devs if they forgot to check for nil. There's really no reason why Go sholdn't have nullable types.
Don't you mean non-nullable types? Go does have nullable types, everything is nullable.
The blog post you linked calls them "non-nil types".
Like, if you already had type X that could be null, and you change the language so X can’t be null, now you need the type Nullable<X> (often written something like X?) to let it be null again.
C# and Swift are examples of this evolutionary process.
Null as a language concept is the issue since it's inherently flawed.
I want to use values over pointers for as much as I possibly can; the sort of stuff I work on is not so performance sensitive that it would be a problem. But zero-values are totally insufficient for modelling optional values, especially for numeric data.
Now it has generics I could implement my own Optional[T] type, but backup from the compiler forcing me to check a value exists before doing anything with it is what really makes optional types shine in languages that have it.