Dependency injection in Go with Uber-go/fx
vincent.composieux.fr
vincent.composieux.fr
I've never worked in a large codebase that did this manually so I'm not sure what it looks like. The large codebases I've seen that don't use DI have used something like a service locator, singletons, or constructed everything where it was needed (and used extensive mocking framework functionality for testing).
Manual DI reveals problems like this way earlier.
DI frameworks, IMO, are some of the most useless around. Solution seeking a problem.
If it's 5000 lines and messy it is indeed a problem, but only because it's messy.
If you really want to use DI in golang, for some reason, wire[1] at least still gives you type safety and compile-time failures.
What does dependency injection give you that a simple combination of Singletons, Constructors, and Factories doesn't? I feel like the only thing you get is the ability to combine multiple independent dependency trees without having to make a sane structure. Kind of like, what redux does for state in JavaScript. It make things easier because it takes away some responsibility, but, in my eyes, that responsibility is an important one, and ignoring it makes code very hard to reason about.
I've never worked in a strict OOP paradigm, coming from Python and JavaScript mostly myself. I've been puzzled by his insistence that DI is important. He wants us to implement DI containers and take a "code to interfaces not implementations" approach. To me theres not really any value in this approach in JavaScript.
I don't know Go, but I get the impression this stuff is also not as valuable there.
My question is always "what does this get me and what does it cost me?". Outside of strict OOP paradigms like C# and Java, it's pretty unclear what the benefits are to me.
Which is probably true in some cases, but I find most of my code is pretty testable without it. Haven't run into any tests I've wanted to build where I couldn't.
I'm wondering if I have the terminology understood differently to other people, because to me, DI == IoC. DI Frameworks are build on top of the concept that dependencies will be injected, but doing it explicitly in your start-up code is the same thing to me.
This is the same conversation I had with the new hire I mentioned. And as the other comment here mentioned, it's JavaScript. You simply overwrite the object method with the mock at runtime of the test, then restore it after the test finishes running.
Most JS testing frameworks do this under the hood I believe, without any change in how you write your code. It's a fundamental difference in how JavaScript handles things versus something stricter like C#.
It's a dangerous footgun if you use it carelessly, of course. But a useful language feature when applied carefully.
My personal preference would still be to be explicit, to echew secret mutable state even in the tests, but that's just me. My personal baggage makes phrases like "then restore it after the test finishes" bring me out in a cold sweat ;)
I guess I sympathise with your colleague, but I agree it's not idiomatic, and not being idiomatic is not helpful in a team.
Basically, the logic not being inlined into the function is enough to provide this behavior in certain languages.
This is not true for other languages, which is why DI frameworks tend to be more useful there.
In go specifically, I just always throw the dependencies behind an interface and shove them into a struct. Then you just create a struct of mocks for your testing.
Because what I think of dependency injection is extremely common in golang - they made interfaces satisfy structurally rather than nominally so that consumers could specify interfaces which anyone is free to satisfy. That the consumer owns the interface (packages shouldn't export interfaces for concretes they implement) is a pretty core tenet, and goes hand in hand with good dependency injection (IoC).
DI frameworks are extremely rare (and IME even more painful to use), because injecting your dependencies explicitly is straightforward. But injected they should be.
Furthermore, DI frameworks also usually have lifecycle management, which is quite handy in many cases.
You have a good point wrt lifecycle management, but I feel like that’s actually a separate class of problem.
For example, rather than have to read the GoDoc for every single constructor i'm supposed to use, I can just see how to use my DI framework to set up this client library and move on with my life.
That being said, DI always strikes me as a layer of abstraction that I don't need in my day to day. But I don't work somewhere at Google/Uber scale so YMMV.
IME at tech megacorps, DI just obfuscates and distracts, shifting the focus onto understanding the accidental complexity it introduces instead of dealing with the intrinsic complexity of the dependency relationships.
You don't need to create a "simple combination" of Singletons, Constructors, and Factories.
Easy refactoring of an interface/constructor? I've been using go for a while, and I decided to follow the advise of not using a DI for my project. Every time I refactor a constructor, it is a fun hunt to make the same changes everywhere I'm using that constructor.
This is also what DI amounts to in practice. Frameworks abstract over it in the name of DRY, but at the same time introduce all the downsides of frameworks.
This is easy to see in frameworks like dagger, which compile time generate the boilerplate you could manually do.
And if everything is a singleton with no lifetime management, the framework doesn't buy you too much. The pattern, though, is kind of nice. I rarely have to question how a dependent section of code is linked to the one I'm at. (Contrast to python, where I don't know what is going to happen if I add that import...)
I felt that in my bones.
1. Complex lifecycle management, maybe. For example say you need to restart a set of go routines after loading a new configuration, without killing the process. I’m on the fence about this one.
2. When there is a combinatorially large number of components that need to be combined arbitrarily at runtime. The only examples I’ve seen for this in the wild are games and simulations using an ECS (entity-component-system) and queries to discover components that fit a certain criteria.
Unfortunately, thanks to enterprisey-OO zealotry, it's become a terrible monstrosity of frameworks, obfuscation-by-configurable-injection, and other terrible practices.
Every single application I've worked on where DI (in practice, not theory) is in use has been fragile, hard to debug, hard to test, and hard to maintain.
And this particular framework looks like it has all the markings of making any go application worse.
Topological sort and run. That’s all there is to dependency injection and I cannot imagine a graph so large that I would care for toposort
Which, yes, some lifecycles are confusing. But using them well can be very beneficial
From what I have learned working at various jobs it seems to be the feature and not bug in system design. When things break it is considered a problem worthy of attention of those numerous architecture astronauts. It would have just been college project if things work without any fuss.
Application wireup should be explicit, and if that results in “too much” code it means (in most cases) that the design is too complex.
[1]: https://youtu.be/MysHL0XYJeA?t=680
var ktx interface {
kacontext.Base
log.KAContext
datastore.KAContext
gqlclient.KAContext
web.AuthedServiceContext
web.AuthedUserContext
} = kacontext.Upgrade(ctx)
With this approach, you ask for the just the interfaces you need from the context and you have statically typed access to those resources.I expect the blog post will be live within a couple of weeks (but haven't seen a draft yet, so no guarantees): https://blog.khanacademy.org/engineering
It makes it incredibly easy to create "God structs" that can do everything. They have access to every service in your application when really you should be limiting the scope of them.
Unsatisfied dependencies are now a runtime issue, not a buildtime issue, one of my biggest gripes when working with IOC containers in .NET.
Not to mention that Context, originally built as a cancellation token, is now doubling as a service locator? It feels like something completely adjacent to what a Context is.
This is the exact sort of reflection magic code that a lot of Go developers dislike. And there's nothing that this gives you that passing dependencies as parameters can't. If you've got too many to pass then you're doing too much and you're not separating concerns.
Project Loom in Java will solve this problem in a much more superior manner.
If those struct members are typed to interfaces and not types, then it's just as testable, too, because you can drop in mocks as required.
What am I missing? This just seems like a way of obfuscating the fact that, in any modern program, we're going to need 3-20 "global variables" that aren't actually global global (but in practice are singletons in the process).
I've never understood the motivation for these kinds of tools. Have always come across as over engineered.
Took me a while to buy into it, because as you said, it doesn't seem any better than hand-written code.
But after a few years I noticed fx was being adopted by more and more teams, and it reduces cross-project contribution friction, some teams started building conventions around it, and more importantly, iterating on these conventions, etc. It's become just really handy for us.
This is many many small teams (5 people per team, more than 4k devs around many offices and countries).
import _ "example.com/feature"
Á la https://pkg.go.dev/net/http/pprofReads a heck of a lot easier!
I suppose elegance is in the eye of the beholder. FX is very elegant in my opinion: just specify your constructors and your needs, let the computer do the topo sort. But you’re right it is not idiomatic Go. Idiomatic Go is always to maximize the volume of rote, low-information-density code to accomplish any given task. Any time you are being clever and automating grunt work you are certainly violating the spirit of Go.
When you call a function that takes this struct as an argument, how do you know which members of the struct really need to be populated? Its dependencies aren't clearly documented. (They are over-specified. It's like a function with many unused arguments.)
If a subsystem only uses some members of the struct and you want to enforce that, maybe you pass in a smaller struct, or pass the struct members as separate arguments.
This is something you'll likely have to do if you want to extract a library for other people to use. Libraries are used in multiple systems, which might or might not have their own big struct.
Interfaces + duck typing? Your function shouldn’t ask for the struct, but rather for an interface describing what it needs.
Practically, though, I often just ask for the struct because I’m a slob.
So then it might be better for the function to declare its own interface with just the methods it uses? But then, all the callers need to be changed if you decide to call another method.
There's no principled solution to predicting what dependencies code might need someday. It's a matter of taste.
If your project scale is small, you don't need DI frameworks, but you should still use IoC so you don't have implicit singleton access and your tests get flaky and stupid.
If I miss something, I'd much rather know when I try to compile instead of waiting 'til a specific code path that uses a missing dependency fails.
The documentation for Fx, along with nearly all the applications I saw internally, use the dependency injection container only in main - once the application starts successfully, there's no more interaction with the container. For Uber at the time, this struck a useful balance between safety and the difficulty of distributing yet another versioned code gen tool to thousands of repositories.
It really does work really well in practice:
- It pretty simple and lightweight so it’s blazingly fast even though it happens at runtime (in contrast to e.g. the Java DI frameworks I’ve seen)
- The module concept is extremely powerful to make modules that plug in with zero effort
- Modules have a lot of autonomy (unlike wire, as I understand it - I have no personal experience with it), like being able to do things at various stages of the application lifecycle (startup, shutdown hooks), collaboratively populate dependency groups (e.g. implementing handlers or middlewares independently and injecting them separately using the grouping mechanism), optional dependencies
Once you have a nice standard library of common modules (this is really crucial for it to work well IMO), it’s a huge speedup to make a high-quality service. My biggest issue with it is that it doesn’t match structs with interfaces, so you effectively end up depending on structs/pointers or returning interfaces.
Does fx.As[0] help match structs to interfaces? It was added long after my time, but seems to target this problem.
At the time we wrote Fx, Uber had ~1500 engineers writing Go. The company had 25 million lines of Go, spread across more than a thousand microservices and an unknown number of shared libraries (likely hundreds, perhaps as many as a thousand). Nearly every project was in a separate git repository, with effectively no tools to make large cross-repository refactorings. As an engineering organization, we struggled to make relatively simple cross-cutting changes.
For example, we spent years rolling out distributed tracing. The actual change required was simple: upgrade all your dependencies to something recent-ish, add the tracing library, construct a tracer in main (or something main-adjacent), use the tracer to construct an RPC interceptor, and add the interceptor to your API server. All in, we're talking about ~20 lines of code and a dependency upgrade. It took multiple TPMs, spreadsheets, quarterly planning, and several high-level edicts to get this mostly done.
Why were changes so painful? At root, because nobody cared much about most of the Go repositories. From the perspective of the teams who nominally owned the code (often after several reorganizations over the years), the code worked fine and solved the business purpose - why invest time in changing anything? On the ground, changes were painful. Go's simplicity makes semantic versioning _very_ restrictive, so trying to pull in a year's worth of dependency updates often produced a variety of breakages. (Keep in mind that many of these libraries were used only in a handful of projects and weren't particularly carefully designed or maintained.) Taking on all this pain to change 20 lines of code in main was a difficult sell.
Fx codified some basic back-compat best practices (if you're paranoid) - mostly param and result structs, so constructors have more flexibility to add inputs and outputs. Fx also made most of these problems a negotiation directly between library authors, leaving the microservice team out of the picture: the "standard Uber stuff" package provides a distributed tracer, and the "RPC stuff" package takes an _optional_ tracer and installs the appropriate interceptor. No changes to main or application logic necessary, just a dependency update (which is hopefully safer, since more libraries are forced to follow better semver practices). The reflection-based wiring came with lots of magic and downsides, but the tradeoff was worth it across the engineering organization - it made us _overall_ more able to change our own systems.
Bluntly, IMO Fx made individual codebases less understandable (especially codebases carefully maintained by engineers who like Go). It made the whole company's code more maintainable. The bulk of the Go engineers at the company agreed (the developer experience org tracked NPS, which went from double-digit negative to +40ish).
In the years since Fx, I left Uber and the company has moved most of their Go to a monorepo. I'm not sure what the current cost/benefit tradeoff of this approach is.
Go doesn’t really have this problem so I’m not convinced it needs DI at all.
C++ doesn’t really have much DI mindshare because it has templates (and macros).
Goice (Guice for Go) anyone?
However I loved it so much that I then wrote my own Node DI package (property and constructor injection). It didn't take long before I abandoned the idea - Node doesn't really fit with DI.
And I feel the same about Go. Some languages (eg C#, Java) work amazing with DI. For some others (eg Node, Go) it simply feels wrong. I can't put my finger on why, but reading the sample Go code in the original article makes me feel how I do when (in film) I see a human body with a limb bent to an unnatural angle.
Just to clarify I'm all up for well designed IOC with Node and Go, I'm just unconvinced it should be done with DI.
It was OK -- quite good, really -- in small projects... but of course pretty much any organizing principle is workable in small projects. Good organization needs to scale up.
I'm not quite ready to write it off.
For one, those large projects used a framework. Perhaps the lack of friction helped lead to thoughtless injection.
And, of course, no organization scheme can prevent the spaghetti when the project and its leadership are disorganized.
So I'm not quite ready to write it off, but I'm awfully skeptical. And it certainly doesn't seem necessary.
That’s what is missing from the conversation: do the abstractions seem justified based on the problems it claims to solve.
Who knows, it could be as influential as jquery was for web development in 2006 (I’m exaggerating of course).
I just find the tone is generally dismissive and that robs viewers the chance to evaluate the tool within the problem space: DI injection for Go.
I dunno, maybe it'd be more useful for a more generic framework but I wouldn't use this in any of my code.