Inversion of Control Containers and the Dependency Injection pattern (2004)
martinfowler.com
martinfowler.com
Though the funniest part about it is that the things we use every day, programming languages and build systems, still do not generally use this concept. Your imports are dependencies, you should just pass them in. Wham bam, no more environment variables, ambiguity, lookups, multiple version ambiguity, rewiring challenges, naming conflicts, etc.
Well, you still need somebody to know so they can instantiate the correct dependency versions in the first place (which you could implement as a snazzy automatic dependency resolver), but all of the internal ambiguitys disappear. Replacing a library (with a drop-in compatible one) would be just instantiating a different codebase into the same symbol name which will get passed in exactly the same way as the old one.
No.
Dependency Injection means using tooling to "inject" the dependency, instead of passing an explicit argument.
https://github.com/google/guice
> Think of Guice's `@Inject` as the new `new`.
Like many architecutral design patterns, toy examples don't illustrate the concept well. It's used for doing things like switching a large service-based system from using external databases and remote services for production, to in-memory everything for testing.
This is a common misconception. Guice’s docs delineate between dependency injection as a pattern and Guice as a framework that supports that pattern.
2. Oh boy, this is the first time I have seen Guice and it is exactly what I am talking about with languages not understanding the concept and having to be worked around with utter hacks. If you just had symbols you could bind constructors/types to and then pass those around into your implementations you could get rid of most of the common cases they mention in their docs.
You know, if the DB is passed as a parameter, I can do that too. Without @inject, or XML, or the god container object.
const testdb = new InMemoryDB();
const result = await myThing(testdb);
expect(result.foo).toBe(42);
expect(testdb.rows).toEqual(bar);There is a point where software diving down meets the hardware coming up. When you import a library you start creating harder to separate internals and testing becomes more of a blackbox approach (I don't care what happens inside as long as my results are consistent) as opposed to unit and function. It eventually does reach a point where dependencies are harder wired but the deeper you can create this DI, the more dynamic things become (function arguments, library injection, os virtualization, even hardware at points [PCI-E, serial]).
I think you are right about DI being less common but that is because it's not a natural reaction until you reach the maintenance side of software. It's much easier (and more performant) to just load it up on startup and call it directly. When you get into the way languages work you will see DI is integral to their accessibility and maintenance.
I remember when I first discovered the pattern, at a job I took 10 years after this article. The codebase seemed so clean, easy to evolve and test, even for a newbie and 'half programmer' as me - my mind was blown. I was thinking - what kind of black magic is this ?!
Say you're new to programming. What's the best place today to learn about design patterns (with python examples, since it's the most used language for beginners) in an easy and accessible way ?
I love this part of the journey for folks because it opens the door to a world of possibilities that has dramatically more structure and form. Thoughts and ideas become less "what if.." and more puzzling together the best interactions between certain design implementations.
At risk of rattling on endlessly in my excitement for you, TDD became a very interesting and enjoyable way (for me at least) to implement these patterns and gain a better understanding. I think you'll find Martin Fowler, TDD, and IoC/DI are peas in a pod.
(I realize you said Python and this was Java. I'll see if I can find something Python friendly. If you'd like to try your hand with Java....I highly highly recommend Spring Framework and more specifically Spring Boot. Spring is an IoC Container Framework where the Dependency Injection is done for you on the fly. So much to throw at someone but if you'd like to connect on it, I'd be more than happy to give you a running start.)
Testing Specifically: https://exercism.org/docs/tracks/python/tests
I really enjoyed working with my engineers on this site. A lot has changed since I used it last but the idea being it gives you small bite size challenges to test and exercise testing muscles.
Below turned into a bit of an impassioned rant/soapbox. I still wanted to share in case it offers any support for you on your journey.
You are absolutely right. I spent a good portion of my life testing code on a "line coverage" basis to satisfy management. It was hard to see the value in it when I viewed it as this necessary evil.
I think it is hard because we are often left boiling an ocean of a problem when trying to build software or needing to implement "this thing" before I can do "that thing". Where do I even start? The setup eventually becomes so much that it really brings into question whether it was all worth it.
One thing I can say is, at least for me, TDD was something I had to see done well before I could start doing it. I was never strong enough to understand from an article, book, or video. I paid an expert to pair with me who, himself, got wrapped around the axle trying to implement the simplest of structures.
Ultimately it came down to me working with some folks who had exercised the "muscles" of implementing the practice and had several tools that made the process feasible. Intellij for Java was core, shortcuts, having one side of my screen the test, the other side implementation. Autorunning tests, a simple pipeline that tested, built, etc.
I just needed to see what good looked like to at least start forming an opinion on what was possible for me. Eventually I learned to take smaller and smaller chunks at a time. You want to write an API, sure. Let's just send a {curl equivalent in your preferred language} to this endpoint and get a 200 OK response. Alright, now let's make sure this header is present.
Try not to solve for the future as much as addressing what is in front of you at the moment, or maybe the next couple hours. You'll eventually notice common traits APIs have, database objects have, business logic you care about vs layers that don't belong to you and you can't really control anyway. There will be times that future issues will require you to build certain ways and you just won't know until you've been there a few times.
I learned to approach things with what they called the Triple A Pattern (AAA) and each test had these three blocks of behavior:
Arrange - Create all my objects, initialize, stub out skeletons I've maybe required parameters that I don't actually care about.
Act - Execute my function, catch my object.
Assert - Test that whatever I received is true, equal to "someVal" etc.
There's honestly so much more to say and I glossed over so much. In the beginning I can distinctly remember this existential pain of pulling my repo from git and my environment not running, some cached dependency or browser session breaking things. I constantly felt like I wasn't understanding but I was ultimately learning how much I didn't know about the toolchain, the language itself, the editor I was using and I was finally getting a very clear understanding because I was looking for a very specific response. I started learning how to say, "Based on what I understand, I expect this to happen..." If I'm surprised, then I'm on a new adventure or maybe need to regroup.
I think people believe TDD is supposed to be some panacea but it's only as good as the consistency someone leverages it with. Someone I really respected sold me when they said:
"It just felt nice making some changes and knowing these things I tested for wouldn't pop up again in prod."
That dread of 3am and my name being pulled into a Sev1 call because something was crashing over something simple (and it happened a few times) gave me anxiety constantly. You almost forget how much pressure you feel weighing you down until you leave that team, project, etc. It was a very freeing proposition that had the added bonus of not getting gaslit by people who didn't know what they were saying.
I hope my experience offers some value as I really enjoyed sharing it.
I will give the website a shot and report back.
https://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compens...
It was put up or shut up. If you said you could do it you can almost guarantee someone would ask you to show them and explain.
A lot of "knowing something is not the same as knowing the name of something"
Examples are in C# but it's a great book about OOP in general.
An example of why certain patterns aren't necessary in Python:
I have two types of repositories, one that wraps my API calls, and one that wraps an in-memory database for unit testing. A module called repos.py chooses which of these to instantiate when it is loaded based off an environment variable.
Because every module in Python is loaded once, every file/class that needs a repository for data access can import from repos.py and receive an instantiated repository object.
There is no need for dependency injection because the dependency can be imported from a module, and the module can be modified at runtime. This is extremely powerful too, since, so long as my in-memory repository and API repository have the same behavior, I can write a test suite that doubles as both unit tests and integration tests, all I need to do is change a single environment variable.
I think Java's limitation of "one class one file" and no support for first-class functions made design patterns necessary. Then you try out a language with some functional features and you're like "oh wait, that was just a workaround for what other languages have always been able to do".
The other thing is that Python doesn't actually seem like a good language to discuss these abstractions compared to Java or C#: it's dynamically/gradually typed and in general dependencies are handled in a more "UNIX"-y way. In particular Python doesn't have first-class interfaces (at least last I checked, maybe mypy has something?). But statically-checked first-class interfaces are critical for why e.g. the service locator pattern works "seamlessly"[2] in real-life Java/C# codebases with lots of people doing lots of different things.
My gut is that a lot of inversion-of-control design patterns don't actually work so well in practical Python compared to just throwing things in a dictionary/etc and using strings + careful unit testing. I do see there are some service locator Python packages so maybe I am being too closed-minded. But for teaching C#/Java is probably a better choice.
[1] https://news.ycombinator.com/item?id=39459732
[2] I have been mystified by too many null pointers to say this with a straight face. But when the bugs are squashed it really does manage a "magical" level of complexity.
In other news, ten years on the game's been up a while now, Fowler's profusion of wordage plastered onto simple functional concept, eg. in the present case: partial function application, doesn't really fly so well, even in the enterprise.
This also created a bit of a spaghetti nightmare, where it was very hard to reason about which classes (services) were being used by what other services. The Inversion of Control article (later renamed/aliased Dependency Injection) was a reaction to this: it inverted the control away from the using site to the code creating the services.
(I appreciate the article does talk about Service Locator, but I think it's worthwhile having an understanding of the normalities of the time when reading it.)
I feel like there's a general feeling that:
> It is just passing dependencys as arguments.
(And: oh no, magic!)
That's often what consumers see, sure.
But it's also managed repositories of objects and factories. It's also injection logic, with scope and conditional logic. It's also runtime assembleable interception layers for objects. It's also awareness methods to observe and track your managed repositories & objects.
There's simple things we see. But zooming out from the trees I think the forest here is something more than the sun of these pieces. Programs are mostly here to juggle entities of various sorts. Mostly we programmers write code that imperatively creates & modifier objects & their references...
And Dependency Injection and Inversion of Control is having explicit tooling & stable patterns for this construction. Having actual places & tools for managing runtime objects.
I think it's so damned cool. It's so enticing to me, of being expanded beyond a programmer tool. It could & should pioneer waves of general systems research! This tech should offer external hooks beyond the app, be integrated into repls & hosting apis that let us talk to the computing objects of our world.
Invert the control! I cannot wait for us to stop doing all coding bottom up, instantiating and passing everything from the bottom.up, and starting to create some top down tools that manage the entities of the runtime!
Let's say I want to write small functions that do one thing very well:
args = {LLM parameters, e.g., prompt}
1. foo(args): calls the LLM API with args
2. goo (foo(args)): uses backoff (retry) and makes sure the output is JSON
3. hoo(goo(foo(args))): extracts and desers the JSON
At any point, things could go wrong. I could use a monadic approach and turn each of these functions into a monad: 1. foo: args -> Maybe(API_res)
2. goo: Maybe(API_res) -> Maybe(JSON)
3. hoo: Maybe(JSON) -> Maybe(dict)
But before I knew about monads, I thought: wouldn't be cool if when a function goes wrong and needs to be called again, it had access to its "parent" function which called it? Like:Currently args is only passed to foo. What if, depending on how things went wrong, goo needed to see args?
One approach is to make foo pass its args as well:
foo: args -> args, output
Then goo would take that and do something with it. But what if now hoo also needed to know about args to make sure the extracted JSON conforms to the JSON schema mentioned in args? Now we'd have to do: foo: args -> args, output
goo: args, output -> args, output, JSON
goo: args, output, JSON -> dict (deserialized JSON)
I think this is not "elegant". Is there any better solution?goo should take foo as argument and delegate to the command in its loop
hoo shouldn't be a Maybe. if you want that function then lift hoo instead of writing it, that's fmap hoo
Honestly I think this is the most important pattern in my line of work. It has become so common for me. That when I see c# code not using it. It almost looks like another language.
Testing becomes so easy because everything is easy to mock out. Be it in a unit test using some mock framework. Or a real 'integration-test' where you write a small in memory queue mock by hand to easy your tests.
Object lifetime is easy to change and manage. Just how you register the dependency. As scoped (to http request for example) as transient (every time you ask) or singleton?
Want to do something before/after with some hook? Just Decorate the dependency and register it to the container.
Nothing feels tangled. Just inject the service. Call the method. Do the thing. No need to worry about cleanup or how to create the dependency. It's already setup.
On the other hand
While I see the value of DI containers in web apps (ASP.NET)
Then I don't really see the value of it in console apps (tools) (yea, asp app is console too, ik.)
You may ask what's the difference and in my opinion the difference is: input/output flow model
In web app you run your app with some args/envs, and then you're receiving request and return them results.
In console app your run your app with some args/envs and then you're doing stuff until you complete (ofc unless you're listening/waiting for things... like in web app)
In this 2nd model DI containers seem to not give me anything useful. You even were writing about this here:
>"Just how you register the dependency. As scoped (to http request for example) as transient (every time you ask) or singleton?"
It fits asp request/response model very well, but for other things? I dont feel it
ASP web framework creates instances of Controllers (handlers) on request basis, that's when they need to resolve dependencies in order to create them, so it fits it very well.
But when I need to create instance of some repository, some background job handler and some csv writer, then why would I want to do it via DI container?
>DI
I'm talking about containers, just containers.
DI != DI containers
> why would I want to do it via DI container
I guess I don't follow why you wouldn't. Like, already assuming we are competent people, then you extend your object graph when it makes sense.
Does a prototype bean for {new ArrayList()} make sense? Probably not. But I could definitely see wiring up a CSVWriter with config properties from a container populated from various resources.
I buy the simple argument that DI is a better "new" and containers are just the interface to an named object graph.