In Defense of Not-Invented-Here Syndrome (2001)
joelonsoftware.com
joelonsoftware.com
I worked somewhere in the 90s that had built its own database infrastructure to meet business requirements that existed in the early/mid 80s. If you set out to build the same system in 1995 that they'd built before, you absolutely would NOT use the same approach, but they plodded along with a mishmash of internally developed tech for a long, long time -- long enough that it really hurt them. And the first place that this damage showed up was in personnel. To be productive there, you had to know a ton about the internal tools that existed ONLY at 5251 Westheimer (<-- if you know, you know), which meant those skills were useless in the marketplace that was rapidly moving towards open-standards based development, etc., or at least using commercial databases.
It got hard to hire anybody but fresh kids, most of whom would split after 12-24 months.
So even if you do build your own C compiler or whatever, you have to keep an eye out for offramps to that internal dependency if the realities around your product or project change. In this sense, your internal dependency is no different than an external one -- you've always got to be looking for an alternative even if you're perfectly happy with the current path, because the current path might stop being viable.
Frankly, I think the article, like a lot of Joel Spolsky's writing, has aged like milk. For example:
If you’re developing a computer game where the plot is your competitive
advantage, it’s OK to use a third party 3D library. But if cool 3D effects
are going to be your distinguishing feature, you had better roll your own.
The short version of the article is "don't outsource your core competency" and that very much holds true today.
Factorio or The Witness might be better examples of games that did/are doing novel, exceptional things and their engines were developed in-house. And it shows. The unique look and feel, the smoothness, depth, coherency etc.
The comparison is maybe unfair though. We're basically comparing a product focus with an art focus. Artists _must_ do things their way. Risk and cost are secondary. Success is a bonus. In a more survival oriented world you have to weigh these things differently.
What you describe above seems more like a problem with clinging to legacy and missing the world around them rather than developing things in-house. It should be obvious, but the thing you do in-house has to provide you unique advantages over the main-stream. That comes with costs and only works if you have the stomach to be honest with yourself.
I dont think The Witness really required a custom engine, except that Mr. Blow had a team that could (and it might've been easier for them). Ditto with factorio tbh.
If a game like https://en.wikipedia.org/wiki/Antichamber could be done in unreal engine, most other games could've been done with it. The fact is, understanding the engine is harder than writing your own, and for some programmers, they'd rather write their own simply because they won't have to read someone else's code.
On the one hand, there was no attempt to generate your DB code. OTOH, since you had to write it yourself, it was easy to tune.
100% agree with this take re: gaming, but his overall point -- if your value-add in the marketplace is X, you'd probably better build your own X -- is more true than not. Games blur that, but at the end of the day nobody buys games PURELY for the 3D side of things. They're looking for story and gameplay at least as much, and (for me at least) probably MORE than technical beauty.
Is TeleCheck really all that mysterious?
When were you there?
The other side of the coin here is that a team with some NIH code (if it is well engineered, modern, maybe open sourced or on its way to open sourcing) can use it to attract and retain certain engineers who might want to eg work on building a database or a compiler.
Now of course it's a pretty fine balance but I've worked at one non tech company that had basically no internal software projects whatsoever, and their engineering culture was abysmal. I think in that case they couldn't have made good software anyway but the fact that everything they did was pieced together from a handful of commercial vendors meant that there was no interesting code to write at all! For example, we were encouraged to report bugs to upstream paid support but never attempt to fix them, because they didn't want to deal with internal versions of software or open source licensing. I didn't stay for very long.
It's a balancing act and the balance is different based on the size of the companies and the specific product.
- can read the code, and
- is good at this technology, and
- has the domain specific knowledge to work on it,
but also someone who will be part of your team.
Further, NIH only works out well when youre not under strong pressure. Someone tells you to build XYZ, and you have NIH syndrome and build dependencies yourself, you will do so under a lot of time pressure, and it will be shit no matter how good you are.
It can go both ways though. You have the opposite of NIH, and say "there's no way in hell I have time to build this", so you pull in some huge unwieldy library that mostly solves the problem, and end up spending more time jamming a circle into a square hole and plugging up the gaps than you would have spent just solving the problem.
This is a good time to push back on requirements unless there's real money waiting to pay for that feature.
Usually things are not completely established up front, so you never have a good picture of things you're going to have to build with some dependency. In these situations, your choice of software and how well it supports future features may as well be a dice roll.
It's definitely a judgement call, and that's where having the experience to make that call correctly can save a ton of headache down the road. Some times YJGWI (You Just Gotta Write It).
Maybe it's from a place of privilege, but I have no interest in working where the team is not competent.
But, even if your team is incompetent, there's no guarantee of competence of the people who wrote the dependency you use instead of building it yourself. Most of the time, I'd rather deal with local incompetence than remote incompetence.
At the end of the day, if you want your project to work, you're responsible for the whole thing, include the dependencies you outsourced to others.
> there's no guarantee of competence of the people who wrote the dependency
This is, in my opinion, the best reason to build stuff in-house. Thank you!
Kind of funny to read this, now that the practice is no longer a fantasy. I'm thinking of devs who have outsourced their jobs, and Amazon/Etsy sellers who simply resell/drop-ship goods from Alibaba.
Find a cheaper way to do something Find a better way to market something
He wasn't wrong. End of the day, these kind of 'businesses' are merely exploiting a market inefficiency that won't last forever.
One of the issues with being a middleman without a moat is that there's downward pressure on your margin over time until you are squeezed out of the market.
Given the amount of ads I get encouraging people to become a Amazon drop-shipper, I assume that making money doing that has got hard enough that selling shovels became more profitable.
In other words, it's more profitable teaching people to become Amazon drop-shippers than it is to be one.
1) I generally only need 10% of the library's functionality, and the rest is just extra weight.
2) I don't want to have to choose between freezing an old version of a library or having to update my code to integrate a new version.
3) For things that are not cryptography, I can usually write it better myself.
4) Third-party libraries typically do not prioritize accessibility or compatibility nearly as much as I do.
5) I have a much better sense of what the code is doing this way.
Edit: As jedberg helpfully pointed out, this post is not advice, and not intended to be treated as such.
We've accepted specialization in most aspects of modern day society as a good thing. I fail to see, outside a small subset of programming, how you will always be able to write better code yourself for whatever domain you're working on.
> 5) I have a much better sense of what the code is doing this way.
This is not scalable.
Through practice.
But also, through realizing that when you look closely, many libraries are crap inside.
> This is not scalable.
I specialize in things which do not scale.
To the uninformed what your are doing looks insane. I think people project their fear of the "black box" that is "libraries" and forget that mere mortals like themselves wrote these tools to begin with.
The hard part is knowing when / what to re-use, and most importantly when you are ACTUALLY skilled enough to do it better and it makes sense to do so. There's a confluence of prevailing winds that makes this strategy possible or not possible.
This is what expertise looks like: it isn't Googlable, nor is it ChatGPTable.
You can get a rough idea of the quality of most libraries in about 10-20m typically, usually by playing with the library or browsing the code. Some red flags I look for: lack of tests, excessive number of commits, excessive number of dependencies, lack of focus in scope, etc.
One alternative I use frequently: lock dependencies up in a module, and write some glue code intended to act as the interface to that dependency. It is sometimes the case in which you have to use a library but it is not very good, or it has a horrible bug you can correct by exerting more precise control over how the library is called.
It's strange that using fewer third-party dependencies has to be defended on a site called Hacker News.
Also, the code you NIH doesn't have to be superior to the third party code. It merely has to not get in the way for what you're doing. This is different from being "better," which has many dimensions to it. Example that comes to mind: I could reach for something like libuv, or I could just use POSIX for synchronization primitives. I'd prefer to the see the latter in most code unless there were explicit reasons for libuv, such as cross-platform support.
You need to have a working model of how a library works to use it in a performant way, which is why most libraries document their architecture, and have usage guidance.
And you'll find bugs, because there are always bugs. You need some strategy for what to do when that happens.
Fix them, I imagine. Which is easier if you're better at writing code than the author of whatever someone threw on (package manager of choice).
And if you don't, it sounds like you'll never get your flywheel going so that you can stop working or work on something else.
I'm mostly saying, to people reading this, it would probably be unwise to follow this advice.
I'd say this is the approach used by anyone in the top 5% (give or take). Which is probably most of HN. You don't need to be Linus or Fabrice or DJB to know most programmers are bad at programming and realise you can do a better job.
Does that mean he writes his own kernel and C compiler? No, OP writes:
> I avoid using third-party libraries for anything I can write myself.
Which shows awareness of his current limits, another trait of good programmers.
Grabbing a quick javascript library to do something relatively trivial that is going to bring in megabytes of supporting libraries whose provenance is unknown and that change often is a completely different matter.
If we wanted to NIH all the things, we'd have to start doing crazy shit like implementing our own JSON serializers or hand-rolling artisanal CSPRNG/SHA algorithms. We'd almost have to want to make our lives harder on purpose. There's no way we could make it look like it was in pursuit of a grander product outcome when the obvious answer is already being pushed on us by intellisense. We could try to lie to ourselves about all of this or paint various vendors in pale hues of evil, but none of this is fun or feels like much progress.
Starting with something that includes nearly 100% of the batteries takes a lot of the temptation out of the way. You won't even begin the dangerous "should I write it or vendor it" adventure because you will be so distracted with all of the features you are shipping.
These two are not analogous. Writing your own JSON deserializer/serializer can absolutely make sense depending on what you are doing. You'd better have a very good reason for writing your own sha algorithm.
Any sufficiently gifted programmer will reach a point where they realize that there is no third party tool that is quite right for the specific problem they are solving. The art is in identifying where exactly the line is. Personally, I tend to fall in the batteries not included camp but this only works in a high skill/trust environment, which frankly is not companies. You're probably right that in most cases preferring batteries included is a better default disposition. But almost definitionally these types of places will tend not to be technologically innovative.
I mostly agree with you, but to me this is presented as a logically-inconsistent argument.
Why is writing my own JSON serializer relatively better than writing my own SHA implementation? Don't we first need to look at the actual problem? Why does it have to be a very good reason, as opposed to simply the same quality of reasoning used for the JSON decision?
It's funny he mentions this. I know someone who makes millions a year doing exactly this. Her value add is finding vendors, finding drop shippers, and marketing the products. But she doesn't build any tech or hold or make any inventory.
if you are making money, you are adding value. You are putting products in front of people who want them at an attractive price for them. If you make the products from scratch and do that, or if you drag them the full length of the silk road, or if you buy a pack of cigarettes and stand outside and sell them as loosies, if people are buying from you, you're putting in your effort and making money which is the measure of value-add. Convenience stores can make money right next to lower priced supermarkets because walking accross the parking lot and to the back of the supermarket to get a quart of milk can feel like the silk road.
edit: downvoted for explaining how actual economics works? If you haven't studied economics, you shouldn't be making your feelings known, it's not a matter of opinion or preference, it's a field of study.
how is value-added tax calculated? on your feelings, or simply on how much money you make? QED
But now that the venues manage their own secondary sales, the scalper is just extracting rent.
My sister was coming to visit me in NYC, she's a huge Red Sox fan, I bought her and her friends Red Sox Yankees tickets, they had a blast. I was happy to have done it. The tickets were expensive. I got value for my money.
ticket scalpers are no different than stock brokers, I go to them when I want to buy shares of Tesla, and I have to pay the market price. They are the people who are willing to sit there and wait for my transaction without knowing if I'll ever show up, an activity that I don't want to engage in and I'm happy to pay them for it.
Yeah, you have a weak moat, but if your niche isn’t big then even a weak moat may not be worth crossing
I was already answering "it depends" on three of them.
> 1. Code Reuse is:
"Code reuse" sounds good, but if you abuse DRY ("Don't Repeat Yourself") then you can easily get into a mess that would have been more maintainable if it had been separate structs/classes/functions/modules in the first place, even if there's some duplicate code.
My rule of thumb is, if the purposes are different, then they should be separate things, even if the code is exactly the same line-for-line. When (if) it becomes obvious that we're always updating the code on multiple sides, then we can start considering combining them into only one thing.
It more or less aligns with the Rule of Three.
> 2. Reinventing the Wheel is:
It sounds "bad" because there's apparently "duplicate effort that you could have saved if you just used this library", but you have to remember that now you're at the mercy of this library, and unless you're paying for it, the maintainers can do whatever they want with it, including removing it or even using the name for a different thing (e.g. "tree-analyzer" being initially something related to ASTs (Abstract Syntax Trees), but then being re-purposed as an OpenCV tool to identify tree species (plants) from pictures).
And also following the wheel analogy, we can't forget that reinventing the wheel was necessary because modern cars can't use wheels destined for horse carriages; and airplanes can't use wheels used by cars. It wasn't just a matter of making an existing wheel thicker, or just using a different material; the whole build process is different.
> 3. The Not-Invented-Here Syndrome is:
The article sums this up nicely: "If it’s a core business function — do it yourself, no matter what."
He was bucking the prevailing wisdom a bit. We had gone from cowboy coder teams building everything themselves, to highly structured/abstract code bases. "Don't build if you can buy, don't roll your own if there's an existing tool, if you do build a tool, make sure you make it in a way that's reusable." Etc.
Here, Joel was saying that there are exceptions to the rule. That better teams should consider these exceptions before just leaning on the conventional wisdom.
I think in the decades since this article was first written, his way of thinking has become more common.
When I'm choosing a new tool, I always look at if a solution already exists, and make a focused effort on determining all ongoing costs, such as maintenance, adding new features, managing the vendor relationship, etc. My preference is always to buy if it's something another company might need, because then I get the benefits of the features for that other company, but tempered with what my internal cost to build would be (including maintenance and adding new features).
On the other hand, for enterprise software, there's usually some seriously non-trivial integration cost - particularly if you're not starting from a green field. So the problem is "you can't buy a solution - you just need to decide how much of it you're going to build yourself according to your needs, and how much you're going to buy and tailor/integrate."
Except that the most valuable company in the world has consistently tried to own or control every part of their value chain, including businesses they had little experience with and probably no "core competency" in. And that's not an outcome of them having so much cash they can go into chip design, tv production, software tools etc. Steve Jobs would have made his own microprocessor in the '90s if he had the resources. Maybe that's the exception that proves the rule?
Cost me some credit. Now I am trying to get us away from the awful mess we produced.
I hate that some builds pull in gigabytes of libraries for everything under the sun to replicate what was done 20 years ago with a few scripts and some HTML.
Yeah, it's a spectrum and not everyone is qualified to make the right call - but their job demands they choose anyway.
Within the team yes; but my experience with management has been anything "Not-Invented-Here" is considered infallible and practically divine. Anything done in-house is subjected to the deepest level of scrutiny imaginable.
Always has seemed like its due to who they can pass the blame off to - If my team screwed up I'm liable, if an external vendor screws up we had no control over them.
a) it's difficult
b) our engineers want to develop the expertise themselves
Sometimes there are good reasons for b) but we often miss the boat.
(a) understand what it does
(b) research the available alternatives
(c) check that you're happy with its dependencies
(d) deal with any licensing issues
(e) check for known issues and bugs
(f) if possible, take it for a test run
which is likely to be a nontrivial amount of work. If it's less effort to just write it, you may be better off just writing it.
One of the smartest things you can do in software development. In my experience, 99% of problems are caused by dependencies. Code that I wrote 20 years ago still works today. With no drama, no upgrade issues, no political BS, no waiting forever to get a simple bug fixed etc. The more dependencies I remove, the more reliable and maintainable my software is.
For example: I am currently the maintainer of very large scale mission critical software written in C++ used by enterprises around the world. And I have reduced dependencies to almost zero. The result is zero bugs in production the last 5+ years. And zero maintenance pain. The result is that I spend close to 100% of my time adding features, improving performance etc. and close to 0% of my time dealing with dependency bugs/issues/open source politics/BS etc.
It’s a great place to be!
"If it’s a core business function — do it yourself, no matter what". The question is, what is "Core Business Function"?
If you were an internal tech company in 2000s, hosting your own servers was a "core business function", as being a tech company meant that tech was core business function. However, as tech got more sophisticated, the domain of "core function" became more focused per company
Another example is the rise of fabless semiconductor companies. Even until recently, Intel had superior chip design and fabrication capabilities (thus intel's tick-tock strategy). AMD founder Jerry Sanders claimed "Real men have fabs" but AMD's spinoff of GlobalFoundaries look like a genius move now.
What you think of "core business function" is going to change over time.
I'm pretty sure now that most businesses / tech companies are glued components with a core idea and a lot of marketing attached to it.
We have a modern version of this - people that aren't selling cloud services running their own Kubernetes farms rather than just buying cheap k8s or whatever other orchestration solution they like from a cloud hosting provider.
There was nothing wrong with the first 150 line version (that was done in an hour or so) There are existing 3000ish line solutions too. They do many wonderful things that I don't need. I even had someone else write a version!
[Only] when it is done it will be added to the monolith of previous fumbles. It has a few things implemented quickly that will eventually get some careful reconsideration.
I'm fairly confident the final version will be a joy to look at for my future self. There wont be tests and there wont be any bugs.
I remember hearing all about years ago how the domestic Japanese software market was behind that of the US in some ways, because companies just don't trust software from foreign companies. I don't know if it is still true, but among them was more of a focus on customer service and software maintenance, and as well the aforementioned tendency for software to work better with English.
Most of those components are written by people who don't seem to understand HTML and CSS.
I worked on an app where a third party component was used to show a box, with some initials in it, and a background color. It was five divs.
Excuse me WHAT? Is there a list of goofy MS decisions somewhere?
You could learn to properly use ANSI C, or you could write your own compiler instead for some subset of C and pretend like youre smart for it.
It would make sense why MSVC has pretty much no C support, and why so many MS products are so uniquely bad, instead of being bad in the same ways (uniformly)
One of the worst things about the modern internet is people can be so rude and disrespectful to people they've never met for decisions they know nothing about. The custom compiler was used in the 80s and early 90s for P-Code and was a way of dealing with the perceived proliferation of multiple CPU architectures, as well as producing significantly more dense code. This mattered a lot back then.
MS-DOS was written in assembly, and very stripped down and small, but office apps needed to be somewhat fatter with luxurious features so memory was an issue. Microsoft was working on a C compiler but wasn't selling it as a product (although, did they license Lattice C for awhile?) Ultimately, the cs compiler was added to Microsoft's C compiler as a p-code option. Ultimately ultimately people stopped caring about small memory and the utility of pcode went away.
This led them to build Trello though, so maybe not a bad outcome?
Words to live by
> Pick your core business competencies and goals, and do those in house
Say it ain't so, Joe(l)! Never meet your heroes, I guess...
I've always enjoyed his take on things. I'm very much a "dependency skeptic."
As I watch disaster after disaster happen, fueled by The Dependapocalypse, I haven't seen much to dissuade me.
Sure, there's plenty of really awesome stuff out there, like Linux, Git, STL, etc., but there's also a lot of truly awful toxic sludge; much of it wrapped in sexy, jargon-filled Web sites, and with large user communities.
Part of my problem, is that I'm not really good at saying "Fuck it, let's ship," when the project is still unripe. I know that I've left a lot of money on the table, with that attitude, and have received torrents of scorn, as a result.
But. I. Just. Can't. Ship. Crap.
I just can't bear it. Call it a pathology.
It's difficult to truly vet a dependency. I suspect most people look for the biggest crowd of users, and a glitzy presentation.
Oh, and probably "free," as well.
I find that doing a lot of research on a library or SDK, takes a lot of really boring footwork, and time not spent writing sexy software.
Also, if I will be basing a business, or my reputation, on the work of others, I had better be prepared to pay for it; or at least, roll up my sleeves, and work hard for it.
> the market pays for value added
I'm not so sure about that. It seems that the market pays for garbage, and that a lot of people have gotten quite wealthy, selling it.
The market pays for whatever solves their problem, even if it is garbage.
What the "hype-driven hustle" modern software has forgotten, is that the market would pay even more for software that solves their problem AND is not garbage. People value quality. 2023 software engineers don't, they just value time-to-ship, so we have garbage software everywhere valued billions, because people need it.
Want to disrupt any niche? Make really good software. Put power users first. Or at least this is the philosophy I strive to follow, we'll see if it pays off.
Do not ship crap. But also recognise you'll never be truly happy with the most polished of gems either. This is a very hard skill to learn, and there's few role models to follow.
I really want you to be right.
I've found that high Quality tends to be fairly pricey. There's a reason a Mercedes costs three times as much as a Toyota, and the car doesn't seem to have many more features.
Incremental improvements in Quality can mean a great deal of extra time and detailing.
Wasn't Apple once the paragon for software quality? Didn't people flock to it when it was still incredibly expensive because it was objectively better made than the competition? I remember wishing my parents were richer and would buy me a Macintosh instead of having to use a PC.
A Ferrari costs more than a Toyota, but you have to decide if you want to make a car that sells a lot, or a premium car. Can't have both, but these days in software they're all making Ladas, and no one is shipping Mercedes, let alone a Ferrari.
I saw the company I worked for (once synonymous with "Quality") take a big hit, when they started making money, and selling in the millions, but they also got clobbered, and I hope they are going back to their Quality roots (although it may be too late).
This is why I said we lack the role models. The philosophy of "ship, then fix" and VCs demanding a quick RoI creates a culture of Cooks. With all due respect to the guy, he is not a visionary nor an idealist.