The Code It Yourself Manifesto (2016)
pestilenz.org
pestilenz.org
I was once heavily afflicted by NIH, but with experience I realized how much time I was wasting stubbornly trying to reinvent the wheel. That’s actually a great analogy if you think about it—-the original wheels were fashioned out of wood and are quite simple in concept, but creating modern wheels require s tons of specialized tools and knowledge to build. It doesn’t make sense to attempt this if I can just buy a premade one that suits my needs. That premade wheel also has the benefit of huge amounts of iteration in response to problems encountered over time, and I would probably need to replicate at least some of that in order to build something competitive with the existing solution.
In the realm of code, using other people’s solutions when available lets me focus on the original problem I want to solve. It also allows me to minimize the amount of code I need to actually maintain myself, since typically there’s a group of people that are collectively much more knowledgeable than me doing that for free. If I ever do have a reason to know how the library works (patches, bug fixes, behavioral questions, curiosity), I’ve found it much quicker to figure out how the code works rather than write my own.
You actually need to change things for it to be called an invention:
http://move.rupy.se/file/wheel.jpg
That said I always write everything from scratch, to me it's the meaning of life;
If you are only using things you don't understand, you cannot (re-)invent anything.
Today we also do not own anything, which makes the problem even worse.
Rules for a good life: 1) own 2) understand 3) change!
I now have a complete grasp of the file format, and the code base. I will never need to find a library for it, or have a dependency, the code wont change without me knowing about it, and I can very accurately make decisions about when using PNG is the right format. I can speak about the format with authority.
All this have a lot of long time value.
It only took me about 2 weeks and it was one of my most memorable experiences of the year. I learned a few great data structures, I learned about JS microoptimization (I got an 8x speed up from the first port to the final version of the code). And I learned all sorts of practical physics - moments of inertia, rotational momentum, restitution, solvers, etc.
It was a silly thing to do by most standards, but acts like that make me a much better engineer.
Similarly, finding authoritative comparisons of which image formats do what best is pretty easy.
I've also never run up against shortcomings of existing PNG solutions. Even in assembly for the Z80 I remember using a PNG converter that would handle greyscale on a calculator!
But 3 or 4 days of work! Mam I can think of a lot of things I could do with that.
If it was for the fun of the experience I'd totally get it, but spinning it as an "I needed to do this", I don't think I could go that far in the areas I've worked in
If you look at my codebase (www.gamepipeline.org) you can see that it looks more similar to a platform holder like Microsoft/Apple/Google, then what most individuals github profiles.
The most common critique of my coding ethos is that its not efective (I write everything in C and I have Zero dependencies, so I write everything from scratch). But looking at what i have been able to accomplice over the years, I cant think of many people who have produced as many applications as I have (http://www.quelsolaar.com). If writing everything in higher level languages and using lots of dependencies was as effective as people say, then there would be plenty of people running circles around me in terms of productivity, and I just don see that.
I strongly disagree with this. I believe the exact opposite is true. Creation spurts from learning, so if you want to be creative it pays of to spend most of your time learning things that already exist. In the context of programming, this means implementing algorithms for which several other implementations already exist.
Sportsmen spend nearly all of their time training, and just a tiny (but very focused) amount of time competing. Likewise, scientists spend most of their time preparing experiments and reproducing results of others, and a tiny amount of time "creating". I do not believe that programmers can successfully escape this scheme. The best programmers I know, spend an inordinate amount of time rewriting well known algorithms (and oftentimes, improving them with slight variations).
I mean, on one's own time, this seems like rather a good thing to do.
But to say "Yeah, instead of something that works now 0 days work let's spend 3 months building something that may not work very well and we have to maintain" - is not a reasonable premise in most scenarios.
It defies the very point of trade specialization upon literally which the economy is built.
So while we should always be learning, learning in and of itself in most instances is not a good enough reason to do something at least at work.
I find that strong opinions either way on this question to be indicative of time preference. My own view is there's a time to do it and there's a time to avoid it. It becomes very valuable when ones navigating an unfamiliar area or is in the midst of a fundamental shift in the landscape. It's best to avoid it if the team or project is an a do-or-die scenario where certain milestones must be met to ensure it remains a viable team or project.
This is an interesting example of Conway's Law. If I hire a contractor for something, I'll want them to just import something that works, not try to explore things deeply. If I have a long term employee or cofounder, I'll want to encourage that experimentation and growth.
Earlier in my career I could just use any old tool that got the job done with plenty of dependencies, without a care in the world. Then three things happened. First, I got exposed to some of the highest quality tools and libraries in the industry. As in, the ones that only a few people are lucky enough to get to use, and it made using anything less polished afterwards painful. Sort of like living in a mansion and then needing to move back into a slum. The other thing was just finding out about too many of the nasty corner cases that exist in technology, which in most cases are safe to ignore, but are hard to unthink about once you know about them. The third was just having too many things I depended on break over time or abandon the principles that attracted me to the project in the first place.
So now, later in my career, my biases are more tuned towards reinventing things than ever before. I'd rather minimize risk by having something perfectly tailored to my specific needs. If it breaks, I'll only have myself to blame. I view that as a fun learning experience. Much better than the alternative, which is nagging folks in the open source community and filing issues expecting them to support me, and then feeling guilty afterwards. Only thing I'm worried about is that my standings will continue to grow so high that the joy of programming is so hard to find that I'll just do management, and just let other people have fun with all the churn.
Namely, the amount of understanding and effort needed to solve my problem correctly and safely...is frequently less than the amount of understanding and effort needed to use some opinionated library or framework.
Not always; there are plenty of times a library or framework is a better choice. But what tilts the scale in that direction is making them very simple, with minimal assumptions, minimal state, and simple APIs into them, because that can keep it simpler than writing myself.
Of course is their choice and if they enjoy reinventing the wheel good for them. But they'll spend 6 to 18 months working on reproducing some fraction of the features they'd have for free instead of working on what makes their thing unique.
An egregious example of what I'm referring to is what we saw with the left pad library. That always blew my mind because the amount of effort to find a library, learn it, include it, and use it, seemed really high for the functionality. I don't understand why it saw such adoption - I wouldn't even -think- to Google that functionality as a library; I'd just write it (if I was unaware of the string method, which to be fair, I only know of because of the leftpad kerfuffle).
A game engine is an example where learning the framework is assuredly going to be faster; you'd need a better reason to write your own, and as a dev manager I'd be pushing back on it hard (whether it was coming from the devs or the business).
If any Hackernews wondered why anyone would want to live and do all their work from within Emacs, this alone would explain it.
Sometimes the most extreme NIH tendencies are perfectly justified due to these constraints, and sometimes they're absurd. I don't think any point on the continuum is generally "correct" nor should be generally advocated for. Instead we should be discussing how to figure out where the point lies on the continuum, within the context of the software we're currently writing.
Unlike actual wheels, the ability to quickly and easily tear down, reconfigure, and rebuild my digital tool box is the defining hallmark of software.
- download a project in a suitable language that claims to do what you need
- read its list of direct dependencies
- implement your own thing using those same libraries
- write your own replacement for any of those libraries that seem more trouble than they're worth.
Indications for when this is likely to go well:
- the top-level piece of software has grown its own configuration language
- the the top-level piece of software has dozens of options for tuning its behaviour
- the maintainers of the top-level piece of software have spun parts of it out as libraries, and they're being used by other people.
At least thanks to tree-shaking and bundler innovations, the end-users don't suffer from bloated bundle downloads, but your node_modules folder is still 600MB large.
For example, I've chosen to replace one of the foundational data communication building blocks [1] because, after extensive research, the existing systems can't be modified to handle all of the general use cases I want handled. And it won't stop there, because I also have a number of additional requirements that HTTP won't handle (so there's also a new protocol in the works that will be based off this technology). If I could have simply modified an existing system to do what I want, I certainly wouldn't have spent the last two years on this! I've got plenty of other things demanding my attention...
Your time is generally better spent working on solving your core problem rather than the dozens of ancillary problems that end up needing to be solved along the way (particularly where a whole bunch of other people have spent a whole bunch of time already solving those problems).
Yes, but a large complicated well maintained and widely used library is not necessarily preferable to a small not so complicated library that does exactly what you need and nothing else. And that goes for formats too.
Recently I was involved in a project where order numbers had to be sent from one system to another. Some colleagues insisted that we baked them into a large xml document and then used libraries to both create the documents as well as parse them. In this case the economic thing to do was to write them each separated by EOL. Even the code we would have written ourselves would have been larger if we’d used the XML solution, not to talk about everything needed to include in builds and deploys.
You should always pick the simplest solution to the problem that meets all your requirements, but when considering solutions you should favor standards compliant solutions. A common example is date formats. Lots of places roll their own date format string when sending dates, but using ISO-8601 will save you (and your clients) so many headaches in the long run.
Honestly for your example, not knowing all the details I can't say for sure if a EOL separated value is a good solution, but based on just the description I probably would have gone with a CSV, or possibly a JSON array. I definitely would not have used XML (dear god, why would anyone pick XML in this day and age?), although if they were concerned about needing to add more data down the line I could maybe see an argument for something a bit more involved than a CSV.
Other times, though, either I learn something, or end up making something that is much better for me to use. I would suggest at some time every programmer try build their own limited scope library if they find existing solutions do not meet their requirements.
If the code is for you, and for you exclusively, go ahead.
If you're writing code as part of a team for a customer, then it isn't for you and it's whole purpose is to solve the problem at hand.
Secondly:
The primary issue I have with a NIH code-it-yourself approach is that it doesn't scale over time. Professionally, over two decades, I have seen several teams go through a technical evaluation and decide, in the end, that no open or closed source solution exactly fit their needs. So they coded it themselves.
Fast forward three to five years and everyone regretted the NIH approach. Those open or closed source solutions had matured and easily surpassed the home grown feature set, which still required a team of engineers to invest in.
There are exceptions, of course. Sometimes you have to build it yourself. But more often than not, it's much, much more effective to let go of your ego and collaborate with others, particularly on open source solutions in which you always have the option to fork the codebase and bring it, effectively, in house.
Generally, I find those that strongly advocate for NIH overly discount the long term costs of maintaining software.
This is the root of the problem: a small team generally can't keep up with an open source project over time.
The NIH approach works well at larger companies who can dedicate a lot of engineers to working on in-house infrastructure projects full time. At smaller companies it is a distraction from building the core product.
> mean[s] endless seeking, evaluating and further deviation from our goals.
is simply not true. Only if software were written to serve ultra-particular and individual goals would that be the case.
Software is written to serve needs - of fewer or of many. And while it may serve a more constrained set of needs more optimally, it typically serves wide enough needs well enough, that the vast majority of people have most of their software needs met by software written by others (albeit with room for improvement).
Also, almost no person, even a proficient coder, has enough time and attention span to code most of the software they use. On the contrary, we absolutely and necessary _won't and can't_ code "it" ourselves, where "it" is the main bulk of software we use over the course of our lives.
----
Instead of this manifesto, I would suggest a "Write good, robust, widely-usable libraries" manifesto - because that's how other people will be realistically able to code "it" themselves when and if they need to.
First, I find something which suits me.
Then, it starts growing from under me, typically in the direction of bloat and feature removal.
Then, the usefulness to abuse ratio drops gradually.
Then, I'm faced with having to migrate or simply abandon yet another platform.
It's a serious issue, but I believe we can overcome it with just this sort of approach combining FLOSS and dogfooding.
After dealing with the same sort of rot pattern in both software and services for years, I've set out to replace as much as I can with my own tooling.
Mostly this has meant developing a hybrid blogging, forum, note-taking and writing, DAG database, information archiving and retrieval system and dogfooding it as much as possible.
As always, it all boils down to cost. "Code it yourself" may make your programmers happy, and give them interesting work to do, but if it costs the business a big chunk of money they could have saved or put to more profitable efforts, it's a very bad idea.
Of which there isn't much to go round, and very much of which would go down the drain if one were to follow that philosophy to get things done.
Oh, and ... there's also all those pesky people who don't really know how to code.
A lot of tools will be fine or meet your goals, such as your OS. If your OS doesn't match your goals then you're left with the three options detailed. In the case of an OS, the "big" OS's probably meet most people's general computing needs. It's not uncommon to write a custom "OS" for embedded devices. Some people do feel like making their own OS and there are plenty of examples out there of them and that's perfectly fine.
Anything like this is also not written as law. Your text editor annoys you under these certain conditions? You don't have to rewrite it, fix it, or switch to another editor. If you're compelled to write a replacement, it shouldn't be stigmatized which rewriting existing software frequently is.
We are a social animal - "zoon politicon" in Plato's original Greek - and our activities are social. For better and for worse, we can't declare that not to be the case or claim that we want to just be "left alone".