Wheel Reinventor’s Principles (2024)
tobloef.com
tobloef.com
Firstly, if you want to actually understand how the 'wheel' is invented then yes, you should re-invent it. The process of re-invention involves discovering what actually goes into some of the tools you use. Even if you never use your re-invented wheel in public (often advisable), the process of learning is invaluable in understanding the tools you do use.
Secondly, however, what wheel are we even talking about? The wheel is a timeless design, seemingly perfectly suited its task. The software libraries and tools that are usually picked as targets for 'not re-invention' are not wheels. They're higher level abstractions that pre-suppose certain ways of working. There's no timeless design here, just a bunch of arbitrary desicions about how something should work at a higher level with some amount of the decision making you'd have to do without it already done. Is this a bad thing? Of course not. But understanding that all of the 'wheels' are just this and are not magical black boxes that can't be understoor or shouldn't be looked at is important.
There are good times to not immediately go and re-implement *and publish* existing tools (emphasis on the publish, you should do things for learning), but understaning why you're choosing to do or NOT do a 'reinvention' is crucial.
If we could create physical things as easily as software, we’d absolutely see bicycle hobbyists and certainly little shops designing their own wheels.
Stone, then wood, then wood with spokes, then wood with spokes and iron trim, then we eventually added rubber, rubber tubing, then all metal spoke with rubber. For Mars rovers they made new types of air-less wheels.
The saying ‘do not reinvent the wheel’ is just silly
No, absolutely not. This is a first person problem.
The primary reason to reinvent wheels is to provide the most immediate and/or portable solution to a problem. By immediate I mean only from the perspective of the product.
That is a first person problem because many people cannot, such as neurological impairment, imagine any operating condition beyond the efforts of their own individual labor. That is where the cliche of not reinventing wheels is most used as an empty defensive argument.
Sometimes a simple bespoke solution minimizes costs compared to the complexity of using a massive hairball with a ton of power that you don't need.
One big caveat to this: there's a tendency to underestimate the cost and complexity of a solution that you, personally, developed. If new developers coming onto the project disagree, they're probably right.
There are plenty of things that look simpler than an established library at first glance (I/O of specialized formats comes to mind quickly). However, a lot of the complexity of that established library can wind up being edge cases that you actually _do_ care about, you just don't realize it yet.
It's easy to wind up blind to maintenance burden of "just a quick add to the in-house version" repeated over and over again until you wind up with something that has all of the complexities of the widely used library you were trying to avoid.
With that said, I still agree that it's good to write things from scratch and avoid complex dependencies where possible! I just think choosing the right cases to do so can be a bit of an art. It's a good one to hone.
The classic "I'll write my own csv parser - how hard can it be?"
I did as part of my work. It was easy.
To be very clear: the CSV files that are used are outputs from another tool, so they are much more "well-behaved" and "well-defined" (e.g. no escaping in particular for newlines; well-known separators; well-known encoding; ...) than many CSV files that you find on the internet.
On the other hand, some columns need a little bit of "special" handling (you could also do this as a post-processing step, but it is faster to be able to attach a handler to a column to do this handling directly during the parsing).
Under these circumstances (very well-behaved CSV files, but on the other hand wishing the capability to do some processing as part of the CSV reading), likely any existing library for parsing CSV would likely either be like taking a sledgehammer to crack a nut, or would have to be modified to suit the requirements.
So, writing a (very simple) own CSV reader implementation was the right choice.
You were incredibly lucky. I've never heard of anyone who insisted on integrating via CSV files who was also capable of consistently providing valid CSV files.
To be fair: problematic CSV files do occur. But for the functionality that the program provides, it suffices if in such a situation, an error message is shown to the user that helps him track down where the problem with the CSV file is. Or if the reading does not fail, the user can see in the visualization of the read data where the error with the CSV file was.
In other words: what is not expected is that the program gracefully has to
- automatically find out the "intended behaviour" (column separators, encoding, escaping, ...) of the CSV parsing,
- automatically correct incorrect input files.
And for anyone who's not convinced by CSV, consider parsing XML with a regex. "I don't need a full XML parser, I just need this little piece of data! Let's keep things lightweight. This can just be a regex..."
I've said it many times myself and been eventually burned by it each time. I'm not saying it's always wrong, but stop and think whether or not you can _really_ trust that "little piece of data" not to grow...
relevant:
> ruby-saml was using two different XML parsers during the code path of signature verification. Namely, REXML and Nokogiri
where "REXML" does exactly what you described, and hilarity ensued
Sign in as anyone: Bypassing SAML SSO authentication with parser differentials - https://news.ycombinator.com/item?id=43374519 - March 2025 (126 comments)
Once strings are in play, it quickly gets very hairy though, with quoting and escaping that's all over the place.
Badly formed, damaged, or truncated files are another caution area— are you allowed to bail, or required to? Is it up to your parser to flag when something looks hinky so a human can check it out? Or to make a judgment call about how hinky is hinky enough that the whole process needs to abort?
And if you implement your own, you get to choose the answers you want.
Regardless of the format if you're parsing something and encounter an error there are very few circumstances where the correct action is to return mangled dat.
It's not usually realistic to force your data source into compliance, nor is manually fixing it in between typically a worthwhile pursuit either.
Very well put and I would argue this applies to general software development. This is one of the biggest difference between my freshly-out-of-college self and me right now and something I try to teach engineer I'm trying to grow into "seniors".
Too many time have I seen a lot of wasted efforts on trying to build hyper flexible components ("this can do anything!") to support potential future use cases, which often never come to be. This typically results in components that don't do that much and/or are hard to use.
Design your components as simply as you need them, but no simpler. This typically gives more flexibility to grow rather than accounting for currently-not-needed use cases.
Keeping components simple, decoupled and with minimal dependencies allows for a high degree of optionality. When you pair this with a simple system that is easy to reason about - you're doing huge favours to your future self.
On the other hand, hanging off a bunch of unused features, especially ones that have interacting configuration - that's more like adding lead weights to your future self. It weighs you down and adds friction. And we tend to do a terrible job of predicting our future needs.
Kent Beck does a great job discussing the costs of these tradeoffs in his recent book "Tidy First". It builds upon the YAGNI principle, but adds a level of cost analysis that should allow you to sell these ideas to the managerial level.
But then as you start dealing with over-engineered system, you become intimately aware of the downsides of poorly abstracted system and you start becoming much more careful in your approach.
At least, that’s my pet theory.
I still sometimes use it as an allegory. As it originally shipped it had very low power density. It’s an unremarkable engine. But what it has in spades is potential. You can modify it to increase cylinder diameter, you can strap a giant header on it to improve volume and compression more. You can hang blowers and custom manifolds and more carbs off it to suck out more power. IIRC at the end of its reign they had people coaxing 3, almost 4 times the first gen OEM horsepower out of these things. They had turned it into a beast for that generation of “makers”.
Some people have an intuition for it, but it’s sort of an ineffable quality, buried in Best Practices in a way that is not particularly actionable.
So people having been scarred by past attempts to refactor code reach for the abstraction in fear, just in case, because they don’t know what else to do and it’s not written down anywhere, but abstractions are.
Refactoring tries to avoid this but the slope of that line can still end up being positive and you can’t refactor forever unless you’re very careful. And “very careful” is also not yet quantified.
Re-inventing things independently in parallel (parallel evolution analogies) is perhaps a strong indication that something interesting is going on. How do we know we got it "right" if we don't converge independently? If we invent a square wheel, and stopped because "wheel", we'd be in a horrible place. Science is a process, the process of reinventing is a great way to realize new things, and to train, at a low level, scientists. I suspect the process of re-inventing is also important in building out our (long term) ability to depend on our "gut feelings", thus providing the ability to nudge us to experiment along one path or another.
[Edit ... all things the article mentions.]
The logical conclusion of this mindset is mathematics, where people literally prove all of algebra and calculus to themselves as they learn it. There are good pedagogical reasons for doing this.
OTOH, if you're going to invent something every day anyway, it's really not that bad to reinvent the same thing more than once, even in the same year ;)
Can be more variety than watching the same movie more than once, plus you might even get more out of it the second time either way.
Now what hits me is when you don't even know if your goal is possible but it would be good if it was, and it seems maybe within reach more than other impossible things, it takes a lot of efforts like this before any pay off. You eventually get the feeling of what to pursue, and when to wind down if a fruitless pursuit is the kind that cannot be brought to bear.
Now with things like the wheel, you know it's possible to begin with, and quite popular already too, so that's a different starting point when you know it's a proven quantity of some kind, you're not taking the same kind of risks.
Then you have to consider if you eventually do make something difficult possible, chances are it's barely possible and kind of touchy. But at least then you know it's possible, that can be one of the best milestones right there. Probably still need to keep on inventing so you can make it even more possible, though.
I think it's most satisfying to have a mix of reinventions along with an ongoing effort on things that haven't been fully invented yet.
- Did YOU invent the last wheel, or any before it? If not, then you will make the same mistakes the last inventor made. Until you make a bunch of wheels, you'll probably suck at it.
- You learn more by studying old wheels than trying to bang one out yourself. Study the principles behind the designs rather than shooting from the hip. This is why we study medicine, science and engineering, and don't try to invent new medicines, sciences, and engineering disciplines from ignorance.
- Novel-ness is only good when it fixes more problems than it introduces. Novel-ness introduces not only "bugs" from a new, untested design, but also the problems of changing people's expectations, requiring new training, and possibly requiring all the other parts to change to fit the new wheel (which is often more work than just dealing with the old shitty wheel!). New things are not inherently good. Incremental change is almost always better, even if it's harder because you have to struggle with all the existing limitations. Your job isn't to do something easy, it's to make a product better.
- Only after you make your new wheel will you find out if it's good or not. Don't go into it assuming what you have is better just because you like the idea of it better. In fact, the more you like the idea, the more you should question it; ego is a dirty liar. Kill your darlings and be prepared to accept others' judgement of the thing, and the reality of it moving on the road.
Especially the last one is just a painful reality of the process. I think it's somewhat similar to the scientific method in that regard. Often your hypothesis is just false, but that does not make the attempt less valid.
There is so much nuance that doesn't get captured in all the study you can do about how a certain thing is made.
(I am a wheel reinventor btw)
*https://www.schneier.com/blog/archives/2015/05/amateurs_prod...
The boxes are always weirdly shaped with odd edges, and the holes they have to fill are always oddly shaped. The code written to join the two is at minimum glue that seals the two edges, but also usually involves converting one shape to another.
Then just wait until you speak with lawyers.
I thought I knew what those words meant but as I read the README I realize I don’t.
I am clearly not the intended audience for whatever that is.
If you're a PL/compiler/GC hacker, then here are wheels you should reinvent in order to even just have a basic idea of WTF is going on in the Big Serious Production Wheels that you might end up being gainfully employed to maintain:
- Invent your own language, and write at least an interpreter for it, to get a feel for what makes a language work at all, or not.
- Invent your own compiler IR. Don't worry if you make a bunch of mistakes. Don't worry about whether you follow my advice for how to do it, or anyone else's advice. Make it your own and learn from your mistakes.
- Invent your own way of doing the major compiler optimizations. Of course there are established ways of doing SSA conversion, CSE, constant prop, regalloc, instruction selection, etc. But you won't know why they are that way unless you try to make your own, and then either succeed because you are smarter than everyone else (it's possible that you are), or succeed because you literally reinvented the wheel (then you understand the compiler's wheels better than your friends because you got there from first principles), or you'll fail (most likely outcome) but then you'll understand why the real wheels work the way that they do better than others.
- Reinvent memory management. Write your own GC or whatever.
That's how I learned the craft. Can't think of a better way to learn.
When I tell people that file systems are antiquated and need to be replaced with something much better; I often get strong push back.
This is a wheel that I have been reinventing for some time. It's not something that can be fixed with minor tweaks.
This is a terribly simplistic take. To start off, it ignores the fact that major rewrites are known for failing.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
It also does not help that the only item you could come up to include in your risk matrix is... Someone else's bad will? Absurdly short-sighted.
The alternative, of course is to simply accept that all wheels already invented work just fine and don't need to be fixed.
Which approach is the most 'short-sighted'?
You can still organize things using folders, but you can query for things based off their tags as well.
find $HOMEBREW_PREFIX/opt/pnpm/libexec/lib/node_modules -type f | wc -l
1450
Unofficially I just wanted a decent filter engine[2] that worked well with my canvas library, and there were things like SVG "filter chaining" which I really liked the idea of - but could they be done in a simpler way[3]? Also: proving to nobody that canvas filters can be done (fairly) efficiently without the need for WebGL shaders (because: why not?). And then I discovered I really like coding up filter functions and went a bit mad with them[4][5] and now the hole is so deep the only option left for me is to keep digging ...
[1] - Though I think that's changing this year? May already have changed - but I'm not gonna un-reinvent my filters even if the Safari folks have shipped the fix!
[2] - https://github.com/KaliedaRik/Scrawl-canvas/blob/v8/source/h...
[3] - Why do the SVG filter primitives need to be so complicated to work with?
I would like to work with this person.
> I would like to work with this person.
Before you make such a bold claim, consider that this way of doing programming leads to an accumulation of knowledge that is barely transferable if you switch jobs.
I would claim that a central reason why many programmers like third-party dependencies is that being knowledgeable in these is a set of skills that is better marketable and transferable to jobs at other companies. In other words: applying these approach to programming can easily result in a career trap.
e.g. making your own static site generator is not reinventing the wheel. It's making your own wheel, which is a perfectly cromulent use of time.
why would you make a new glovebox. its got a latch, and a hinge.
maybe I dont even want to build a car.
This is a great way to learn why libraries and tools like compilers are the way they are.
I practiced this in the late 90s by making my own 3D maths library and my own “standard” library.
I ended up using the built-in 3D maths in DirectX and the C++ STL in the commercial game engine code I worked on later. But having practiced on my own libraries helped me understand the standard ones a lot better.
DYI?
For large, complex systems with multiple developers this approach rarely works.