Opinion-Driven Design
stitcher.io
stitcher.io
This helped to clarify so much for me. I have done this so frequently not just with software that I've built, but websites, products, even romantic relationships. I can start to imagine different cases (and edge cases) and then eventually get stuck in an infinite loop trying to make it work for everyone.
One of the latest examples: trying to price a SaaS or ebook for friends/customers across different geographies and economic classes.
After reading this post, I felt relieved, maybe because of this line:
> So I started to prefer opinion-driven design when I realised that flexibility comes with a price that is often not worth paying.
Helped me realize just how much energy I've spent trying to keep options open.
This was known already by the pioneering software engineers in the 1960's. There are two things pulling us away from it:
- Business naturally wants to tick off as many boxes as possible on a feature list. This leads to software that purports to do many things, but do them all quite shittily. Like Deming, I believe this is a swindle to acquire many customers in the short term, but screwing them over and losing them long-term.
- There's some sense in trying to generalise the code, too: sometimes that leads to cleaner and more readable code. Often more easily testable code. Well, maybe. It's a very difficult point of judgment, and I personally get it wrong quite often.
Now, I don’t disagree with you. Maybe this organization is very beneficial. Typically systems with small, specific pieces that talk over a generic, well designed interface do work well. HTTP, Unix Pipes, currency, mail, bees... the whole becomes adaptive and accommodating.
This kind of design is discussed in the latter half of Elements of Clojure by Zachary Tellman.
https://elementsofclojure.com/
What I like about this, to come back to the article, is that it gives a clear sense of where generality should reside.
What's ironic about it is that about 1/2 the time developers correctly anticipate the general shape of the upcoming problem but it comes in a way they didn't expect thus they'd have done better by assuming it was never going to happen and dealing with it when it does.
This is a subtle distinction most people don't get.
I don't view it as a software development issue per se. Humans are just bad at predicting the future, bad at knowing how bad we are and bad at recognizing when we've been bad at it.
Trying to build a REST API coz the customer will definitely want it some day and thinking you've got a system at the slots in Vegas are two sides of the same coin.
And, I've learned the hard way that if a developer has got the same confidence that a granny at the slot machines has, preaching YAGNI isnt going to convince them they're wrong.
I added code and lines to handle edge cases - what if the XML prolog is not on the first line? What if it starts with whitespace?
But these are all edge cases that are unlikely to happen, and if they do... just let my application crash or error, it's not worth adding code until it's actually needed.
Same with performance, I'm thinking "I should add caching", but I don't even know how efficient my application is if it has to read & parse the files for every request - I'm assuming fast enough, until proven otherwise.
That's how I feel for my clients; I hammer in all the nails I can think of. The result is, usually I'm shocked by how few bugs there are in production. I've had business software running 24/7 that goes for years without any intervention. The occasional upgrade or minor UI fix. Error-proofing is the state of mind you're in when you write that parser and google for every edge case you can think of.
[edit] I just want to add another thought. There's such a thing as needing to ship. If you add every bell and whistle, every speed improvement and cache, you'll just keep going and never finish. The important thing is to think about these possibilities as you're writing the code, and document them so you know where the pain points might be if this software is still running much more intensively ten years down the road.
TL;DR I still think writing more options is better as a rigorous thought process, if not as actual code. If you must, hide them from the API consumer, but have them in the arsenal. (In itself, that's an opinion).
15 years ago, the opposite happened: working on unopinionated, complicated Java Enterprise Edition and app servers, that got partially steamrolled by Ruby On Rails (opinionated). That didn’t quite last in wide popularity but for a moment it was huge, and led to good reactions in the Java community like Spring Boot.
The moral of the story is that the market wants what it wants when it comes to “platform” software, or software that you use to help build other software: sometimes it’s opinionated software (like Apple and iOS), other times it is relatively unopinionated (like Android). It almost always is a fight between camps. There are many factors at play, not only developer desire: what outcomes does this software lead to, and who does it benefit? What new problems does the software create? These all have an impact on the reaction in the broader market.
It's also allowed to have weak, overridable opinions on relevant things, but as soon as it can't do some task that you need, its opinions have gone way too far and become a problem instead of a solution.
How does your web framework manage sessions? Who cares? it's not relevant. But what kinds of servers you can provision is quite an important opinion.
For example, companies that want strict ITIL processes and change control boards may not like software that makes some of those processes and ceremonies less relevant by de-risking large classes of change and opening up pre-approvals to a wider category of changes. This arguably saves a ton of time and money through reduced delays, and lowers risk, but it also lessens the power of the ITIL/change management team. So they resist this software. Who is right? It depends.
This is the trouble of opinionated software that tries to change the way people work (enforcing a new discipline): you can get a force multiplier of benefit, but you have to think the way the software wants you to. Some really can’t handle that. And sometimes the software causes more problems than it solves. But will you be able to get their on your own?
I liken it to diet and fitness programs. They can work, but it’s mostly about the discipline.
There's not a day that goes by where I'm using the "current stack", i.e. K8S, Terraform, etc., and not wish I was using Heroku or something Heroku-like instead.
In the community, from my POV anyway, the fact that Kube lacks the end-user experience of Heroku is irrelevant: what matters is how problems are solved: in the Kubernetes Way, or not at all. Some day developers may get their “cf push” or Heroku “git push”, built in Kubernetes, but it must be done “the right way” (no one knows what that is until they see it) to be accepted by the community and thus standardized. There are attempts at this today such as CNCF’s pack/buildpacks, Knative, or Openshift’s S2I… but none of that really has critical mass. The Heroku experience may never happen, but there’s a lot of money being spent to give it a shot.
Kubernetes is more than software, it is a movement. It is trying to drive a standard architecture of how to build and run distributed software with a set of (mostly) elegant design patterns and principals. The YAML overload is a temporary side effect of limited scope: the endgame is that every piece of server software is manageable as a CRD with an operator, and the greater Kubernetes API ecosystem subsumes all categories of cloud and server software.
It reminds me of past major tech waves like Java or Windows 95 or the Web itself: it sucks a lot of oxygen out of the conversation for a decade or so, and solves some problems while creating new ones.
Getting development dialed into Kubernetes usually also often takes several years, and often isn’t quite as productive. More choice, more potential for problems.
I can't count how many times I've asked a product owner "will this ever need to be able to do X?" and been assured that it would not, only to have X be requested months or even weeks later.
When I'm building something I focus on getting it to meet the immediate requirements, but when there are opportunities to make something more flexible/configurable/reusable without hours of extra work I take them. It's a constant balancing act.
People are very good at handling rules with one exception. All mammals are live birth except platypuses. All accounts have foo unless you live in California. Et cetera. Two implementations of a bit of logic is just a case of this.
When you get to three cases, you’ve lost the plot. You have to reformulate the question to restore the two-ness of the system (even if that’s splitting it into two twos), otherwise it doesn’t scale conceptually.
YAGNI isn’t a rule, it’s a dismissal. A dodge. A cop-out. As such, it gets misused, all the time. Anyone who wants to sound smart and be lazy will just say YAGNI, and it takes a lot of experience with a person to know whether they’re just lazy or lazy like a fox.
The Rule of Three is concrete. It contains a contract/escape hatch which prevents inaction from becoming unbounded.
Only when the abstractions proposed actually have a pragmatic use. As developers, for a lot of us, the immediate instinctual reaction is "add another layer of indirection". Each of these makes the software more complex, adds cognitive load and increases tech debt, making it a risky proposal.
Those risks are often glossed over for purely philosophical benefits and - for lack of a better term - personal taste. Maybe you don't actually need factories, strategy patterns or even DI just because the Gang of Four book mentions it. Maybe worry about problems when they occur, not because the might occur.
YAGNI is just a concrete implementation of KISS. I prefer the approach it proposes: question everything, say yes only when it makes sense. That is what YAGNI symbolises and reduces use cases, scope, feature creep and abstraction creep. For most companies, code isn't what's sold and merely enables (parts of) the business. What matters is whether it's fit for purpose, not whether it smells nice.
No, not really. YAGNI is about making it clear that you do not need to waste time putting together abstractions that aren't required to implement anything, and are nothing more than nice-to-haves that increase complexity but buy you nothing.
At best, YAGNI is a tradeoffs between the risk of piling up tech debt (and typically a very low risk) and improving the maintainability of your code base (because you avoid piling up cruft that you do not need).
That's not how it gets used most of the time now. It is rare that I hear someone use it the way it was meant to be used.
That's what YAGNI still is and always was. It never changed. At all.
Another thing that never changed are those who criticize YAGNI because they feel the abstraction they wish to add is somehow a must when it really isn't.
And when properly employed, is extremely effective.
The trick is to know when the A stands for "ain't", and when the A stands for "are".
Is there any difference at all?
They're complementary though.
I mean, let's be honest here: "you ain't gonna need it". That's what YAGNI stands for, and represents. That's what you say to anyone who claims "but it would be far more flexible if...". It's YAGNI, no matter how much makeup you want to slap onto it.
Libraries give more freedom and take a bit more effort to be wired together, but if you know what you are doing (which can be a big if) then they are more flexible and hence productive in the long-term.
Essentially, you could say that the "opioniated framework" that the author talks about is just a library that does one thing well.
If you are building a framework and its sub-components are frameworks that act on their own behalves, you better build exactly what you need in an opinionated style. But if you are building a framework and its sub-components are libraries, those libraries can be as general as possible, and your framework can then call those libraries in just the way that you need to. The library strategy will always out-compete in the end if you can handle it properly.
If you try to do it half-way like these developers did, you will be ruined just like they found.
Reposting from a similar comment yesterday -
> Frameworks are easier to setup initially, but they do not scale. Why is it that Microsoft Windows has 13 different dialog generations? Because each is a framework on top of a framework on top of a framework. It's amazing that they can even get that done.
> When the 2 UNIX devs were in a basement building UNIX and were able to out-compete Multics[0], they did it because they were building libraries that could talk with each-other using pipes around the boundary. Applications that communicate based on input-output with no internal state behave just like pure functions do. Pure functions compose. When Linus built git in 10 days, he was able to do this because the core idea of git isn't actually that much work. The framework is built out of composable blocks that neatly come together as libraries. Microsoft's TFS Source Control is a framework that acts on your behalf and therefore the bigger the project gets, you need n^2 people to work on it.
[0]: https://www.youtube.com/watch?v=3Ea3pkTCYx4, thanks to this HN comment(https://news.ycombinator.com/item?id=27494671) for this reference.
The other way, which I have personally seen, is that the core developers fight the pressures that lead to generality, but this has one of two side effects. Either non-core developers grow frustrated as they increasingly encounter problems that the framework handles poorly (or not at all) and need to choose solutions to fit the tool, or they just don't use that framework in the cases where it doesn't fit well. There are strong incentives to never choose the second option since it is either a design choice that makes the tool fit poorly or a difficult to describe edge case that requires a different solution. As an analogy, all problems can be solved with X language, so why would you ever need Y language? A framework is something approaching a domain specific language, so I feel like the analogy is appropriate.
People naturally fall into this trap of "one framework to rule them all" and what starts off lean and well suited to particular tasks ends up unwieldy and bloated because people aren't willing to pick up a different tool instead of just generalizing the tool they are using. Yet, the expression "one size fits all" naturally leads people to instead think that the fit is going to be ugly on most people nonetheless.
So if you want opinion driven development to succeed, you need to tell people what problems you are trying to make easier to solve and admit that there are similar problems for which the framework fits poorly and maybe another opinionated framework is a better choice. You need people that are willing to work with a wide variety of technologies.
This is one of the many areas where good judgement matters, and where good judgement cannot be bought through a collection of judgement-free rules. One reasonable rule of thumb, however, is that something intended to be general-purpose (like CSS) will probably need lots of options.
I'd much rather use a tool like that than one that shallowly appealed to every hypothetical user.
Whenever I take a project, I often find myself building in "what if" scenarios - e.g. what if the client asks for some further functionality down the road. I view my job as a programmer as trying to think of these things the client might not yet have conceived of, and leaving room for them in future iterations. I'll bring them up to the client and if they don't need them, I'll often write them in anyway. From the POV of putting out a concise, open-source package, I wouldn't do that. There's no need to expose half-baked functionality. But for myself, for my own work, I'll at least make a code stub and some notes every time I think of one of those things where an extra argument could be added, or extra functionality could branch off. At least half the time, clients come back years later asking for some functionality I stubbed, and I at least know where to look for it. If it seems useful enough, I merge it into the core.
So I don't think it's a bad impulse to try to think of every edge case. That's a great impulse. You just don't have to write the code for each one right now unless you think they'll impact daily operations. But if you were to lose the urge to do it, I'd argue you'd be a lesser programmer.
The truth is, no one will ever have to maintain them, because when I'm gone they'll just rebuild from scratch; they've already gotten their money's worth over the years of operation.
I think that accepting that some things are going to be bespoke or if not they will have limitations and just moving on with them is a key skill to develop.
If you’re going to write software, know the “Why” behind the software and follow through on that intent. This is especially true for internal software like a framework — the marketing department won’t be there to keep reminding you why your customers are your customers if your customers are your colleagues.
The intent can change. For example, your intent might change from “Hey I just noticed I can do this thing and want to explore why it might be useful” to “this framework helps a programmer avoid missing data dependencies”.