Stop future-proofing software
medium.com
medium.com
The problem with taking the experience that "future proofing" is never worth it, is that it suffers from survival bias, which is to say that when you did it and it made the future event a non-event, it didn't even register on your brain. If you practice more mindful engineering you can teach yourself to recognize the things you did in the past that made what you are doing today easy or even possible.
That is not to say that all future proofing is good, especially if it ties up the group in knots trying to predict all possible futures. That is where the mantra of keeping it simple really helps. Simplicity is the greatest protection against the future precisely because it has the smallest disruption surface. The simpler a process or product is, the more difficult it is to find a change that would invalidate its effectiveness.
It means going out of your way to try and handle scenarios that you don't know will occur, or whose details you don't really understand yet, with clever one-off facilities.
E.g., I've come into a couple projects now, where I'm supposed to add in some new feature, and I discover that the original author has thoughtfully added a bunch of extension points to try and help me with that. Both times, I also discovered that building in those extension points accounted for a large amount of the pre-existing complexity in the code, even though they weren't actually being used for anything yet. And both times I also found that none of them actually addressed my needs, so all that extra effort and complexity was a waste. One time I was unfortunate in that the existing code was of the "600 line functions and a whole mess of mutable state and temporal coupling" variety, so changing it would have incurred excessive risk, and I opted for just jamming my square peg into that round hole. The second time it was well-factored code, so it was easy enough to rip the round holes out to make room for square ones. But it would have been even less effort if I could have skipped that cleanup step and just jumped straight to adding the extension points.
Second phase of a developer:
2. The philosophizing abstracter
"This code works for now, but if I move this part into a factory, and create an interface for these methods, it'll also support all these future cases I can think of!"
They've read all the articles on how to structure code. There's a lot of patterns out there for object-oriented (or functional!) code. Their code has all sorts of useful interfaces, abstraction layers, factories, extension methods, data structures.
Things break down when the project needs to move in an unexpected direction and it turns out that implementing the change requires changing a lot of the codebase because there were a bit too many abstractions that ended up creating hidden dependencies across the code.
Simple systems require extreme vigilance to remain simple.
I think this is the most important sentence in the article. I've worked on a few extremely abstracted and horribly-indirect codebases that happened to be so very flexible and extensible in exactly all the wrong directions, so much that it made it much harder to change in the way that was actually required.
Specific example from TFA: distributed app vs. large server. A distributed app might give you better scalability, which you might not need because large servers are cheap. However, a distributed app also forces you to think about state, so if you screw something up and have to fix it middle of the day, you can BG deploy the app instead of having a 4 minute outage. It forces you to modularize the logic and think about interfaces, so if you need to add, remove or change functionality then it'll likely be faster.
Etc.
What this article forgets to say is: "Stop future-proofing your software if you are still looking for a good product-market fit. As soon as you know what your business is going to be, make sure your technology stack is flexible."
As a form of devil's advocacy - Why? Just improving the codebase under the hood won't bring any new users in the initial market segment where your product-market fit is proven. Focusing on marketing, will...
Having a "flexible" code base (to me) means being able to change the code easily. That change could be to add new functionality, or to refactor the design, or both. "Flexible" (to me) does not necessarily include having actual facilities to do anything.
As an example, your normal "Hello, world!" C program is very, flexible. It's easy to modify. A similar "Hello, world!" written using a large framework is less flexible because you have more constraints on what you can do, and how you can change the design, even though it has more facilities.
Adding flexibility does not mean adding more facilities to the code. Often it means removing unused facilities from the code in order to make it simpler. IMHO, this is the true distinction with YAGNI. We remove YAGNI code in order to increase the flexibility of the code. Sometimes people are tempted to write wonderful elaborate designs for their code in order to ensure that it's easy to do something in the future. This is a classic case of YAGNI. We want to remove that code in order to increase flexibility (because often new requirements move in a direction that is opposed to the original design).
However, I sometimes wish that there was a cute acronym for "I Actually Need It Now" (suggestions welcomed :-)). In this case, let's say we've written some code and it's awkward, but it does the trick. A little while later, we end up doing the same thing. It's relatively easy to copy and paste our previous awkward code. Then we have to do it again. In those cases, you've found that you do need it, and you're adding complexity to your code base every time you copy and paste the same awkward code. It's beneficial to build a nice system to make it easy to do it. Next time, not only will it be quick to add, but it won't add unnecessary complexity to the code base.
You've probably heard of the "rule of 3". Of course, it's a rule of thumb, so you have to use your own judgement, but the idea is that if you have to do it once, then just do it. You can't generalise what problem you are actually solving because you just did it once. There's not much sense in agonising over the "ultimate" design, because you are likely to get it wrong anyway. But by the time you've done it 3 times, you've got a pretty good idea of where this is going. At that time you should invest in an appropriate solution. The solution will likely increase the architectural complexity, but it will simplify the interfaces and will slow down the increase in overall complexity.
Again, 3 times is a rule of thumb. Sometimes you know right away that you need something more complex. Sometimes it takes you a lot more than 3 times before you can really wrap your head around what direction is best. So a better rule is: delay making design decisions until you have enough data to answer your questions.
And to roll it all back to the beginning, YAGNI happens when you make decisions before you have the data to support your decision making. This can be both with user features and with design. By delaying these decisions until we have good data, we keep our options open and the system remains "flexible". But if we do not make the decisions when we do have enough data, the system becomes complex because it lacks cohesion. We have to act promptly to address those issues, or else we will have a very uneven development experience. "Hard" things may take a short amount of time, because we didn't build all the infrastructure we needed. "Easy" things may take a long amount of time because we suddenly need to build the infrastructure that we didn't build before.
And that's what I mean by "amortising" the expense of feature development. We don't spend a lot of time up front to build a complex, but inflexible system. However, we spend time intermittently, to maintain flexibility and to keep the overall cost down. Even more importantly, businesses depend on the predictability of development. By maintaining a high degree of flexibility, we allow the cost of solving problems to approximate the complexity of the problem. Without having a flexible code base, the cost of solving the problem is often dominated by the complexity of the code base instead of the complexity of the problem. This leads to management not being able to "trust" development and often leads to project cancellation.
Another aside: I often say that upper management is the most dangerous part of the team for a project because they are the ones that cancel the project. If they do not have a good feel for a project, they may very well cancel it for extremely poor reasons. Thus, aligning the project to the expectations of upper management is one of the most important parts of software development.
I personally don't like doing development work that isn't tied to direct economic value. This includes refactoring. Refactoring may contribute to indirect economic value (by reducing development time int he future), but it is risky work. As I mentioned in the previous paragraph, upper management depends on the predictability of development. If you randomly (from their perspective) say, "We're going to do some feature sized work, but it's not going to result in any different functionality", it separates them from the logic of the development. They may tolerate it, but it makes planning difficult.
Instead, I prefer to spread that kind of work out and to "same size" the work. This is harder to do from a design perspective because you have to find small goals to achieve and to work piecemeal. You need to have good communication with your team and good buy-in for the process, so that each team member takes the appropriate opportunities every time they touch the code. In short, it makes the development process more difficult, but the advantages are many. In that way, I do exactly what you suggest: I only work on code that is adding customer value. But I intentionally take a little bit of extra time in each story to find ways to improve the flexibility of the code. I also spend time each story to add facilities (which I've called "capabilities" previously) where appropriate. Occasionally you get the, "OMG, we need to do a big refactor", but if you have been diligent in maintaining flexibility, then it should limit the amount of time that is necessary to move in the correct direction.
Ultimately, even code and toolchain flexibility comes at a cost - for example, should you go for a non-relational database because of potential future scalability needs, or cross that bridge when you come to it? Or should you make your code internationalizable, even though the first release is in English only? The right answer may depend on the probability of the business need occuring in some foreseeable time frame.
Unfortunately, the OPs point is too black and white, where the reality is much more grey. In my view decisions in a project need to be subject to a cost-benefit trade-off, and future-proofing is no exception.
Thats the crux of the problem. Different people have different definitions of flexible. How are you defining it?
You end up “making your codebase flexible” in all the wrong ways by introducing abstractions, interfaces, and extensibility designs that end up being the wrong tool for the job once the future requirements are known.
There may still be a simplistic, high level where you can have certainty about how to factor code. E.g. separating front-end and back-end, using tests as scaffolding for helping verify deployments and facilitate changing code, using a few simple tricks here and there to reduce boilerplate or make sections of code reusable.
It’s totally fine to pursue these optimizations, especially if they are part of a healthy backlog process to prioritize them.
But the problem creeps in when principal developers or “philosophizing architects” take it as a goal unto itself and begin trying to mandate it all the time, especially with the anti-pattern where huge architecture discussions become synonymous with routine “just get something done that works and refactor later” code reviews.
So “making it flexible” hinges entirely on whether or not you can correctly forecast exactly which types of flexibility you’ll need. But this is no easier, and often harder, than predicting how end user requirements might change in the first place!
Rather it is very modular design, that is very simple, bordering on dumb.
This is all very much easier said than done. "Simple ain't easy".
>We need to hire a team of developers and build in-house software, despite wordpress and shopify being a much easier alternative, because when our customer base grows to 100 times what it is now, it will make our lives easier.
I think this comes from "not invented here syndrome" rather than the assumption you'll need to support hundreds of orders a minute in the future. Also job security.
I also saw lots of "split it into microservices so we can scale out for load" on systems that years later still only saw minimal use.
As others have mentioned - it's orders of magnitude easier to add a layer of abstraction to a simple system as requirements evolve, than it is to remove an unneeded one down the line. The latter is often impossible.
I rarely, if ever see this done. Probably because it takes a big risk as an engineer to actually do it.
I couldn't find hard numbers from the Bureau of Labor Statistics, but this Stackoverflow survey [0] from 2016 points out that 50% of developers work at companies with less than 100 employees (that's employees, not developers). So many many software jobs are at smaller companies. Why don't we hear from them about their trials and tribulations?
Here are some ideas I've had (off the cuff), would love to hear other thoughts.
1. Big companies invent the future because they have the resources to do so. That wisdom then gets rolled out often in the form of software (Hadoop, k8s, React).
2. Small companies are boring. They are still struggling with product market fit or are stable and boring. They are not sexy, especially if they aren't growing quickly or are a services company.
3. Bigger companies have a bigger platform to announce work and/or support people to write and speak. Software they develop touches more folks.
4. Smaller companies are not doing anything interesting to the wider development community. They are focused on niche problems that aren't really interesting.
5. Folks at smaller companies don't write/speak as much.
6. We are hearing from smaller companies, just not as much/it's categorized as 'startups'.
7. Consultants, who tend to do a lot of proselytizing of new software trends, work for larger companies because "that's where the money is".
8. The scale of small business is such that software quality doesn't matter as much.
I don't really know what the reason is. I've primarily worked for small companies my entire career and I think the impact that software can have on a small business is transformative and well worth writing about (as I have for years and years [1]).
[0]: https://insights.stackoverflow.com/survey/2016#company-size
It's a half-truth that one doesn't have to do it from the very beginning.
You don't have to set up a whole cluster where a single host suffices for a long while. You do have to plan that your project may eventually become big enough to require a cluster - and consider doing yourself some design favors here and there to make that transition easier. Or knowing that you're going to spend a while redesigning stuff, while being pressed by the growth.
I mean, I've regretted not doing some design assumptions (e.g. about storage) from the start more than once or twice.
This surprisingly leads to “future proof” code. I didn’t anticipate the needs of the business. I just wrote what needed writing. Yet over time I build the abstractions in that the code is actually using.
I say surprising because code written this way looks obtuse and... messy. You get very senior and very confused developers asking you why you didn’t apply The Singleton Factory Pattern or use the XYZ architecture. So you say, because I didn’t need it and they just don’t get it.
But that’s part of the reason why I like Haskell so much these days. And why I liked Python for so many years. I like languages that bake in good ideas and abstractions for you. Python baked in the iterator pattern, the decorator pattern, etc so that you could use them in the language with little ceremony. Haskell bakes in the most powerful abstractions of all: type theory and algebra.
And still the idea of only writing the code you need to solve the problem; nothing more or less; stands up. People may look at you like you’ve lost your marbles. But if you can get them to stick around in the codebase for a year or two they’ll start to see it too.
The problem is that devs do not create enough leverage in their solutions, so they're never that useful in the future. And, devs tend to over estimate what is actually re-usable. The second point may be the most important. Too often, I see solutions that are half-way reusable, which means they're not re-usable at all.
In the end, building out a solid foundation is key. The problem is, this is pretty hard and takes a lot of experience to know how to do well.
I see too many startups burning time/money on scalability they dont need. If you have 1000 customers, and your rack can handle 20000, dont talk to me about future-proofing against the day you are building your own datacenter. Keep your code working for today's customers. Worry about scale when it happens ... IF it happens.
Imho, limit your perspective and planning to double your current customer base (good day senario) and half (bad day senario). Any day beyond those bounds will require a rethink. A rethink THEN, not today.
Of course this depends on whether your product CAN become viral in the first place, but I think many people are just hoping it will, even if it never does and it looks like just a waste of time.
> despite composition being a much easier alternative, because
> after 5 years of codebase growth, it will make our lives
> easier.
And a equally future-proofy statement:
"We need to contort our code to avoid our implementation language's inheritance features to instead use composition because after 5 years of codebase growth it will make our lives easier"
There's a reason why the GoF says "prefer composition over inheritance" and then spends half the book covering inheritance.
Does it really? The bulk of the patterns are about composition and delegation which is why wags using languages less restrictive than C++ have been able to do things like show 47819.73 patterns are variants of one or two things.
Here's another one though. We need to use Promises despite callbacks being trivially understandable and supported everywhere in arbitrary JavaScript versions because in 5 years everyone will probably always use Promises for everything.
Edit: to be clear, when I hear "code reuse", I think "using a component outside its original intent". Composition does impose some duplication sometimes, but it's braindead plumbing code, not buggy logic code, that gets duplicated. I generally think that cost is worth it to gain more statelessness. Hence my greenfield bigotry
I'd wager that it's rather good code then. There's also the inheritance-based code that isn't though, which slowly is driving me to the conclusion that "unsupervised" inheritance is some sort of petri dish for cthulhu-esque architectures.
That doesn't mean inheritance is bad, per se -- just that we should stop and think much more about using it. Composability, on the other hand, isn't nearly as "invasive" since the whole point of it is the interaction of uncoupled things.
This somehow reminds me of the promise of fusion vs. fission energy, where it's argued that in fusion you don't have to constantly prevent your reactor from exploding... :)
You don't need testability or 'greenfielding' for this. 'Design by contract', as you mention, is a useful way to think about this, without attaching value judgments to composition or inheritance. Inheritance is a much stronger and therefore burdensome contract. It's much less likely bits of your model can really adhere to it. The advice to lean towards composition is a function of that - few models can meet that high bar.
I talk of greenfielding because it's rare that the inheritance alone holds a codebase back so much that it needs to be rewritten. But code I originate is always composition based
There's a difference between future proofing to be able to deliver a roadmap or using your market research to get the next X customers and trying to guess at what some as yet unknown market might one day want.
It's really easy to follow these rules with Haskell. We develop our syntax to be easily extensible. Being strongly and statically typed allows us to extend our parser/compiler with relative ease, and not think too much about how the future will look.
Engineer: I can build it in different ways. This simple way will accommodate N users, this more complicated way will accommodate M users but will take three times as long. I recommend we do the first option, because it's unlikely we'll see more than N users.
Management: no, we want to get to M users, build it the second way
there is no future proofing as there is no way to predict or time to put effort in predicting how business needs are going to change, how apis you talk are going to change, and how technologies you use are going to change.
it's easier to just write simple functions, that you can come to later and see what they do quickly and change them.
Only on going problem I do have is dependency, that I've to check every thing that used the said function, does it continue to work