In my experience, I have seen much more problems because of over engineering by well meaning people than because of technical debt.
Simple but under engineered systems are much easier to rewrite than to simplify over engineered ones.
Simple but under engineered systems are much easier to rewrite than to simplify over engineered ones.
for instance one smart and experienced guy I knew said if there was an error, your program should just fail instead of giving a ton of messages and recovering.
and if you're just throwing code together, that might be what you do.
on the other end of the scale, a well written "second pass through everything with cleanup" might have tastefully written code, a few relevant comments and consistency.
and then I've seen a lot of code that appears to look like that, but is copy/paste garbage. (sort of the coding equivalent of a wordpress theme)
I would bet that he was right in that situation, but that he was also making a nuanced statement. Not all code should fail loudly, but when it should, "recovering" can sometimes be a distraction from a very real problem that needs to be addressed by the right person.
So fail early, but not too hard.
Logging is my go-to example for this sort of behaviour. People write stupid amounts of log statements in their code, so much that it’s hard to even read the code to understand what it does, in the hopes that it’ll make debugging easier. Use a damned debugger! Take a traffic dump! Use strace!
What’s more, it’s extremely rare that libraries provide logs themselves. So the actual complex parts of your application, like say the HTTP library or (God forbid) the TCP stack, can’t be debugged this way.
If you find yourself writing a bunch of statements like “DEBUG: updating balance from 1 to 2”, stop and write some tests instead.
I have encountered a fair amount of cases where that does not work.
- debugging a super large program ? loading gdb may take two minutes while recompiling to add a printf only 3-4 seconds.
- not an admin on the machine you are and the person with the admin account is not around ? sorry, you can't debug on macos (and likely in some linux distros)
- likewise, no traffic dump (and I'd assume no strace) if you don't have root access
I've got a program with log statements like that all over the place. Since stepping through it with a debugger would not even be possible. My IDE takes care of hiding the debug statements, since they're all encapsulated in different regions and if-def statements to log different types of things.
Flipping your argument, you’re basically saying that you can’t debug multithreaded code if the program doesn’t log.
But, I think we're depending on different programs in our workflow. I much prefer a logfile with statements that focus on what I want to inspect at the time.
I'm quite proficient with debuggers on both Windows and Linux, but I tend to use them less when dealing with my own multi-threaded code.
To be clear I’m not talking about adding some specific print statements to the code while debugging, I’m talking about keeping those statements in production code.
Of course, there’s cases where logging might be your only option. In that case: Go for it!
Relinking a super large program may take minutes, whereas loading gdb can get me a backtrace in 3-4 seconds! VS natvis files and other visualizers can hot-reload while I'm looking at a crash dump from production that took hours to trigger/gather/repro!
There are cases for logs - where sufficiently efficient conditional breakpoints are particularly painful to setup, for example - but the only time I've waited minutes for a debugger to respond has been with VS after major system updates as it refreshes a universe of symbols from the symbol server.
Do you have some really slow python scripts auto-loading in GDB or something?
well, we have definitely opposite experiences :) the main software I work on creates a ~1gb binary in debug mode and lld links that in a few seconds. Bud gdb and lldb, even with gdb-index, and all the optimizations I could find, both make startup slow enough that I can go for a coffee and it's not always finished loading when I'm back
It was only when I added proper logging to the critical junctures in the code that I could finally see very clearly how it behaves and see what input results in what code paths and output. I learned more in an afternoon than I did in a month.
“I apologize for such a long letter - I didn't have time to write a short one.”
https://quoteinvestigator.com/2012/04/28/shorter-letter/
There may be better information.
People are so afraid of making a bad decision that they refuse to make any decision at all, and then make a giant mess in the process. What you should do is spend your energy on finding reversible decisions, and then not spend a lot of effort on actually making them. We know where the paint store is, we know they can make up paint in 15 minutes, fuck it, paint it blue, we can always paint over it later. Hard no to black, though, since you can't paint over that shit.
People are so used to avoiding decisions that on a few occasions I've entirely flustered someone who wanted to tear into me (sometimes with an audience) but cutting them off and saying, "Yeah that was a mistake, and here's how we're going to fix it." None of them had any idea how to recover from someone saying "I was wrong" and going on to try to fix the problem. I still have a little video in my head of one guy's eyes bugging out when he realized what I just said.
When you start out, you just write code and don't think about how things interact with each other too much.
Then you learn that you're rewriting a lot of code, you overcompensate and start to over-engineer to avoid duplication and often to handle more scenarios than you need. You abstract out possibly too much.
Finally, you get to a phase where you realize a lot of abstractions that you've created aren't actually used and you're writing more "meta code" than you need. In this phase you learn to engineer just enough for what you need at the time and design things to be easily changed in the future. You trust that even if you're system design can't handle everything right now, you've designed options for yourself to expand as needed.
The dept is that you take time and effort from your future to save time and effort in your present.
There was a separate testing client that was used for a lot of the demos. The demos on the 'real system' would fail so often that somebody came up with the idea to just take a lot of the code from the microservices and drop it straight alongside the testing client, as a monolith.
It was so much more stable that after becoming the default way to demo, it became the default way to use the tool.
5 years on and I believe it's still running that way. In this case microservices were massive over engineering.
They often are.
Well-meaning engineers and architects seem to reach for them while forgetting that microservices are about scaling your organization, not your software.
There are very few technical deficiencies in a monolith that are solved by microservices. Indeed, they typically bring technical hurdles: distributed systems are hard to reason through and IPC over the network introduces latency. And you need a very strong ops team to implement them without major headaches.
But where they really start to shine is when regression testing takes ~days and your development cycles are screeching to a halt because you've got too many in flight features and not enough runway to land them.
What takes real skill is recognizing which of your monoliths should be grown and which ones need to be sliced down the middle.
A lot of over engineering is due to getting this completely wrong: you prepare for something that never happens, at great cost and while underestimating the difficulty level. So you end up wasting endless amounts of time on stuff that has no business value.
On the other hand, a lot of under-engineering is due to not having a clue about what is obviously coming next and getting caught by surprise by completely obvious requirements.
I would rephrase this as make guesses based on probabilities and urgency estimates from past experience. You can't know the future but you have stats of the past. So a good engineer is one that has relevant experience and applies it appropriately.
Seems to me the default is obviously getting things wrong whether that be over or under engineering.
How would one go about reliably making choices that strike exactly the right balance consistently?
One thing I'm starting to really internalize is that to do so requires a deep understanding of software engineering as a domain, the domain and existing system, and the technical vectors in the business.
To me, engineering is taking a problem and the corresponding constraints and building the solution that satisfies it with the least amount of resources.
"over engineering" in software is about trying to make something adaptable? Or maybe just satisfying someones sense of beauty.. Or maybe it's just about programmer convenience at the expense of customers/user experience and hardware resources..
It could also refer to building resilience to load before it is necessary. Example, distributed databases with failover and recovery capabilities for a side project.
Poke holes all you want, those are bad examples, but they should illustrate the point. Over engineering is a real phenomenon.
It's even more frustrating when you work with engineers who refuse to believe that any code they write could become technical debt in the future. These tend to be people who overcomplicate systems to anticipate future requirements
I don’t think I’ve ever anticipated a future requirement that did not ultimately turn out to be necessary.
Conversely, I’ve had a lot of people tell me something was not necessary only to find that, surprise, surprise, it was necessary after all.
This is how I approach software, but there are so many people out there who won't approve simple code, because it doesn't have enough configs, or classes, or whatever. It seems that the person who wants the most over-engineered code tends to get their way. Psychologically the absence of 'things' is always inferior and harder to argue for than having more 'things'.
Reactive frameworks indeed hide a lot of complexity but that does have positive impact on the user code. (IMO)
I leave breadcrumbs and openings in my code all the time so that the next time I'm in there, I'm set up, invited even, to ask the question I left unanswered before, and maybe do something about it. I leave the option of adding a feature, instead of building a config framework to support it and then defining one implementation. Which will end up not being the correct API when (if) I write the third one.
Sometimes people beat me to it, and get excited because they had an idea they're now invested in. I usually just let them have it, don't point out that I put the idea in their head. It's so infrequent people get that invested in functionality, you have to encourage it. There's too many things I'd like to do and never enough time anyway. They've just nominated themselves as a candidate for maintaining that module when I get tired of looking at it, freeing up my time for things nobody else cares about until it's done.
It's like if you were designing a house with plans to expand it later. You'd be very careful about certain decisions, like whether you should put a bedroom in the logical spot for an addition, because now that bedroom also has to be a thoroughfare. For 20% extra effort, you've extended the calcification threshold for that house/code by 80%.