Over-engineering considered useful | Hacker News Reader

Over-engineering considered useful

isbullsh.it

21 pointsmartius19 comments

Failure is the fastest way to success. I wrote over 500,000 lines of game code before I ever graduated high school and only 1 project ever saw the light of day. Learning from all of my mistakes back then made getting the degree in college much easier.

Also, completely rewriting things is a major skill for a professional. Churning out great code, and doing it fast and with few bugs is worth a lot in the industry

Completely rewriting things is one of the worst decisions an intermediate engineer makes. That's what killed Netscape.

Applying learning to a new project, now that is something different.

Being closed to the possibility of rewriting a project is one of the worst decisions advanced-intermediate engineers make. How much technical debt has accumulated? How many future features/changes need to be made? How long does it take just to maintain the current feature-set?

If a re-write saves more time than it takes, then it should be considered. If it takes you a week to implement something that should be done in a day, and you realize that you'll be doing lots of those things over the course of the next few years, it's probably worth considering a re-write. It's not a solution to every problem, and it's probably done more often than it's not, but there are certainly times when it pays to do it. It's easy to have simple rules that say don't do things. It's much harder to make the best decision for the business. Just because some businesses in the past have made the wrong decision with regards to re-writes doesn't mean it's always wrong.

there is a big difference between rewriting netscape and rewriting the time tracking software for a 100 person company. Netscape was a very large, complicated product written by a lot of developers. Most software isn't like that. I'm not saying that learning from netscape's mistake isn't valuable but there is an important distinction to be made. Rewriting software can often get rid of a lot of cruft and edge cases caused by developers not following YAGNI.

> Completely rewriting things is one of the worst decisions an intermediate engineer makes. That's what killed Netscape.

This is commonly trotted out to discourage re-writes, but how does that square with another piece of common advice: "build one to throw away" ?

I'm genuinely curious, and I'm not trying to pick on you specifically. There are just many different situations where one or the other makes sense, so I don't think it's so cut and dried.

It's not cut and dried, but a good guideline is this:

If you've written a program, but it doesn't work yet, and you've realized you made major architectural mistakes or whatever, feel free to rewrite from scratch.

If you've written a program and it's been in real use for a while, solving real problems for a lot of happy users, that means it embodies a lot of knowledge that will be difficult to re-create from scratch; in that case, think very carefully before embarking on a rewrite, and look hard for a way forward that will preserve that knowledge.

I'm not saying you should always rewrite your code in a professional setting. In the scope of my previous comment, I was definitely considering personal projects.

That said, when systems need to be re-engineered, completely thrown out and re-written, or a new feature needs to be cranked out to meet a business need, the developers who can design and create large coherent systems quickly is irreplaceable or very expensive. Constantly throwing your code out gives you this valuable experience.

When you're talking about physical devices and structures, I like "over"-engineering: e.g. those extra cables keep my elevator in the shaft when one fails. In the case of the physical world, I think we can define over-engineering as engineering to much more than the minimum required. Another example is two-lane bridges: the sign says "16 T" but obviously, two vehicles could be on the bridge at once, headed in opposite directions, and it has to hold up for decades. Exactly what specifications are used to build the bridge, I have no idea. But it's obvious to me that it's "over-engineered."

I don't think we know enough about solving problems with software to detect over- and under-engineering before ... I don't know. Before we need to? Before users complain that it doesn't work? Before MegaSoftCorp spends a few million too much?

In 'real' engineering, every engineering calculation has a safety factor attached to it. You figure out how much you need, and then you multiply it by the chosen safety factor to get what you need. The choice of safety factor varies. In some cases, it may be regulated/required. In other cases it is based on the engineer's judgement. And of course, it will vary from field to field. In aerospace, safety factors are typically much lower than in civil engineering, since weight is a much greater concern.

Which brings back to why we have a safety factor at all. The only reason why we have it is because of uncertainty. The real why we have safety factor at all is because we don't know enough.

The terms "over-engineering" is so ill defined as to be mostly useless.

I have a strong personal opinion of when things become over-engineered, but that means little.

I've worked for smaller companies, which tend to under-engineer due to time restraints.

And I've worked for large corporations with such overkill process that it is amazing anything gets shipped ever. And yet it does, it takes almost forever, and costs far more than it should, but it does eventually ship.

Discussing "over-engineering" may be fine for general interest communities. But I think on HN we should aim higher. I'd love to have a rigorous discussion on how we can identify designs as either over or under engineered.

Some parts are not really that ill-defined. e.g. an interface with a single implementation is code-smell. That's a pretty good start.

"smell" is ill-defined!

Sure! This post is like an invitation to discuss the subject, right?

"over-engineering" was not the point anyway. The title is a lie. The point was: dont be afraid of making mistakes (Over-engineering and the likes) when you can afford to, so you can learn from them and aim at high quality discussions on HN.

well said

When I discovered the phrase "Speculative Generality" in Fowler's Refactoring book I suddenly realized there was a name for a problem I already knew I had. This is one specific kind of over-engineering which we can identify -- when creating an abstraction, are you doing so in anticipation of an actual need, or merely of an imagined need?

Another kind of over-engineering is speculative optimization (often called "Premature Optimization", but I like to emphasize the speculative nature of it, because that's the heart of the issue). Am I working to make the system faster in response to actual evidence of a performance problem, or am I just guessing about performance?

Over-engineering isn't always bad. It's all about predicting the future and weighing risks, with everything that entails. It's hard, but it's a fundamental part of design, and of life.

Whenever I'm unsure about whether a particular speculative generality or optimization is justified, I try to step back and find implicit assumptions. I try to turn "This might help some day" into "I'll need this if XYZ happens". If I can't do that, it's a bad sign.

And I agree with the original author's point: making mistakes in places where you can afford to fail, or where you can afford to recover, is a great way to learn.

To me programming is both engineering and something akin to literary composition. You want your design to be engineered well - perhaps "over" engineered for redundancy and reliability - but you end up actually building it with a formal language. This brings me to a wonderful quote:

"A scrupulous writer, in every sentence that he writes, will ask himself at least four questions, thus: 1. What am I trying to say? 2. What words will express it? 3. What image or idiom will make it clearer? 4. Is this image fresh enough to have an effect?"

- George Orwell

The first two apply to composition in a formal language. Over-engineer the structure of the thing but be as merciless as Hemingway when actually composing your thoughts and sentences. Eschew excessive lines and actual tokens. Those are the byproducts of the rushed and/or cluttered mind (which commercial projects may necessitate ;)

I agree with the premise, that over-engineering is bad.. but the only way to know it when you see it is to have done it before in the past.

i know i wouldnt have been as completely allergic to complexity as I am, had it not been for some very important life lessons on the subject.

Cant agree more. Especially startups should be very careful not to get trapped in an infinite loop of engineering. In my previous failed attempt we did just that. Built feature after feature but nothing could ever see the light of the day until it became an overkill.

Crazy exploratory programming is the best way to learn and grow your skill set ... but please don't do this in a team project where other developers have to grapple with your insane abstractions that lead nowhere and mental meanderings ... keep it to yourself.

Reply on news.ycombinator.com