Hacker News Reader

Comment by awkward | Hacker News Reader

This is the opposite of my experience.

Under engineered code tends to be simple, straightforward work with a low blast radius, such that "make one change and test" covers most cases.

Over engineered code tends to be more convoluted, with more fan in, more fan out and a large dependency graph. Changes become more like high pressure bomb squad work, where cutting the wrong wire blows up the whole project.

If I get a task to "make the button blue" I'd rather do it in a repo where I need to grep around a little than in one where I need to debug which button factory factory library is being pulled in and applied to the parent docker image of the one running.

View on HN

If we're sharing experience with underengineered code, let me have a go.

At $BUSINESS we have a very successful marketplace that brings together buyers and sellers! We've recently IPOd. We get a lot of new items in and we have an internal page which is used by multiple full-time employees to approve new items, maintaining quality and defeating spammers. The code is written in HTML::Mason templates in Perl (which is basically pretending to be PHP, but for Perl).

The code makes a query to the database that joins the main `items` table with millions of rows with about 5 other tables. It does complicated locking logic in this query, and if this logic fails, the multiple full-time approvers cannot effectively coordinate their work, and the site cannot make money off new items. The HTML and the code of the loop are interspersed, and there are additional queries issued as you go through the loop. The code outputs JavaScript snippets to the page inside the loop which manipulate data structures incrementally.

> Under engineered code tends to be simple, straightforward work with a low blast radius

Hahahahaha this was a delicate multi-month project to split code and presentation within the existing codebase (outputting a single JSON blob instead of writing out incremental append operations to the page source), installing a separate locking system, performing a zero-downtime cutover to this new locking system, following up with a zero-downtime cutover to a backwards-compatible new subsystem, switching that system over to more-scalable queries — then finally developing a much more ergonomic but less acutely critical new frontend to improve productivity.

That feels like a really high class problem to have. You have a product that got you to IPO, and it hit the end of its lifespan and needed to be upgraded/rebuilt.

You did all that in a few months? That actually sounds like something that went quite smoothly. The time needed to engineer that kind of high-scale system at the outset is also going to enter the multi-month range, and with less background info to inform it. The phase-shifts of scaling are by no means easy, but they respond well to effortful, issue-by-issue grinding away at the problem.

The worst-case overengineered projects generally have organizational issues that preclude ever making concrete progress or result in a Juicero-style "why did you even make this" product. Those are the projects that burn people out.

LOL I'm very sorry that happened. None of this is scientific - over and under engineering in this thread are abstracted from actual results. I'm more than willing to admit that convoluting concerns between presentation and billing is nuclear waste grade underengineering.

For my example of overengineering, I was specifically working with a project where every web page on a public site was taking a couple seconds to load. A team had reimplemented npm using inheritance in docker. Because they had one library that just imported stuff (to populate the parent docker image), their webpack build was unable to distinguish between imported and unimported code, and was just packaging everything. 20 mb websites.

That sounds over-engineered to me. But just poorly over-engineered, the worst of both dimensions.

Your experience largely mirrors my own. At a previous employer, I had to make some changes to a process that was importing data from a vendor. Typical straight-forward ETL, right? Not even a lot of data, like 20-30 records daily.

The process? Load the data from CSV to JSON, ship the JSON off to Azure. Pull the data back from Azure, check to see if it's been processed, apply the change on premise. If failed, reschedule for later. Multiple processes, multiple scheduled jobs, on-call alerts, etc, etc. The whole thing could have been replaced with maybe 50 lines of Python. Instead, it was probably around 10k lines of C# and dependency on a 3rd party ETL tool. It was a fucking mess. Worse, yet, I wasn't allowed to fix it.

To be slightly pedantic, one might call this situation over-architected rather than being purely over-engineered. I would agree that over-architected solutions (regardless of the internal engineering quality) are definitely hard to alter due to the many disconnected disparate parts, not to mention any human/political boundaries that have grown up around the implementation.

Following along that train of thought, under-architected solutions are often great to update because you get to make logical cleavages that are informed by time spent in actual production use, giving you a much better basis for any decisions.

This reminds me a colleague creating a Python class like this

import os

class Bla(): def __init__(self):

        self.var1 = os.env["VARIABLE1"]

        self.var2 = os.env["VARIABLE2"]

        # ...

        self.varN = os.env["VARIABLEN"]

    def get_variables(self):
        return self.var1, self.var2, ..., self.varN

They took 2 months to write an insanely complcated code to just copy few objects from an S3 bucket to another... And because the Lambda was timing out, they set the timeout to 15 minutes, leading to our Lambda costs skyrocketing because the function was failing/retrying all the time for some objects.

I was allowed to fix the timeout issue to save cost but I was forbidden to fix the code itself because our manager said "Well, it works so let's move on".

6 months later, on a Tuesday morning, bored, I decided it was enough so I rewrote this bloody Lambda in ~90 lines with a proper handler, retries, logging statements, etc ... in a couple of hours.

What did my manager say? "Good work but you should have taught them rather than doing it yourself".

He is right though, gotta stop the problem at it's source.

Often however that is a far larger/difficult task and it is easier and better for ones sanity to just fix the things that affect you directly.

Amen. And the worst part? Can't even fix it easily, because they all depend on each other. Someone is getting the data in json format and depends on the filename being exactly YYYYMMDD in a specific directory of a specific server. Some horrible open source framework can only handle data at 200 requests/sec (and of course every row is a request) and you have a distributed asynchronous queue plus rate-limiter to make it work. Yada yada.

Your experience agrees with mine. It may be TEDIOUS, but it's generally easier to fix underengineering which (IMO) tends to be wide but not deep issues.

Overengineering has its hooks everywhere; even WITH tests, changing something changes everything.

I've come to realize that at least for the level of engineering I'm exposed to that boilerplate is not always bad, copy/paste is perfectly valid to a point (which for me is usually "2-3"), and DRY is a tool, not a design goal.

Over-engineered code has a lot of coupling where incidental equivalence may have been misapplied as fundamental sameness. This was discussed here on one of my favorite PLC programming blogs:

https://www.contactandcoil.com/automation/industrial-automat...

Copy and paste are great tools. Making all 6 buttons in a particular grid with the code:

    grid.AddNewButton(1, 1, "Thing 1", Color.White, Color.Black, onClick1());
    grid.AddNewButton(1, 2, "Thing 2", Color.White, Color.Black, onClick2());
    grid.AddNewButton(1, 3, "Thing 3", Color.White, Color.Black, onClick3());
    grid.AddNewButton(2, 1, "Thing 4", Color.White, Color.Black, onClick4());
    grid.AddNewButton(2, 2, "Thing 5", Color.White, Color.Black, onClick5());
    grid.AddNewButton(2, 3, "Thing 6", Color.White, Color.Black, onClick6());

keeps this incidental sameness in mind. Just looking at the above code causes programmers everywhere (myself included) to imagine ways in which the above could be done with a `for` loop, computing the row and column numbers from the index, using string concatenation for the labels, creating an array of onClick handlers and indexing into it... But that forces fundamental sameness where there may not be any; a little repetition doesn't hurt.

Foreground and background color should be moved out to avoid repetition, e.g. BUTTONG_FG, BUTTON_BG. All other parameters are different, so no repetition.

> If I get a task to "make the button blue" I'd rather do it in a repo where I need to grep around a little

Except it's not one button, it's 12 of them and they all use different UI elements with different names and syntax, and you can't change one of them because doing so would break some unrelated business critical class.

> Under engineered code tends to be simple, straightforward work with a low blast radius

Under-engineered code can easily become over-engineered code, while retaining the appearance of being the former, as engineers keep working on it over time. It only becomes easier over time, as bugs are uncovered by users and then "fixed", as features are requested by users and then bolted on, and so on.

> If I get a task to "make the button blue" I'd rather do it in a repo where I need to grep around a little than in ...

In the kinds of codebases I'm talking about, you will successfully end up making the button blue, but then, either:

1. sth else breaks and you wouldn't know about it until later; and/or

2. a bunch of seemingly unrelated tests break and you will need to debug which button provider is auto-injected into which tests' setup routines.

Bad code is not related to over engineering though.

Over engineered means that is better than the specifications, however such things being over specifications happen to be useless or unused. Then time and money was wasted, but the product is not worse by any mean.

Reply on news.ycombinator.com