Imaginary problems are the root of bad software

We also get slower with so many people. The coordination overhead is killer and losing context as the product is sliced up into small parts that move on without you

Until we figure out a nice metric for "removing complexity" and then rewarding for it, it's not likely to change, IMO.

This is a massive change from my first software industry job in 1997.

I was essentially a “design intern who knows HTML” on a team that built a shrinkwrap Windows application for enterprises. The core of the design team was a graphic designer, a cognitive scientist, an industrial designer, and a product manager with industry experience in the customer domain.

The application was Windows native. User research was conducted on site with scientific rigor. Adhering to platform guidelines and conventions was a priority. I think I spent a few weeks redrawing hundreds of toolbar icons so they’d be in line with the Office 97 look (the kind of boring job you give to the junior). If the app stood out on the Windows desktop, that would have been considered problematic.

Today a similar design team would only have a graphic designer and a PM, and neither of them would care the slightest about platform guidelines or customer domain. The UI is primarily an extension of the corporate brand. Hiring a cognitive scientist? Forget about it…

Everything certainly wasn’t perfect in the Windows/Mac desktop golden era. But the rise of the web wiped out a lot of good industry practices too.

However, a large and insidious cadre of B tier engineers was constantly writing docs proposing meaningless arbitrary system changes and new designs for the sake the changes themselves. These new projects were the only way to get promoted. The entire e4->5->6->7 track was written in such a way that it only ever encouraged “TL”/“architect”types to grow.

This led to constant churn, horrible codebases, and utter bullshit self made problems which could only be solved by digging a deeper hole.

There are companies who handle this well. Ultimately it comes down to engineering culture.

I am. I'm paid more than most developers to run a team doing just this. We make minimal change, have an absolutely non-negotiable focus on stability and minimalism and reject any change that isn't absolutely driven by validated majority user need. Even then, the bar is high.

I'm not saying this is a common situation, but it certainly isn't rare in my experience. Software is a vastly wide scope of software types and requirements. I'm paid to be ruthless, and to know what ruthless looks like in terms of delivering consistently without downtime/data loss/critical issues.

From my view, that and pressure from the analyst racket are the main drivers behind feature bloat with self promotion a distant third.

everything about engineering group decisions are about creating a reason to use a newer framework in a way other people can vouch for.

I hope that I live to see a time when software applies modern algorithms to classic designs and produce “hand tools” in software.

And also why the most frequent request from the users is: Please don't change anything.

The principal-agent problem looms large in software.

FOSS software has vastly different incentives than commercial software, yet suffers from many of the same problems: bugginess, poor performance, lack of documentation, feature misprioritization, bad UI.

That alone indicates that the problem is not merely "misaligned incentives".

Actually, you can reduce most problems down to "misaligned incentives" if you're overly reductive enough. That doesn't mean that it's a useful way to think about the world.

In retrospect - it was recognized that they built or took good care of money making systems with little drama and that was well appreciated by the companies.

In FAANGs I see now more of "what will get me promoted" versus "what is gonna make the company money" ethos.

I’ll say one thing though: all of those skills you mentioned as not being valued are extremely useful for indie dev/taking ownership. Shipping simple + correct code frequently is extremely possible with sufficient practice and discipline.

More to your point, this is why I switched toward being a research engineer. There is a higher barrier of entry, projects are quite technically challenging, and constraints often force thinking of a world of computing beyond the tiny sphere of the web browser.

It’s hard work, but I love it.

If this resonates, you are in the US, and looking for a change, drop me a note (see profile).

Not gonna convince anyone but hopefully someone reads this and starts thinking about such options. They are not as impossible as those in power would have us believe.

- There is bias towards rewarding more lines of code or more code commits (which is often the exact opposite of what characterizes good software design).

- There is bias towards rewarding speed of initial implementation (which often goes against medium-term and long term maintainability and ability to handle requirement changes which is usually far more important). The costs of this tends to fall on other developers who come in later and act as scapegoats for the 'apparent 10x developer'.

- The industry keeps dismantling narratives which would otherwise help to reward talent. For example, many companies dismiss the idea of a '10x developer' - Probably because they are aware of the previous point about the fast developer who cuts corners and creates technical debt for others. 10x developers do exist, but they're not what most people expect because paradoxically, they may be slow coders when it comes to LOC metrics and speed of implementation for new features; their code really only shines in the medium and long run and it improves the productivity of their colleagues too so it's hard to properly allocate credit to 10x devs and they only really matter on greenfield projects.

Unfortunately, for my money, I think the only real way you can create an incentive structure which emphasizes stability and change is by offering some kind of form of insurance.

My father was an electrician who often complained about how he never got paid adequately for the stellar, stable work he did, and one day I asked him whether he ever thought of raising his rates but providing a kind of service guarantee, where if a problem occurred that could be traced back to his own work, he would step in and perform the additional work at a reduced fee. Naturally he laughed out loud, because that's not how business works.

Ownership of an already-mature product is sort of like providing an insurance policy by default, of course. And sticking with conventional designs can be a solid business strategy if you use their slow-changing nature to e.g. build the thing faster than you could otherwise. That's the strategy I'm using for my consulting: Stick with what we know best (Hugo+Bootstrap for a napkin sketch UI demo as fast as possible, then SQLite+Django+React to build out the main functionality ASAP too). Emphasize solving the _business_ problem over the shiny tech.

All of what you said is true, but there are still people who think a minimalist, rugged and reliable solution is superiour. That maintainability is a value in itself (and thus, one should not choose the wildest, most experimental dependencies).

It was quite telling that after 1.5 years of that project, where all staff had already been replaced once, all they had to show for was a fancy product name and a presentation. And that the manager that led the project for ~2 years left right before or right when it went live - and he did that in a previous project too, where a working .NET backend for ecommerce was replaced with a Scala microservices on AWS system.

I did hear about the latter; I heard they went back to .NET, but the Scala services are still up and running and a maintenance nightmare.

But the lead developer got to play with his favorite tool and landed a job at lightbend because of it. Career-driven development, and I don't even believe he did it for his own career, but for self-gratification. Ecommerce is boring; REST APIs and a front-end are boring. But Scala, distributed systems, AWS, that's all cool and new.

I'm so tired.

This. I recall the case of a couple of FANGs who on one hand expect their engineers to deliver simple, maintainable and robust systems to minimize operational costs, but on the other hand they demand engineers operate at the next level to be considered for a promotion, which means they are expected to design non-trivial systems which have a certain degree of complexity and require significant amounts of work to pull off. Therefore, as an unintended consequence, the pressure inexperienced engineers to push needlessly complex projects where they are required to design solutions well above their level of expertise, and put them in a position where their career is personally threatened if anything gets between them and their promotion-driven project.

I blame Career-Driven Development for a lot of "shiny toy" (i.e. failed and complicated) projects.

Whether that's wrong is a deeper question - if it leads to better salary through job-hopping and promotions, despite a catalog of failures, is it actually the wrong approach from an individual engineer perspective?

I still say yes, FWIW; I enjoy seeing my projects succeed and thrive. But I know others may disagree.

There is no technical problem. The problem is nobody worked on actually marketing the product. Build it and nobody shows up is the norm.

Those constants represent the markets supported by that app. By the time the app supports even a few dozen markets, every engineer involved will have exercised their stock options and left.

Not necessarily. It's just that most developers have never worked in a setting where they got to work on problems properly.

Solving real problems for real people is very addictive. There is a reason some people like working for startups. It's because you live very close to your users and when you make them happy you know.

The second interesting fact is that if you just plow ahead and solve many real problems fast you will eventually run into problems that are both real and interesting.

After having tried that there has been no going back for me. I am allergic to imaginary problems. It feels as pointless as watching people who are famous for being famous argue on TV.

I think we are all victims of our feedback loops. (University) Education sublely teaches us that the only important problems are those that are very difficult and preferably have never been solved before. Those same problems also make for better blogposts. In the real world the incentives are mostly opposite. Problems with no known solutions (or only really difficult solutions) are generally bad. They can be worth it, but you should stay away from them until you know that are worth it. Software engineers seem to almost pride themselves on not knowing what their users want.

It takes a while to scrub all that bad learning out and replace it with something better. Unfortunately some people are stuck.

Unless you're working on something like the Linux kernel, that's an imaginary problem.

At the beginning of a project, before I write a line of code, I try to: a) if it's user-facing software, get some UI designer input to shape functionality/interactions, not styling. To end users, the UI is the software, not just the shell, so mediocre UI = mediocre software. It can also illuminate needs you didn't consider that affect architecture, etc.; then b) block out the broad stroke architecture, usually on paper, c) intentionally choose languages/environments/tooling/etc rather than reflexively going with whatever we've been using recently, and d) spend some design time on a reasonably sane and extensible, but not overly detailed data model.

There's no perfect solution, but at least in my cases, it seems like a good middle ground.

> Simplify the problem you've got or rather don't complexify it. I've done it myself, it's fun to do. You have a boring problem and hiding behind it is a much more interesting problem. So you code the more interesting problem and the one you've got is a subset of it and it falls out trivial. But of course you wrote ten times as much code as you needed to solve the problem that you actually had.

[1] http://www.ultratechnology.com/1xforth.htm

But their day job actually isn't writing code, it's solving a problem for an end-user; the programming language is just a tool.

Rethink your job from a coder to a problem solver and you should be able to get over the compulsion to overcomplicate things for your own gratification.

Every feature is done quick’n’dirty and eventually you have people whose full time job is to respond to customer complaints and fix data straight in the production database.

If it's something more fundamental like language choice or system architecture... Well fine YAGNI now but if you ever do need it you're screwed.

I'm sure Google as an org has accrued plenty of staff that are working on mild to non important tasks over the years, and I get where he's coming from, but reality is far more nuanced.

> Have you ever heard about those three web engineers who figured out that secure online banking is actually quite an easy problem to solve?

> The storage and transfer of numbers is not a particularly hard problem.

These quotes are so incredibly disingenuous that make me question any advice OP has to offer.

First, banking is quite a complex domain, and it's complexity increases exponentially with the kind of services that you offer.

Second, banking is a highly regulated industry, which makes everything why harder than "it should". In fact many "neobanks" have appeared in the last decade, and this is usually their biggest hurdle.

Third, online banking needs to deal with quite a few hard technical challenges. That's why the likes of Monzo, Starling or Revolut often given tech talks.

So no, imaginary requirements weren't the root cause of bad software when I worked in banking. A 20+ years old big ball of mud, inability to pay off any tech debt (unless you wanted to get literally yelled at in front of the entire team), flaky and severely insufficient tests, and a very toxic working environment were all causes of bad software.

Equity is the entirely the answer for cutting through all the bullshit. At least in my head. I don't know how it plays in other people's minds but mine sounds like: "If we ship and go live, I get x% of all profit moving forward in my personal scrooge mcduck money bin". Pretty big carrot. It's kind of like time share in my own personal business, but I don't have much of the headache that goes along with running my own 100%.

This has some caveats, namely that equity in a 10k person org is often times not nearly as meaningful as equity in a 10 person org. Shipping your code 2 weeks early at Ford or Dell means what, exactly? If the code you are shipping is the business, then things are different. It also really helps if you care about the problem you are solving.

I'd say this - if the idea of direct equity/ownership doesn't get you excited about pushing simple & robust techniques, then you are completely in the wrong place. You should probably find a different problem space or industry to work in. Hollywood might be a better option if the notion of equity or straight ownership in the business still isn't enough to put your tech ego into a box.

I've often found myself now having issues even updating my resume, because what I did for the last year at work barely is explainable to other people on my team, let alone to someone in HR at another company. Or the more simple explanation is something that sounds like I'm doing work barely more complex than an intern could have done. Which often isn't wrong, but the intern wouldn't know which simple work to do.

My years of experience in the company's stack and org is valuable to the company, and nontransferable elsewhere.

On average, and depending on skill. Incentives are hugely important (probably the most important metric any manager could work on), but even they do not guarantee results. If you hire so many juniors that nobody is there to upskill them fast, you only get one lottery ticket per employee. Conversely, if you hire a bunch of geniuses and fail to give them incentives to work on realisable, useful problems together, you get two lottery tickets per employee at twice the price.

(This comment feels woefully incomplete. Does anyone know of good resources to learn more about incentive structures and how they relate to individual and company success? I feel like the problem is that incentive structures change massively when companies grow, so even for unicorns there's just a short sweet spot where we can actually learn how they are supposed to look.)

It's hard to admit that your baby is ugly.

This person might just have been terrified of admitting that all the (very tangible) time they spent on payroll building this thing was for naught. It’s unclear how their manager might have responded to that. And they wouldn’t have been able to put „built system doing X used by Y developers and deployed to Z customers“ in their resume.

The reasonable choice often doesn’t have much skin in the game.

but if you start convincing yourself that the fun distraction is the actual work you need to be getting done, then you might have a problem. (not to say that actual work can't be fun too. jut saying make sure you know which is which)

"If I get 5-10 of these done today, I'll reward myself with..."

Or use your imagination in someway like kids do with action figures. It can seem strange but there's ways it less mundane indirectly.

And the gap is widening...

this runs into another common mistake businesses make: putting people onto a task because they're available, not because they're suited to it (in any sense).

the junior person (+ a little mentorship) would be great for the job, but they're mired in some big project. but hey, you've got this super senior fellow sitting around waiting for work, and we can bill the customer more for them, anyways.

Maybe optimal performance is a feature? Feature worth working on from start?

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."

Sometimes boring is just pure torture.

YNGNA is generally true, but Product's should have a very clear vision of what kinds of entities the system is going to handle over then next 36 months before they start asking for specific functionality. I've seen extra shit get built, but I've also seen e.g. a travel booking system that was built without a "flight" being a first class entity. Flights were deduced on the front end from attributes attached to seats... which worked well until a PM asked for the UI to show fully-booked flights, which HAVE no available seats that make it to the front end. Same product couldn't handle the booker and traveler being different people, when they knew from day 1 that it would be a necessary feature. It would have been little extra work to incorporate into the data model from the beginning, even if the two values were always the same for a while.

I think the majority of the technical debt I've seen that isn't ci/cd related is disconnect between the domain model the product team is working in and the data model the engineering team is working with. Formalizing that domain model is now one of the first things I do when joining a team, so everyone agrees on precisely what the major nouns and verbs are and how they interact. Not just for the current system, for where we think we will be in 2-3 years. With everyone doing agile, it's amazing how many incompatible, un-written assumptions you discover that hadn't been ironed out.

They shouldn’t be checking in after two months. They should have regular check ins.

They shouldn’t have had such a vague brief.

They should have discussed a wide range of off the shelf options from the get go to see if they needed something bespoke or not.

A restauranteur wouldn't develop a menu without input from a chef or prior experience as one. A layperson wouldn't design a real building without input from an architect or engineer.

Yet for some reason a myriad of non-technical people (read: laypeople) feel empowered to design, spec, and strategize about software. It still boggles my mind that "product management" is a real profession.

Further, project managers tend to focus on delivering the product stakeholders have asked for on time and on budget.

Product managers however I find are frequently searching for a new expanded scope of stakeholders to brag they're solutioneering to. I find them more often in the "solutions looking for problems" business than I find developers, or at least developers are only doing it on a micro-scale, while product managers will take an entire team/org on 6 month fruitless missions to build software destined for the dustbin.

That sounds really high to me. I personally found 15k way too high already; guess it depends on the details/market/circumstances

Either you have the bootstrap founders that will trawl through fiverr to find the cheapest labor (the same types that will scoff and get insulted at anything over $10k), or you have the well-funded founders that will pay whatever it takes.

From experience: Cheapskates and low-ballers will never be happy. They always want something free, more haggle room, more discounts, and always have high demands and expectations. The best thing one can do is to price yourself away from them.

all the author needs is a wordpress site with a couple plugins and a few weeks of back and forth on the design work. if you can't make a good profit on that with a $15k invoice, you're doing something very wrong.

From October 2022 to now the shareprice tripled, adding half a trillion dollars to its market cap.

Explain that in terms of the Efficent Markets Hypothesis? "share prices reflect all information" - what half trillion worth of information change came out in 2022? and what in 2023?

What about insider trading laws? How can we say "share prices reflect all information" when we know there are people who have more information which would give them an unfair advantage, which means the current shareprice cannot be reflecting the information they have?

1) it's debuggable 2) it's fixable 3) it does exactly what you need to do how you want to do it without any extra cruft and nonsense.

Libraries or game engines are too generic to be fast and easy to use because they need to solve for every possible use case. And even then, there are still edge cases where what you want to do cannot be done because of the architecture of the thing and so you're stuck with a slow weird ducktape solution to get around the 3rd party code's limitations.