Hacker News Reader

A lot of what you describe is attributable to resume-driven development, in my opinion. Complex solutions are used to insinuate that the engineers who implemented them are solving complex problems. This in turn improves their marketability by insinuating they've got the razzle dazzle and trendiness under their belt that justifies $500k a year.

Sometimes it's also not resume-driven development, at least not intentionally. Sometimes it's because the engineers just want to work on overcomplicated shit. For example I have my personal blog running on half a dozen AWS features - S3, Athena, Cloudfront, Cloudwatch, Lambda, SNS, SES, ...

I don't need any of that for a simple blog, and I write on it like once a year. And the site is still slow as shit (relatively) even though it's static because I haven't optimized the font and script loads to be non-blocking. But it was fun to set up!

There is also an element of each project optimizing to its own minimum. If each project selects the best language, tech stack, hosting stack, etc, for itself, it creates an overarching complexity across projects.

Fixing this requires lots of communication and leadership. Each project may need to select sub-optimal components (like language) for itself, but in doing so it can remove overall complexity.

What you describe you did with your blog is, in some sense, resume-driven development: you probably learned stuff in the process, which might be useful in a future job situation.

The difference is that nobody paid you for doing all that unnecessary stuff. Doing unnecessary things for learning is good, but not what a company is paying you for, usually.

If the resume isn't what drove them, then it isn't resume-driven.

I agree about the resume-driven development aspect, and I think it's great to use a personal project as an outlet for experimentation! Too many people do stuff like that at the expense of unnecessary complexity in production systems.

I'll grant that collectively an engineering team might decide to overengineer a simple, noncritical service to get production experience with a technology or two that they are targeting for future adoption. Nothing wrong with that, and there's nothing wrong with making room in the schedule for engineering experiments to test out new technologies.

Managers and the rest of the business, for their part, need to understand the need for experimentation so it can be done honestly and not under the guise of other work.

Folks with the outside perspective often attribute some intent to a situation, but if they went through it, it would be obvious and simple.

I think engineers try to do the best they can. It's just old guys might do it the mainframe way, slightly younger unix, then windows, then linux.

Or folks who've been with the culture do it that way, and new folks from outside the company bring new college grad or last-company-i-worked-for culture along with them.

And folks who try something new suffer from the 'fog of war' and your "the docker solution" that got everything started so well makes you a real expert in nested docker configurations and firewall issues. :)

I almost hate to say it, but the answer to your last question might lie in instituting the kind of standards, credentialing, mentorship and licensing that exist in the "real" engineering fields. Stuff that will correctly be called out as unfortunately really "hampering individual freedom" (including the freedom to be a jackass) and being "unnecessary" and "bureaucratic" and (quite intentionally and without quotation marks) gatekeeping.

Civil engineering is sort of the oldest and ultimate example, where a bad design puts people's lives at risk. In deference to said lives, senior engineers in that field review every drawing or calculation that goes out the door, and they also mentor the junior engineers, who have to be graduates of accredited schools, and also can't get certified until they've spent a certain amount of time being mentored and passed the exams, etc.

The mindset you're talking about is basically all about craftsmanship or elegance of a solution. Formal mentorship or apprenticeship programs do a great job of maintaining and transmitting that. School isn't really enough, and certain fields have recognized that fact.

In software however, and I'm not sure whether this is the problem or the symptom, the number of developers apparently (I was told recently) doubles approximately every 5 years. Which means the field is always overwhelmed with new people, full of energy but not experience, who want to do something cool and exciting, but haven't been, for example, burned by their own past decisions to try to do something cool and exciting. They see only upside. Which is valuable in itself, but they need more direction than they're getting, from the older engineers who have the war stories and who aren't especially concerned with being cool (which c'mon let's face it, is essentially a mating display) and who are boring.

I think traditional engineering standards would enable over-engineering rather than discouraging it. In fact, I think engineering standards would be every architecture astronaut's dream, because it would be an excuse to architect every small company's software to handle Netflix load. "This is an X type problem, and the standard says the data for X type problems has to be stored in a distributed event log." We would have to flip the standards on their head and write them to say, you are not allowed to use Kafka until you have a problem meeting certain requirements. Projections from our sales pipeline say we can only use technologies certified for less than ten requests per second.

Can you imagine a bureaucratic oversight committee rejecting a CQRS design with streaming everything and telling an engineer, "Seriously, WTF. This is a config console. It will have less than 300 users two years from now even if you live up to that insane hockey stick graph they showed the investors. Just build a basic CRUD app backed by RDS. Use the same front and back end frameworks you use for everything else. The engineering standard says you can't afford to waste time on this because there's a lot of other work you're going to need to do this year, and some of it might turn out to be hard, even if you don't know what it is yet." Can you imagine that?

Math and physics also have a strong cultural orientation towards simplicity, which seems to be universally embraced. If you solve a homework problem in half a page, and the professor calls on somebody else to put their proof on the board, and they walk up with two pages covered front and back, you don't think, "Wow, they're smart. Their solution is so much more complicated; it must be better." Conversely, if you and all your friends took three pages to solve a problem, and somebody else presents a solution that fits on one chalkboard, you're in awe. I really don't know what the difference is that makes the culture of software different.

I think if it's being enforced by standards or licensure, it's too late and too external. It needs to be internal and cultural, at a gut level, to aspire to do more with less. In software, the primary measure of cost, the definition of "less" in "do more with less," is the intellectual load on the software development team. (We have that in common with mathematics.) Do more with dumber. How do we learn to value that as a profession?

As you ask for simplicity, I think this can be succinctly summed up as a lack of engineering leadership.

I know having completely independent teams is the in thing but there needs to be some high level design and coordination. I'm not even against different teams using different tools but the integration story has to be there from day zero if it's a business requirement.

Ultimately there needs to be an engineer who has both the power and knowledge to coordinate design decisions.

I agree with this 100%, and I think the people providing this leadership should be in management. Software engineers need to be managed by someone who can understand and evaluate their work, because of what you said and also because other aspects of management benefit from the ability to cast a critical eye over the work being done as well.

EDIT: Going further, I think the people trying to hire me as an engineer to fix their problems should have replaced their front-line management with managers who understood code, so they had some clue what their engineers were doing wrong and how to fix it. The idea that engineers should be able self-manage is a rare best case scenario, not a best practice that can be replicated across the industry.

In other forms of engineering, the culture is very clear: simplicity is an achievement. Solving an easy problem with a big, expensive solution is not an achievement. It's unskillful. The best engineers come up with solutions that make you see the problem as an easy one. Solving easy problems with difficult solutions proves you aren't qualified to take on the problems that need those solutions. How do we establish this as a standard belief in software engineering as well?

My experience with things like foreign automotive components suggsts this is not exactly true in general... but it's true that monsterously complex solutions seem to be far more common in the software world.

Can you elaborate? Are you saying foreign automotive components are needlessly complex?

German sport / luxury cars are renowned for excessive "because we can" capability, and the complexity to go with it.

A cynic would say the complexity does not even have to bring extra capability; frequent and costly maintenance is an important part of the status signaling.

This is diametrically opposed to e.g. Toyota's simple, traditional, robust, and scalable transportation appliances.

That's interesting. In the UK Audi, Mercedes and BMW (to a lesser extend) have a reputation for reliability. Is that misplaced?

The German car situation in Europe is a bit different, several popular explanations:

- Lower spec versions are available and common; the brands themselves are not inherently special or upmarket

- This means mechanics who work on the brands do not automatically command a premium

- Parts are not subject to import duties and supply chain cannot get away with as much markup (after Brexit who knows?)

- Culture of better compliance with scheduled maintenance

- Depending on location, better maintained roads and milder weather

Also they have invested a lot in reliability in the 2010s and some of the reputation is a holdover from previous generation models. Problems will still be outrageously expensive to fix compared to Japanese cars but are not so disproportionately likely to occur.

My dad briefly owned a BMW as his midlife crisis car. The bi-xenon headlights looked better than any I had ever seen. But when one burned out it was $1500 between the part and the labor to remove practically the whole front end. This is considered a characteristic, “classic BMW” kind of problem.

I think part of the difference in mentality comes from the difference in material constraints.

If a mechanical engineer designs something that needs a ton of fancy moving parts, those moving parts significantly and directly impact their budget (I assume), because they're going to have to be purchased or made for each widget that gets manufactured. They also unavoidably impact physical properties of the object, like weight and dimensions.

Whereas a programmer can spin out as much code as they want, basically without material consequence. There is no constant gravitational force keeping them in check, requiring them to justify each piece of complexity as they go. It's ethereal, weightless, and its disadvantages usually only surface years down the line.

In my experience, the costs are felt immediately, because there is more code and more complexity. Virtually every cost in software development goes up when there is more code and more complexity. There are more bugs. It takes longer to diagnose and fix bugs. It takes longer to add new functionality. Performance is less predictable, and performance problems are harder to diagnose. Onboarding new programmers is slower. Fewer members of the team can do critical work on the code. Routine library upgrades are more likely to impact the code and take longer when they do.

Code is a liability, in other words. It may not be tangible, but you can think of it as weight that the team has to carry everywhere it goes.

You're totally missing my point, which is that all of those costs are abstract and protracted. They aren't felt as you're literally typing the characters; they can't be listed as concrete numbers in a budget. If an engineer is using a CAD tool and they add a part, they can see the space that's no longer available for other parts. Or they can see that the product would have to be made physically larger to accommodate it. There is concrete feedback to contend with during the development process, not just a vague notion of future reliability.