Hacker News Reader

Comment by michaelt | Hacker News Reader

I've come to suspect the normal way large companies are organised makes performant software almost impossible.

Features are celebrated and make money - whereas performance can be gradually salami-sliced away, with nobody advocating for it.

If you can add a feature and it adds just 50ms to a 300ms pageload - it's unlikely anyone is going to block its release on performance grounds. And once the feature is released, there's no removing it.

This means performance gradually ratchets slower and slower - until it becomes intolerable. And by then, different performance issues will be everywhere in the codebase, so fixing them will seem an insurmountable challenge.

Far cheaper to ignore the problem. "Jira is only slow because customers insist on creating tickets"

View on HN

Development in any corp today, even middle-sized ones, is a big red tape festival. You have a project divided in sprints that are divided in (often meaningless) tasks, then you have a completely bloated CI/CD system and then things get deployed to three or four environments where people pretend to test everything. Things that might be developed by a journeyman developer in five days take six weeks to be done by a team of eight developers.

The whole process today exists to support a big class of managerial people(including scrum masters, PO's and PM's) whose salary depends on not understanding that the process is completely broken.

On the bright side remote development means you can work 1-2h per day and make management clapping their hands in happiness.

> The whole process today exists to support a big class of managerial people (including scrum masters, PO's and PM's) whose salary depends on not understanding that the process is completely broken.

I think it's even worse than that - those unnecessary processes only exist to try to help management attempt to manage content they don't understand in the first place.

And I say this as someone who regularly implements said delivery processes for customers.

Wasn't "Agile" development supposed to fix this, by keeping team sizes small and workflows as simple and iterative as possible? AIUI, the fabled "two-pizzas team" size was intended as an upper bound for the most complex projects only, not as the new normal.

(The "test everything" culture was also AIUI in the service of quick iteration; the first versions of Agile are from the early 2000s when dynamic languages were starting to get popular and there wasn't really any kind of static analysis.)

Yes.

Agile depends on not having a “big class of managerial people” micromanaging [1] every move done by the squad.

No methodology will work when someone is actively fucking up the process, and agile is no exception.

[1] In before some smartass comes to say “so you don’t want managers”… I qualified my statement with the word “micromanaging”.

The problem is not with the Agile methodology itself. One of its core tenets is valuing "individuals and interactions over processes and tools".

The problem is that companies don't understand what this means, and find it simpler to pay an "expert" who will bastardize this definition, and implement processes and tools that make them appear useful to the company. There's an entire industry built on this model.

Never understood the "two pizza team" since I eat a single pizza alone.

How large are those two pizza ? I can eat a small 8" alone but a slice from a jumbo 18" is enough... Also if someone eats a 18" pizza alone, is he a 10x programmer?

That's why software development teams are bigger in Chicago than in Rome or Naples. The size of the crust is bigger in Chicago-style pizzas, whereas italian pizza has a thinner crust.

It feels like a jobs program for a LOT of people, ultimately. This is why the ceremony is almost sacrosanct.

There's a lot of overhead and inefficiencies in how large companies do anything. It's hard to organize thousands of people well. It's the social equivalent of the old 'Command-line Tools can be 235x Faster than your Hadoop Cluster'[1] conclusion.

You put a hundred developers on a project and you'll generously get twice the meaningful output of ten developers (but 50 times the LOC; as demanded by conway's law).

The upside is that this makes it all the more feasible for a small group or even single individual to be surprisingly competitive. Peak productivity per person is probably is probably found in a self-organized 1-3 person team.

[1] https://adamdrake.com/command-line-tools-can-be-235x-faster-...

> You put a hundred developers on a project and you'll generously get twice the meaningful output of ten developers (but 50 times the LOC; as demanded by conway's law).

Based on my job experience, this is rather an artifact of how software projects are (badly?) managed.

Give each of the developers some small "hotspots" to work on (i.e. critical code fragment that is

- hard to implement correctly and fast

- which is critical to implement very fast and correctly

- a correct and intended implementation will likely consist of few highly sophisticated lines of code

), and the scaling will be much better. This is a little bit like theses are assigned to students: each thesis is one isolated non-trivial scientific challenge (often open research problem) that is to be solved by the student.

I guess the "management incentives" nevertheless prevent such a method: managers prefer/are incentivised to "tell stories" instead of explaining the deep, sophisticated challenges and solutions in making great implementations of these "hotspots".

This assumes you know what you need to build. 90% of the challenge in large scale development is nailing down requirements, especially when the problem space is too big for any individual to fully understand.

> This assumes you know what you need to build.

This is what in my opinion senior programmers or project managers are for. If they are not capable of handling this task decently, these people simply are not (yet) ready for this role.

Young engineer here in charge of a project and feeling quite out of their depth, I agree with this. Currently no mentorship and it will take a couple months for a senior hire. Do you have any advice? What does a senior engineer love/hate to see when they come onto a project started by engineers earlier in their career? How can I be most helpful?

My personal opinion on this topic is a little bit "postmodernist": we neither have yet a "scientific theory" of software engineering, management, and management of software project (despite the fact that there do exist people who claim otherwise). So, there exist a multitude of "schools" of how to handle these topics (with often quite conflicting opinions on what is "good" and "bad").

Having this consideration in the back of your mind:

What might be helpful concerning your questions is to consider that many programming languages suggest a specific style of approaching the programming project, structuring the code, often managing the team, ... . There exist good reasons why one talks of "Java shops", "C# shops", "Python shops", ... because these programming languages often imply very different company cultures. Note that very prevalent programming languages can also have various "subcultures" ...

So what will likely be valued is to have a good understanding of the "desired programming style", "desired problem solving approach", "desired management style", ... that is encouraged by the "programming ecosystem" in which the company is placed, and going by it.

This will likely yield a decent, conservative code base that can more easily be passed over to a more experienced programmer as soon as one becomes available: especially for such a situation, it is in my opinion much more important to "not to make huge (also architectural) mistakes" than to deliver the most fancy/ideal code (but note that if you work at a startup that has an "all or nothing" approach (i.e. if the product won't become great, the startup has failed) or is in fear of running out of money, the priorities might differ quite a lot).

Thank you for the response, it is helpful. I am not used to Python and know that I am not using the language well. So I think it is worth focusing on this.

> not to make huge (also architectural) mistakes

I'm noticing it's the first time I'm even having to make significant architectural decisions, which is difficult because I don't have much experience to draw from, so even the smallest decisions often require a lot of research.

Agreed. Also often true in small projects.

I think big projects are worse simply in proportion to their size. I once spent months with a client adding requirements when we were supposed to be doing UAT!

People take holidays, get sick and change jobs - having 1-3 ppl team means you'll hit brick wall that will hurt sooner or later, more so if at least one of devs was really good.

This is a real concern if you're looking at it as someone who is managing people... but a 1-3 person team does not need a manager to tell them what to do, and generally is not hired but spontaneously organized.

It's an inherently more unpredictable way of working, but that definitely cuts both ways. There are definitely more risks, but also far greater payoffs in terms of what you can do working in this fashion.

The principle of ensuring predictable velocity across a team comes a cost of limiting maximal produtivity amongst talented individuals.

It is better to sometimes be hurt than always be hurting. Organizations are so afraid of only 1 person understanding something that they create processes so that 0 persons understand it.

Fun part in context of big corporations is when performance becomes awful business will call in some guys from outside because clearly people currently working don’t know how to make it perform better.

Business skips part where for years they insist on pushing features out. Where there most likely are tons of easy fixes that can speed up performance if you give the team time to focus on that.

Going back to “outside help” then you get people who don’t understand current system and propose some silver bullet like throwing noSQL because that “will fix your issues” - but they don’t say cost is basically redoing the system, project goes under, third party guys count money, put another success story on CV because no one will validate it anyway…

> If you can add a feature and it adds just 50ms to a 300ms pageload - it's unlikely anyone is going to block its release on performance grounds.

I worked on a team that did this. It was easy to justify because the benefits of the feature were almost always counterbalanced by less usage of the product, and we had numbers to back it up.

> Features are celebrated and make money - whereas performance can be gradually salami-sliced away, with nobody advocating for it.

- Optimists tend to be promoted, so the higher up in the organization you are, the more optimistic you tend to be. If one manager says "I can do that in 4 months", and another only promises it in 6 months, the 4 month guy gets the job. When the software is 4 months late, the overall system complexity makes it easy to assign blame elsewhere, so there's no way to judge mis-management when it's time for promotions.

- There's a disconnect between engineering and marketing. It's not surprising -- marketing wants all the whiz-bang features, it wants to run in 16 megabytes, and it wants it yesterday. Although engineering would like the same things, it is faced with the reality of time limits, fixed costs, and the laws of nature.

- The complexity of our system software has surpassed the ability of average SGI programmers to understand it. And perhaps not just average programmers. Get a room full of 10 of our best software people, and you'll get 10 different opinions of what's causing the lousy performance and bloat. What's wrong is that the software has simply become too complicated for anyone to understand.

- There was never an overall software architect

- We should sell 'bloat credits', the way the government sells pollution credits

- SGI software has a cracked engine block, and we're trying to fix it with a tune-up.

From 1993:

https://www.seriss.com/people/erco/sgi-irix-bloat-document.t...

Get 10 different opinions and then have them actually implement them 10 different ways pick your favorite. Software development is already eye watering inefficient, getting people out of the way of each other at the cost of ten different implementations may just come out ahead.

> Features are celebrated and make money

I like your take, but I'm not 100% sure this is accurate.

Obviously to some extent features make money, and "featureless" software by definition couldn't exist. But I think a lot of software contains features that don't drive demand at all - the worst contender for this that I can think of is [Microsoft Teams' Together Mode](https://www.microsoft.com/en-us/microsoft-teams/teams-togeth...) which I doubt has led to a single sale of the product worldwide, but probably adds some maintenance burden.

I think a bigger part is, like you mentioned, the organisation of companies. Engineers like building things, and although maintaining software and improving performance can be rewarding, you'd struggle to find developers that are happy to only do that. Features also lead to promotions more than maintenance, so there's a strong personal incentive to create features even if they don't have a business benefit.

At Google they argued over microseconds, but there microseconds are money. If every software engineer working on search or ads added 10 microseconds each to load times Google would become unbearably slow extremely quickly and no longer make money.

Your comment remebered me of the speed-up loop: https://thedailywtf.com/articles/The-Speedup-Loop

>I've come to suspect the normal way large companies are organised makes performant software almost impossible.

I think this is a bit of an over generalisation. There were (and still are) many large companies producing performant and reliable software. I suspect when things just work (think of the global phone system) people don't realise how much software is involved. When was the last time you picked up a fixed line hand set and there was a software bug preventing you from calling any other handset on the planet - without varying delays or dropped packets?

Depends. In certain fields like trading software, performance is a feature.

>If you can add a feature and it adds just 50ms to a 300ms pageload - it's unlikely anyone is going to block its release on performance grounds. And once the feature is released, there's no removing it.

Unfortunately this is happening, because compute power is cheap these days. It allows programmers to be lazy, take short cuts and not think about good software architecture. There is no economic incentive to optimise, because human perception won't perceive a difference and thus nobody complains. It's wild to think that we have basically supercomputers in our home and yet developers managed it to bring it down to a crawl with things like electron apps.

> compute power is cheap these days. It allows programmers to be ecobomical, take short cuts and select appropriate software architecture for the business problem

Just an alternative take that I don’t entirely agree with but I often find rephrasing a problem in alternative tone it can help to understand it in non adversarial terms.

In this case I see strong parallels with the auto industry and food distribution

> It allows programmers to be lazy, take short cuts and not think about good software architecture.

I think it has little to do with laziness, and more to do with the taboo of reinventing wheels. If there's a very generic and very broad solution to your very specific and very narrow problem, you're encouraged to use (or invent!) the very generic and very broad solution.

This leads to everything being abstracted to the point where most of your application does absolutely nothing. Just look at the call stack of a modern java web application. The depth of the layers of abstractions that you need to navigate to reach anything that performs any real work is vertigo inducing and nauseating.

It far from ideal and I agree that economics is behind it, but I strongly oppose the wording that it "allows" programmers to be lazy.

I do believe laziness is a big part of it. Because laziness IS about economics - in this case, personal economics. Why expend effort when you can get away with less and get rewarded the same?

Your competition will benefit when you stumble. All software has competition.

Goodbye, Atom editor..

Reply on news.ycombinator.com