Hacker News Reader

> It seems like a lot of people here have been in an ineffective large organisation and an effective startup.

I was at an effective startup that turned into an ineffective large organization before my own eyes. Your three (excellent) points ring very true.

However, the company actually tried to implement each of your three points and recognized their importance. The problem was in the execution.

Old products: These were neither retired nor given appropriate resources, so they instead decayed over time. Empty promises about longevity were made and then broken. Attempts were made to outsource maintenance to countries with the cheapest engineers, which backfired when each update become more buggy than the last. The end result was a lot of customers and early adopters getting burned and a destruction of our reputation in the industry.

Shared infrastructure: The C-levels saw shared infrastructure as an opportunity to reduce costs and speed up development, but they took it too far. At the worst, they wanted to have one gigantic team of back-end engineers, one gigantic team of front-end engineers, and so on, spread across every project across the entire company. Individual engineers were assigned to many projects and each had to deal with multiple managers fighting for their time. End result was that nothing got done and politics reigned supreme as the way to get attention on your project.

Best practices: The company hired “subject matter experts” to enforce best practices. They were expected to be involved in everything across the company. Imagine being hired as the “security best practices owner” and then told that you’re accountable for the security of everything at the whole company. Instead of promoting best practices, they went into full defensive mode and interfered with the shipment of every product and feature that they didn’t have time to work on. We would have products ready to ship and have the best practices experts panicking to halt it because they wouldn’t have time to look at it for months after completion. Getting them involved at the start was a pipe dream.

The overarching issue was that they tried to do big company practices in the move fast and break things startup style. For every well-intentioned move toward being a proper company they managed to screw it up by trying to inject startup-style speed and improvisation.

> I was at an effective startup that turned into an ineffective large organization before my own eyes

Somewhat related, I've been in effective teams that turned into organizations, a couple of times. One of the main issues I've seen is that they're especially prone to survivorship bias, on both accounts of process and people.

You get the usual "this is how we managed to get there, it must have been right, let's do more of it", and people get protective or suspicious if you propose anything else.

There's also a tendency of over-attributing success to the people who happened to be here while the team/product became successful. They tend to be promoted to responsibilities, their voice is over-amplified and their opinion becomes dogma.

The result is that rather than adapting to the turning point both organizationally and architecturally, you end up with dungeon keepers protecting their babies under the cover of "this is our culture". The growth is not painful and rather than re-thinking things from the ground up, we correct by patches and ad-hoc fixes - approvals, reviews, build queues, more planning, backlog sign-offs and such.

> There's also a tendency of over-attributing success to the people who happened to be here while the team/product became successful. They tend to be promoted to responsibilities, their voice is over-amplified and their opinion becomes dogma.

How do you fix this? Assume all success is pure luck and ban dogmatic things like style guides and code reviews and force entire rewrites of sowftware and infrastructure every few years so that best practices change and don't become dogmatic?

One of the allures to getting in early somewhere is that you get to build the foundation and become the thought leader and define to an extent which dogmas shall be adhered to because you have the context to make those decisions. I'm sure it can get overbearing when abused, but I also think it's quite presumptuous to join a mid-stage startup thinking you're going to come in and uproot everything and make things happen your way. If that's what you want go make the sacrifice and join a small place and earn it. And grow the business with it. And then incorporate it into what you can bring to the table as an experienced industry professional so you can be hired as a director somewhere else when you're ready to move and do it at larger scale. IDK...

    The overarching issue was that they tried to do big
    company practices in the move fast and break things startup
    style. For every well-intentioned move toward being a
    proper company they managed to screw it up by trying to
    inject startup-style speed and improvisation.

What would have been a better way of handling the transition?

> What would have been a better way of handling the transition?

The problems all came from the top-down direction. Replacing the CEO with an experienced big company leader could have solved so many issues.

The CEO was good at founding startups and running small teams, but that’s all he really wanted to do. He was trying to run a big company like a small startup and actively fought big-company organization styles.

Companies like Microsoft, Facebook, and Google don’t try to pretend they’re still startups. They embrace the fact that they’re a big company and they make it work. They’re not perfect, but it’s better than a slowly failing big company that can’t accept that change is necessary.

>it’s better than a slowly failing big company that can’t accept that change is necessary.

Or worse, a big company that is in a growth/high profit area that takes in enough money to not fail despite dozens of missteps.

If you do the wrong thing long enough and don't go out of business, then the wrong thing becomes "our way".

One of my big realizations as I've gotten more work experience is just how much momentum matters in business. Brand awareness along with existing customers and relationships is huge.

A big established company can get away with a shocking amount of incompetence for a really long time, while a very well run startup can easily fail.

Scale that up and you can see the same effect with entire countries.

Mostly because annexation of conquered territory is illegal. Without it at least Eastern Congo, possibly all of it, would be part of Rwanda. There are a lot of failed states in the world. They'd fail at defending themselves from a competent military too.

Scale that up and you can see the same effect with entire generations.

So are we using google as an example of what to do or what not to do?

Yes

Agreed. It's sometimes hard to distinguish the things Google does that contributes to its success from the things Google can get away with (waste, etc.) because of its success.

Isn't that a conglomerate in a nutshell? "Our random incompetency averages with our random competency, such that we're able to avoid bankruptcy and remain a going concern longer than smaller, less diversified companies."

> If you do the wrong thing long enough and don't go out of business, then the wrong thing becomes "our way".

You just described most of retail banking and insurance (and I assume, utilities).

It took the 2008-prompted investment-retail bank mergers and ACA admin costs caps to nudge these industries into the 21st century.

The financial side of health care providers is still a nightmare (even though the actual medical side is amazing at continuous improvement).

So much this. There is a very big difference between running a startup and a large / publicly traded company. There is no shame in a founder hiring a good ceo to manage the company after the startup phase. I worked for a company where they essentially grew through grit and insane hours and became established and went public. Then everything started going to hell. CEO was just not prepared to run a proper company and as a result constant production crashes, employee and customer churn. Company survives by constantly diluting and selling more shares now.

But that's also extremely risky, it's really not easy finding good CEOs and not easy to tell if one is going to be good or not (see Apple replacing Jobs in the 80s).

This. The things that get a start-up to IPO are not what make a successful company after IPO. Different leaders are needed, Google succeeded by bringing such a CEO on bord. It is hard to see companies on a good trajectory, from start up to mature public entity, fail due to these purely self inflicted things.

This is very true. A CEO of a startup is used to operating under Dunbar's number. For him to not actually not know everyone is hard to admit.

There’s a lot of anti-patterns. Big orgs tend to reward careerism and punish risk. The biggest reason for that is that as you grow, the consequences of failure exponentially increase.

Startup pivots are failures made good. Big org failures means people get fired or arrested.

> Individual engineers were assigned to many projects and each had to deal with multiple managers fighting for their time.

The misunderstanding I saw across several organizations of different sizes was the mistaken rationale that surely, I should be able to take up "just" 25% of your time. When there are 5 other groups, there is nothing at all sure about that statement. Either you are lucky to get 10% or you're riding that employee into burnout and don't understand how they just up and quit one day.

> Old products: These were neither retired nor given appropriate resources, so they instead decayed over time.

This is the biggest one where I work. Call it technical debt, legacy systems, or aborted migrations - we have multiple generations of products doing the same thing.

Management comes in bright eyed with a desire to migrate to the new system. Not too bad at first, we get some quick wins. Then the 80/20 rule hits, people realise how hard the rest will be and we decide to put it on hold temporarily (i.e. forever).

This is the worst of both worlds, as we now only have the most complicated and ugly applications using the old system. There is nothing easy to train new people on. Over time fewer and fewer people truly understand the old system, work gets slower, and more error prone.

But hey, there's another new system coming along - maybe we can move to that, along with all the fresh easy work we setup in the last few years. Cue xkcd comic about a solution making more problems.

The other side of this is policy and procedure cruft: having so many different systems makes mistakes more likely. Every mistake means an investigation and an item added to a multi-page policy document. You'd never get everything done if you fully adhered to it, but it makes people feel safe.

Sounds like they switched to Matrix management. There must be some studies of when this switch is most effect, and to what degree.

I’ve concluded that anyone using the term “best practices” in earnest doesn’t understand them.

This is GREAT summary!

I work in a large organization, in an industry that is naturally slow. The work (in the industry) is slow because it's critical to deliver stable and reliable solution because large part of population and economy rely on this industry SW/HW. I have heard a lot of complains on how the process is slow, but in many cases, building on top of existing work takes a long time. It takes long time to understand legacy work, to go through older studies and projects, and to find spots to add contributions. Not all of us work on green field projects, but many of us maintain code/work that doesn't look "cool".

Another reason that I observed is that large organizations are extremely afraid of mistakes regardless of expected loss. Such companies appoint gatekeepers, and those gatekeepers are by nature risk averse. The gatekeepers will simply reject any idea by default, unless there is a well-proven precedent. As a result, anything moves slowly.

> Another reason that I observed is that large organizations are extremely afraid of mistakes regardless of expected loss.

Not what I have seen. Big companies with big profits will throw millions away on dumb ideas on the chance they make money, when they do not.

Also true. In the meantime, wanting to use Guava 33 instead of Guava 27 will get rejected immediately. Using Spark instead of AWS Batch is an absolute no no. I guess what qualifies as risk is also highly asymmetrical.

> Well, unless you're google and can regularly fire your customers without going bust

This is only true for external products. When I was at Google it always seemed to me that the majority of engineering effort is focused on the inside, and that the same effect exists there.

My experience as well, although my large org sometimes seems to move to slowly it's loads better than the startups I've been at.

One thing I'm realizing is really useful, and which we're doing more and more, is putting "program managers" in charge of projects, totally separate from "project managers", which really should be called "people managers". With that split we can have a senior engineer who is interested in management work across multiple teams without worrying about the day to day personnel issues. This only works if everyone supports this approach but here it's working well.

This sounds like terminology issues. Classically a project manager manages a project (a think with a start and an end, reasonably well defined) and a programme manager manages a set of related projects. People managers depending on intention sound like either engineering team leads, engineering heads of, or line managers.

One crucial difference is around authority. Traditional project managers can generally exercise authority over the people doing the work to get what they want.

With a program manager/people manager split, it might be the case that the person leading the project has to truly lead, because the actual authority sits with the people manager. May or may not be a better structure.

I don't find that to much of an issue - in fact its alot easier. having a conversation with someone about whether this is the best project in the company for them to be working on rather than whether they should be there at all.

maybe the first conversation leads to the second, but its not me :)

> Traditional project managers can generally exercise authority over the people doing the work to get what they want

I don't think this is that different; people have more static line managers, and are assigned to projects, which project managers then govern.

This sounds a lot like the matrix team structure. Can program managers assign work to engineers? And if so, how do you deal with project and program managers giving conflicting tasks?

Yes, the program managers can assign work and they tend to work closely with the involved people managers to balance workload.

I hate matrix management so much.

It's gimmicky and a pain in the arse. I know management does do a job, but it's hard to justify needing two instead of one.

My view I that matrix style management is a solution to a communication problem you haven't figured out.

Instead of fixing the communication throughput problem, your adding more bandwidth and scaling out the number of nodes in hopes it magically fixes the problem.

And it sort of does fix the communication problems, that's why people like it. The trade off is that as an IC its arese to have to jerk two cocks at the same time.

"At a large scale, it starts to be pretty difficult to maneuver if each dept of 30-50 people has built or contracted all their own infra & services."

This is one of the biggest risks I've seen, for both large organizations and small ones.

Large organizations are built of small organizations, and a bunch of effective small organizations will each be moving and shaking, and building or buying things that become part of their infrastructure. Then one day you wake up and discover that your overall organization is paying for 27 different solutions to the same problem.

Then, any attempt to collapse that down to something reasonable results in zero productivity for quite a while and whole lot of pissed off people.

The question is, is it really worth it to collapse it down?

That depends on so many things.

One of the things I've seen cripple large organizations is waiting for the common infrastructure / services becoming mature enough to be used in production.

Teams are kept in a holding pattern, sometimes for years, while waiting for the teams to deliver their common infrastructure / services solutions, and they're actively discouraged from building their own solutions, because the enterprise-wide solution is "just around the corner".

Internal FUD?

It depends, but generally yes. Collapsing to 1 may or may not be a good thing, but you probably want less than 3-5 different DBMS, CMS, Cloud providers, printing services, source control systems, office supply/equipment, etc.

And suddenly I'm reminded of the web app I put together that needed three DB drivers...

At some level, absolutely. Your organizations aren't usually totally independent units - they need to interoperate with other organizations.

To the extent that the spaghetti of infrastructure prevents effective interop, that's a problem.

And as the need for interop grows between two orgs the cost of the various hacks, shims, and other cheap adaptations, escalates, until unification of infrastructure starts looking awwwwfully tempting.

Of course, unification comes with its own set of problems.

> Some divergence is useful for agility, but a lot of it is just waste and actually slows you down when you want two depts to work together on something - or even prevents useful collaboration. An effective organisation will make the tradeoff consciously.

Often an effective strategy for this to purposefully diverge and converge. Allow a new product to strike it out on its own, and if it survives follow back up and homogenize the infra so that maintenance is cheaper.

Takes a lot of labor, but that’s usually something large organizations have in spades.

This is a great summary! Large orgs are designed to move slow, it's a necessity of calculated moves. For example, single policy, product or process changes can affect thousands of people and many millions of dollars. These things take time as the ripple effects are far and wide.

This is not to say that innovation isn't possible. In my experience, the way to forge ahead in large orgs is to achieve progress by example, advocacy and influence.

From what I hear, even Google is mired by years of product, organisational and tooling debt.

That would make sense as to why they kill of products, in attempt to reign this in. In reality, killing products is still pretty uncommon even for Google.

Just today, I logged into Google Calendar on my iPhone and the mobile UI looked like it hadn't changed since 2008 and it was missing a ton of features.

The mobile web UI for calendar has indeed been abandonware for about 5 years and even before that it was starting to look pretty crusty. I think largely because nobody cares and everyone on mobile uses the app. The mobile web interface of Gmail hasn't changed in years either but it is still maintained and also wonderfully good.

Yes, and it actively manages it. Occasionally, one of the potential outcomes of this management becomes visible to the outside world, when it sunsets a product. Other times, lack of investment in this management also becomes visible, when a product is clearly languishing.

As in any other company, headcount is always limited, and some products are funded by nothing but the willpower of particular passionate engineers/managers, which results in a very low lottery bus factor.

Mire is a good word to describe what is going through my mind when I see their recent search results.

What would you say are the drivers that enable effectiveness in an org, regardless of its size?

Just off the top of my head:

An effective responsibility culture and structure. Those who are responsible for something must have the means to effect it. There must be a means of agreeing who is responsible for something and an effective way to find who is responsible when someone is blocked. Things should not languish because no-one is responsible, nor be trampled in because everyone wants a piece. (This is culture as well as structure, because everyone needs to internalise what a good responsibility structure looks like so that they negotiate it into place in a distributed way, rather than having it imposed top down.)

Prioritise making intelligent decisions over raw speed. In my effective employer, if your project was late, you were required to take a step back and really work out when it was realistically deliverable and make sure you genuinely understood what was now needed - better to say ONCE 'it needs another 3 months' than 'just a another 2 weeks' 8 times. (The ineffective company had a 5 day horizon and would burn everyone out constantly trying to deliver next week) Related to this is that profligate agility can work against intelligence. The effective org could afford what would seem like a huge amount of process, but it actually took less of our time than the agile process in the ineffective org because we didn't need to change our minds as often.

Making effective use of staff's skills. The means everything from: Leaders need to recognise what they aren't expert in, not having a destructive people culture, giving people larger pieces of work that they can master rather than splitting everything into a million fragments, allowing for development rather than people getting type-cast.

  >  profligate agility can work against intelligence

thank you for putting into words something i've been inuiting for a while now... i couldn't put a finger on what was going at some workplaces i've been in, but this really resonates with me

it seems in our chase for "agility" above all else, we are sacrificing our intelligence along the way...

two keys: trust and (efficient) communication. these may or may not be sufficient for a given org, but they’re certainly necessary ingredients. they lead to highly effective teams via autonomy and intrinsic motivation (teams that achieve desirable outcomes efficiently without oversight).

these two factors are extraordinarily hard to cultivate consistently, especially during high growth. this is incidentally why management is so hard, and so often “bad”, because managers are often encouraged to focus on metrics, like financials, rather than cultivating trust and fostering effective communication.

kanban (in japanese manufacturing) is a quintessential example of a management system designed to foster trust (no-fault problem-solving orientation) and communication (simple, immediate, and well-known signaling).

edit: should also have mentioned kaizen as a companion to kanban in this context.

I will be curious to see how founder-led companies operate as they get large. Mainly companies like AirBnB, Cloudflare & Coinbase.

I don't think it's impossible for a company to get large and still be somewhat effective.

Can you give an example (non-code related, if you have one) of the first bullet?

Documentation. When you're working on a new build, everything is frequently just in your head or those of your teammates, and that isn't a problem initially. And if you have one main product, you don't need to spend a lot of time on documenting which resources were for what. When you have a bunch of old ones, which you may not actually often spend much time on, it's much more important to be able to figure out why that old thing is running, exactly what was shipped to customers when (in the case of physical product). Otherwise you risk having to do archaeology every time you turn round. Or just not having the information when some customers says that X has stopped working on a device that you shipped 5 years ago but guaranteed for 10.

> google and can regularly fire your customers

They don't fire the customers (advertisers) they fire the users.