Hunting Tech Debt via Org Charts | Hacker News Reader

Hunting Tech Debt via Org Charts

bellmar.medium.com

250 pointsmbellotti87 comments

High turnover also contributes to tech debt. When turnover becomes ingrained in company culture, engineers have no incentive to care overly much about the overall health of the code base.

I can attest to this because the company I currently work for has the highest turnover I've ever experienced. People are barely making it 8 months, meanwhile the bosses are pushing for faster growth to match our newish funding. That's leading to people who have been here for 2 months hiring _more_ people to replace the people who left at 8 months. Just about every project is suffering from simply not understanding what it does or how it works, which leads to more burn out which causes even more churn. I've never seen the cycle this close before and it really is wild just how bad it can get.

The current job market is punishing poorly run companies. It's too easy to pick up the phone and get a new job quick these days if you're unhappy, contributing to churn.

When does the market for engineers turn into a buyer's market?

If the economy goes into recession or worse, do those playing hot potato get stuck out in the cold?

It's a cycle, like anything else. Tech was largely spared during the last recession. Who knows what's going to happen during the next?

Cycles of arbitrary periods are nearly indistinguishable from a random walk. My guess is the latter.

What does this mean? Can you please rephrase?

ELI5: "Who knows?"

Here's what I think he means.

If you look at trends over an arbitrary period, they may look useful.

If you look at the similar trends over a larger period, they look different.

Therefore, the smaller periods are just random variations relative to the larger periods.

Close. It's more a comment on the structure of those random variations, and whether it's appropriate to describe the system as periodic at all.

For example, sometimes it rains, and sometimes the sun shines. Is the weather cyclical? Depends on the place, time, and time-scale you're considering.

Maybe a better example is roulette. If you watch the "cycles" of red and black, you might imagine you could make predictions. Gamblers are often fooled by this randomness.

This is a good observation that I upvoted because it's very relevant. A lot of things look like cycles but have no oscillating dynamic, it's true.

But you could suppose that business cycles are caused by a build-up of bad debt. Things to well, debts can be paid, collateral gets worth more, more investment, etc. This can't go on forever and the reverse cycle then occurs, often as rates come up. Soros store about this and called it reflexivity.

Of course I'm not saying business cycles are definitely caused by this, there's probably more to it than that. But it's a common explanation.

> caused by a build-up of bad debt

Why does it build up? A generalized autoregressive model explains that just as well as an oscillator does. The key difference is how predictable the periods are. The more waves you're composing to force the oscillator model to fit, the less likely it's appropriate.

Many people look at a chart that goes up and down and imagine they see cycles. It's popular to speak of business cycles, for example. Unfortunately, it is not clear that anything in the economy oscillates such that it's appropriate to describe the system as having cycles.

You might have heard of Fourier transforms. Any time-series can be transformed into its sinusoidal components, even if a composition of waves is not a good model.

It's possible, and in my opinion more likely, that the business cycle is not a cycle at all.

I mean, this is pretty much the explanation for share price movements, so, yeah it should be for everything else.

Consider my mind blown - thank you !

Once the talent pipeline fills up (not likely) or leadership realizes that technology is a mixed blessing and may not be worth the complexity. Right now you have to have technology to compete but eventually mid sized companies will realize it's not worth the millions a year it takes to try and keep up with the Joneses and keep the technology as simple as possible.

However, it's also difficult to find unskilled labor so the only alternative is automation. Just look at all the companies that won't answer a phone anymore and force you to do everything through an app.

So IMO tech wages will remain high for quite some time.

Yes who cares

Take the risk or don’t

Yup. I'd never blame someone for taking a better job elsewhere.

How is everyone moving and also moving somewhere better? Is it not the case that the majority of open job opportunities are ones other people have recently left from unhappiness?

I think there's a lot of "grass is greener" moves happening. Unless you have trusted friends working in a company, it is very hard to tell if it actually is better than your current situation until you've tried it. None of the sources of information available to an outsider (including Glassdoor reviews and HN posts extolling the virtues or sounding off about the horribleness of a given company) are particularly predictive of what your individual experience will be, from what I have seen. There's just too much variability, and too much bias in each narrator.

Not only that, jobs seem to have a "honeymoon" period after which you start to notice more issues you might have originally ignored (so things seem fine for the first few months)

> How is everyone moving and also moving somewhere better?

Several of us exited a toxic company semi-recently.

For what it's worth, it was common for people to take minor cuts to total compensation. The old, toxic company had evolved to pay well because it was the only way they could retain people (however briefly) through the obvious toxicity. We just didn't talk about it publicly much.

But it didn't matter. Huge quality of life improvement and I don't really miss the extra money.

It seems like many companies are more willing to offer high salaries to new hires than to give incumbent employees competitive raises.

In addition to the other comments, better for one person may not be better for the next person.

I just ask if they do daily standups, and see what they say. It's a pretty good indicator. The more clueless the more religious they are about them (imho)

Some industries are so backwards and difficult to disrupt that many companies can just continue operating this way for many generations of staff turnover.

I have the opposite experience. When people typically stick around for the majority of their career, there's no incentive to capture knowledge thoroughly. Onboarding time was noticeably longer because of it.

The ideal is almost certainly somewhere in the middle.

The people who have only been on the job a couple months are probably not truly onboarded themselves, hard for them to document what they don’t know.

And everyone new has to run to the lifers...which means that the new people never learn deeply...and always have to run back to the lifer.

It takes a loooong time to learn a system when you have this dynamic.

Not only that, but they lose important context. The goals and priorities are often lost and the new guys end up relearning the hard way

Had to read quite a bit before Conway’s Law was mentioned, but glad it was.

Viewing organizations and software engineering through this lens has been a game changer for me personally. As an individual contributor I now appreciate horrendous technical debt and glacially slow development as systematic organizational issues, rather than blaming burnt out or lazy engineers.

Now I’m much more aware and critical of technical and executive leadership. “A fish rots from the head” is one of those quotes that resonates with me more and more everyday.

This is why my dream is to work completely alone. I still need employment to survive but hope one day I won't

Another good quote is "a cow with 2 owners gives no milk and eats no grain" (castilian proverb)

DuckDuckGo and Google give me no results for that proverb in English; what does it mean?

Each owner steals the other's grain intended for the cow and the cow dies and gives no milk?

I'm guessing it means that nobody actually feeds it or milks it: there's somebody else who can do it.

My guess: Each owner says "it's your turn to feed it", and so the cow is never fed. And when the cow does give milk, each owner quickly takes the milk, claiming they haven't got their share yet.

It's like the human-scale version of socializing the costs/risks and privatizing the gains.

That's exactly right. The original is "vaca de dos amos: ni da leche, ni come grano" (it rhymes as these things often do). Castilian proverbs are neat ("Refranero castellano"). Centuries packed into short sentences

The common english version is "The fastest way to starve a horse is to assign two people to feed it."

In other words: when everyone is responsible for getting something done, nobody is.

See also: Directly Responsible Individuals (DRIs) https://about.gitlab.com/handbook/people-group/directly-resp...

Seems similar to the saying that 'success has many fathers but failure is an orphan.'

Lots of re-orgs is also a pretty big indicator. A lot of VPs/directors need a reason to exist and they create that reason with reorganizations. This is basically the enterprise equivalent of pushing the food around the plate to make it seem eaten. With a re-org you can shed or minimize expensive to operate systems that don't generate a lot of clear value(i.e most tech-debt).

Out of curiosity, are you saying that most tech debt is systems that don't generate much value?

If you had asked me, I'd probably say that tech debt is mostly systems that do generate value but are built poorly or in a rushed manner. Deadlines creep up, devs crunch, they ship a "working" product but it has design flaws that manifest as technical debt.

But now that I think of it, I've definitely also experienced what you're mentioning (I think). I've worked on projects with questionable motives that ultimately end up cancelled or abandoned. The leftover code remains and continues to confuse newcomers.

Sometimes, there is also functionality grafted to the wrong place in the system, or on to the wrong system. This can happen for a lot of reasons: expediency ("we have to foo the bar, why don't you implement it there as the release window is just right for our purposes"), bad design, or changing requirements.

If it stays there for too long, it will complicate the design of the underlying system because it has to be kept alive for daily business to go on. Such features are sometimes hiding in plain sight, and are actually causing a lot of pain or cause long-term risks, but we are too shy or spread too thin to address the issue. And why should we? After all, it works at the moment.

For that I always remember the fable, where animals have bought all those expensive shiny musical instruments, but nobody knows how to play.

They are trying to emulate an orchestra, by rearranging the seats, in the hope that the right seating will make make them sound good. But the orchestra cannot play for the lack of individual skills and lack of the leadership.

I saw in enterprises, they are buying all possible software licenses, but nobody knows how to use them. So the management is constantly shuffling people and churning technologies, in the hope something sticks.

I think most technology professionals refer to the concept of Cargo Cult [1] to describe the same phenomenon as the fable.

[1] - https://en.m.wikipedia.org/wiki/Cargo_cult

The main takeaway:

Engineers prioritizing making systems easier and more consistent end up with overly complex, difficult to operate systems

Product managers prioritize shipping features and end up making the software development process slower

Security prioritizes minimizing risk of change and ends up maximizing the risk of out-of-date software.

Flat organizations prioritize open communication and end up with cliques and internal politics.

Most of the ills in this article would be fixed by better product management that has earned clout with senior management. For example, the product manager should be able to tell everyone what most likely constitutes a minimum viable product (MVP), and be able to defend that against, for example sales-driven feature creep. And on the other hand, product management should be able to communicate to engineering the value of time-to-market. If your product manager is unable to do this and is just tallying feature requests and trying to cram in as much as possible you are screwed in multiple dimensions.

In most organizations that has one, the sales department has an outsized influence on every other department including tech.

Hire a department full of persuasive people. Expect to get persuasion. On top of which, subverting an organization's decision making processes and channels is a core competency of good sales departments. A two-edged sword if there ever was.

> subverting an organization's decision making processes and channels is a core competency of good sales departments

Know of any good books on this aspect of sales?

I assume they're gesturing at commissions? Salespeople are usually paid based on a percentage of the deals they close, incentivizing them to promise more to clients in a given timeframe than the eng team can realistically deliver. Then sales usually leaves it to eng to readjust the client's expectations. That's the classic conflict between the departments.

I personally don't really mind this since at the end of the day the entire org exists to support sales, marketing, and retention. Within reason, if those groups say they need something to bring in revenue then they get it.

Yes of course, but if you don't do it right you might get the sales (and the sales people get their bonuses) but the retention will be abysmal. For example, a salesperson who promises features or deadlines without consulting the tech team about what's doable will cause unnecessary crunch, poor quality releases and angry customers.

If you replace sales & marketing with actual customers, I'm all in.

I thought that was a pretty good essay. I've never had Security running the show, but the other scenarios he discusses match my experience.

*she

Just so; my mistake. (Would have edited, but can't, so my error is preserved for posterity)

Tech debt isn’t a thing.

I mean it is because it’s a thing that engineers worry about mainly because of ego and pride. I know that’s triggering but let me explain…

As humans we all have ego and want our work to be important which is why tech debt is important to engineers who have a myopic view of their value and what businesses can actually bear.

- I have seen companies with loads of tech debt lose 60% of their engineering staff in a year with no impact to business.

- I have seen companies with loads of tech debt cause production deployment slow to a snails pace AND still have rapid consumer growth.

Engineering is important when it’s important but once product-market fit and a great moat has been achieved that value diminishes considerably.

Neither of these points argue that tech debt isn't a thing, just that it's impact is poorly understood.

Companies have plenty of real debt, but still grow. Accumulation of tech debt might be a necessary cost much like servicing financial debt.

Now the question really is - what is the interest rate of your tech debt? And that's harder to know.

The problem with technical debt is, in my experience anyway, that decisions have such a delayed impact that it's difficult for a group of people to understand cost and benefit. To work with the example you gave: You can tell that losing 60% of engineering staff didn't have a negative short term impact. How about 5-10 years later? How fast can the org move at that time vs if they had more staffing and less technical debt?

The whole idea behind the concept is that it _can_ be a good business decision to reduce spending (i.e. new features) and focus on paying off your (technical) debt for a while. It might as well be more important to make progress in the short term, no matter the cost. But the important point is that it's not a technical decision, it's a business decision. And the whole metaphor of technical debt is just one way of moving the conversation to that level.

If you had said ”tech debt typically isn’t as big a deal as engineers think it is” I would’ve agreed with you, but ”tech debt isn’t a thing” is hyperbole to such a degree that it stops being helpful, never mind true. I realize that the former is not as pithy. And I think you agree, given your last paragraph.

For example, I’ve seen companies where tech debt has made them unable to meet the moment and fail, whereas if they could’ve delivered something better, quicker, they might have had a lot of success.

Bologna. I've seen monstrous tech debt all over my Fortune 250, and it's pushing us further and further away from competitiveness and profitability. And it's not just the approach to software. They're also sabotaging long-term strategic viability by only hiring product engineers who are fresh grads from overseas, who never stay in one position long enough to develop or pass on institutional knowledge. Unfortunately/fortunately, it's a business that is ultimately doomed in the face of newer technologies that WILL, eventually, take over our market, so I guess it won't matter. However, I still think we could turn the corner of renewable tech if we hadn't sacrificed properly investing in people and tools. But that would have required serious leadership from several C-levels, and, well, that's just not how the world works.

"Well, who could have seen this coming?"

"Uh, me. 20 years ago."

This is hyperbole and based upon anecdotal evidence.

As a counterpoint I've been part of a company that "failed" due to overwhelming tech debt. I was one of the "lucky" few that was kept on after the redundancies in what was a shell of the former company (50 employees down to 10).

I should have left with everyone else but I had only just started at the company 3 months prior and didn't want to look like a job hopper.

Tech debt is a thing for any non-trivial product. The more tech debt, the slower it is to create working increments of a product and the more risky changes are.

I have experienced products of which the development has ground to a halt due to the system having such complex interconnections that every change required everyone in every meeting to assess the impact of a change. The root cause was constantly churning out MVPs with no thought of architecture and no overall management of the product or teams. I was the architect hired to clean up the mess.

Not being able to use the shiniest version of everything isn't tech debt, it's brownfield development or maintenance mode. Using deprecated versions (beyond LTS), ignoring security advisories, and not replacing unmaintaned OSS packages are. It's a bit facile to reduce all of that to developers just being juvenile divas.

In some cases, tech debt is as small a threat to business continuity as IT security is. That is, it isn't a problem till it's suddenly a massive problem. In other cases, it causes development cost to slowly increase over time, at which point it depends whether the organisation/cash flow can bear it. Your proverbial moat, or perhaps corporate momentum.

I would question whether incurring tech debt is generally really part of a greater strategy at the organisational level. It's a symptom of myopia. In any place I've worked, it wasn't a balancing act, but rather being willfully ignorant about the application lifecycle, technology, security implications and knock-on effects. No one said "let's move fast and fix 'er up later". At least not in earnest.

> Fortunately for us, the realities of flat organizations are pushing them out of favor and lively conversations about engineering leadership as a craft are taking over. Flat organizations are a nice idea, but they just don’t reflect how people behave.

this makes me wonder who "us" means.

i make the educated guess it's "managers", which were dead weight in the Apple engineering scheme I joined, which functioned super efficiently without them, as it was emphasized how flat was a feature, not a bug, for getting things done.

Flat organizations over a certain size just develop their own hierarchies usually based on seniority. If you can’t see them, you’re just on the bottom.

Flat organizations also end up building the same thing 4 or 5 times because nobody talks to eachother. As a consultant, I love flat orgs — it means we’ll be there for a while because nobody can really tell us to leave.

I think the critical Apple enabler of flat orgs was size limits by division, preventing what you saw. (And I was a couple jumps from the top, nowhere near a bottom, which as size was limited wasn't so much a thing it seemed.)

Is there another way to view the full article? I'm only seeing the first paragraph with no errors or popovers, and I'm not about to disable uBlock for medium.

Tip: if you can't read some URL, just prefix it with 'archive.is/' and it will take a snapshot of find existing one

https://archive.md/Cr6F2

clear your cookies for the webpage

> Engineering First Organizations

> Like Google or Facebook these are organizations where engineering trumps everything else. If you’re not in engineering, your pathway to promotion is limited.

Just browsing Facebook's board...

MZ-dropout SS-MBA Mckinsey etc PA-Accounting and Business studies NK- economics & management, Mckinsey again RK - lawyer PT- lawyer

Not being engineers sure held them back!

https://investor.fb.com/leadership-and-governance/default.as...

I know as many non-engineers as engineers at Google right now.

The idea that non-engineers are sidelined is wrong.

Same, a lot of people relocated during the pandemic but to the same places, some Googlers told me they had never hung out with engineers at Google till then

Yeah, especially that in the context of this article, the distinction between Engineering and Product is presented as

> Engineering wins by producing beautiful and scalable systems, in pursuing that they sometimes build too complex too quickly. Product, on the other hand, wins by putting new stuff in front of customers as often as possible.

The quality, in my experience, is very substantially lower. It is peculiar.

Board directors are not usually employees.

1) not necessarily true at all. The board is usually split exec/non exec

2) not true in this case. Two of the people on my list are execs

Boards are usually filled with:

1) Founder/cofounders 2) investors 3) people who do not work at the company and are not investors that people in groups 1 and 2 think will be on their side during important votes

I said usually, there are sometimes one or two but the board is there to supervise and thus is outside the normal corporate structure.

In the UK corporate governance code it is quite clear that a board should have an exec CEO and a non exec chair. The chair runs the board the CEO runs the company. The board oversees the CEO. It would be extraordinary (and pointless) not to have some execs on the board.

And it would be extraordinary and pointless if the board members were all employees, it has a separate function of supervision and thus is usually mostly non-employees outside the CEOs chain of command.

By now they are so big that engineering is not as important. But 10/15 years ago, engineering was definitely number one

Thiel arrived in year 1. Sandberg in year 4.

Facebook has been an investor led organisation from the beginning.

All startups are investor led organizations the moment they take on and raise capital.

Facebook has produced the best open source software seen in the last 20 years. React basically solved front end frameworks, and pytorch steam rolled tensorflow.

So what? The quote above says that not being an engineer will stop promotion...It is clearly not true.

The fact this structure can still produce good software is neither here nor there.

> The quote above says that not being an engineer will stop promotion.

Speaking as a DS who knows a lot of FB people, this is totally true. If you have a product team deliver something cool, the engineers on that team will normally (modulo variability) be rewarded a lot more than the DS people. Sometimes PM's get some of that, if the product is very successful, but the variability is lower and the mean is higher for engineers at that company (apparently).

Did anyone on the board get promoted within Facebook?

Facebook rains money on engineers that produce value, more than anywhere else in the industry. Clearly you don't know about this

Reply on news.ycombinator.com