Hacker News Reader

> If you are an engineer and you start moving up into senior leadership roles, understand that this is your job

Product wants new features and new products, and they want them now. Sales wants new features and new products, and they want them yesterday. Finance people want everything to be cheap, but since most engineers have no idea how to prove to them with actual hard data that it's cheaper in the long run ship features more carefully and slowly, and build things with reliability and robustness in mind, finance people go in the opposite direction and think that hiring inexperienced developers from the cheapest places on the planet is the way to go, because "cost of R&D per head" is a very easy number to quantify.

Unfortunately very few companies have an engineering department that can overrule all of product, sales, and finance. In general even one of those three being stronger means engineering never has the buy-in to take the time to do things the right way. And if you're the engineer who consistently pushes back product and feature releases because you want things built correctly, you get fired, or at best marginalized.

Interesting that you mention Google, since it has a reputation for being more engineering-focused that most companies. From what I read on HN (perhaps not a representative sample, but...), it seems like Google has a culture of "build something cool to get promoted and then stop caring about what you built because maintenance doesn't get you promoted any further". If true, I think that very easily explains Google's "hot messes".

> Interesting that you mention Google, since it has a reputation for being more engineering-focused that most companies. From what I read on HN (perhaps not a representative sample, but...), it seems like Google has a culture of "build something cool to get promoted and then stop caring about what you built because maintenance doesn't get you promoted any further". If true, I think that very easily explains Google's "hot messes".

Countering my Gell-Mann amnesia here (many times I read about my area of expertise I see snappy criticisms trickle into the mainstream when the reality on the ground has long moved ahead and the supposedly insightful HN criticisms are cliches being routinely addressed within the expert circles, think "deep learning is just X, it doesn't address Y!" or "physicists are obviously boneheaded and got stuck at assumption Z!").

So, with the above in mind, I wonder if Google is actively countering this. If random people know this as a stereotype (and I read this often, and hear it as a stereotype of Google also in real life), then perhaps leadership is even more aware and either decides not to change or has already implemented things they think would help. Any current Googlers to chip in?

It is not. It is only getting worse. Src: recently departed Googler.

Exactly right. Thank you for writing this. I’m in a similar position, where I’m responsible for reducing tech debt, reducing internal operating costs, while ensuring the infrastructure still operates, and while being responsible for testing and pushing out releases on time. With 4 people on my team.

If we dropped every new feature today and just focused on tech debt, it would take us months to catch up. But in the meantime, the show must go on, so we just continue accumulating debt. Nobody in engineering, nobody in sales or product management even has the power to stop the madness - this is all driven from the customers. And it gets worse if you are a b2c business or provide services to a government or two - you just don’t have much wiggle room.

The most plausible explanation, to me, is that operating the product was not considered the primary purpose of the acquisition.

To be less subtle: it's to eliminate competition.

If my neighbor has a nice big cow, it's not crucial for me to obtain that cow for myself; it suffices that it die, while my cows live.

Nitpick but this implicitly requires I buy-in to a zero sum system

It's enough that you understand that others have that view. You don't make the rules. The view of others shapes the incentive landscape.

Things aren't zero sum, but our psychology has developed for a more or less zero sum environment. There was no exponential growth or constant innovation in every generation in those times.

It is valuable to learn what drives people and what interpersonal forces elevate some strategies and suppress others.

Many things are in fact zero-sum systems.

It doesn't; it requires you buy into the economics of monopoly.

Your neighbor doesn't want your cow to die, it's the megacowcorp trying to charge four times the equilibrium price for milk.

No, it does not.

Just mathematically the thing you said is clearly false. Suppose I say that, for some system, the return to each player is a function of the number of players: r(n) = 100 / (n+1). One player makes 50, two players make 33 each for a total of 66, three players make 25 each for a total of 75. Clearly not zero-sum, but I can still move up in the world by killing someone else's cows.

One might observe that much of the world is like this; not truly zero-sum, but with payoff matrices that incentivize hurting or helping strangers.

Note that almost all traditional microeconomics is this kind of curve-following system, one way or another. I'm not saying that kind of economics is right or wrong, just that it's a very traditional model, worth understanding as a starting point.

What is being described is a net-negative sum strategy. Value is destroyed, and the monopolist is able to extract more rent because there is less competition.

Nitpick on nitpick: .. or it could be a complex non-zero-sum game.

A kills B's cow, so that milk supply decreases, price increases, and C pays more to A for the milk.

It can also signal that the acquirer is now relevant in this market space, and to buy shares in the acquirer is now an investment in this market space.

It should be noted that I strongly suspect the "declining usage" and "unsustainable maintenance mode" was a self-fulfilling prophecy.

I would frequently want to tune in to a Periscope of a commentator, and then discover that if I locked my phone to listen to, rather than watch, the commentator, the stream would immediately halt. That shitty behavior that EVERY FREAKING USER complained about made me stop using the app.

I've seen this many times in my career: The suits refuse to invest in a piece of tech, and then cite the loss of users as a reason to not invest, despite clear evidence that the loss of users is caused by the lack of investment.

Per the announcement, the live video infrastructure that powers Periscope will continue to power live video on Twitter.

Costs to support a standalone app are more than just infra and CDN costs - think support, moderation, legal, comms, QA, etc.

I had read that, and so the 'cost' is the marginal cost of "support", "moderation", "legal" (assuming no features QA is probably not huge). So is it possible to operate a feature that has user generated content at at a positive margin? One without compromising privacy or incurring an untenable liability? I accept that this may not be possible, but it seems like that is where the engineering work is.

And yes, I know product managers are "we need new stuff" to push all the time. And sales is constantly trying to sell something you don't make. But those are execution issues that good management will moderate.

I don't want to be snarky, but there can totally be a non-technical outer context where all this was a very good strategic move.

Certainly, one should not always assume dark Machiavellian moves, lest we become too bitter and cynical, but business isn't engineering.

Read the book "The Elephant in the Brain: Hidden Motives in Everyday Life" by Kevin Simler and Robin Hanson. If not for other reasons, to acquire a different set of mental tools to understand motivation. Because I feel like you seem technically proficient but lacking in imagination of more important games of power in the background. And these aren't Disney villain machinations, they are normal everyday things that people implicitly understand and participate in, while being perfectly ordinary, nice, polite people, no mustache-twirling villains. Eat or be eaten. Periscope was eaten, now they are being shat out.

> When I was at Google it amazed me how many pieces of infrastructure were 'hot messes' in the sense that the people maintaining them didn't really know anything about how they worked, just how to run a test and revert if they broke. But there was no senior leadership pushing to understand the component interfaces and how the 'fleet' was at risk or interconnected. Just a bunch of fiefdoms. And inevitably people who kept things complicated so that they were "irreplaceable" for their own job security anxieties.

This is the curse of a large bureaucracy. It seemed like Google was immune to this for a while, perhaps due to the engineering culture, but it's only a matter of time when there's enough money and organizational inertia at play. Once things reach a boiling point, the transformational programs pop up, and the cycle continues.

> 5 years after acquisition

Similar timeline with the Vine acquisition and subsequent dismemberment, except Twitter announced Vine's discontinuation after 4 years rather than 5.

> and NOT retire the technical debt?

I would be surprised if this were the honest reason, at least more surprised than I'd be were it any other web giant, mostly because Twitter seems to have a mature grasp on why an average tenure of 2 years is insane for an engineering job and attempts to avoid the problems you mentioned. I reckon Twitter had motivations that simply don't scan well with people who enjoy using a product of yours and don't want to see it quashed.

My bet is that it doesn't make sense as a separate product when every other platform has live streaming. As simple as that.

I disagree. It is NOT your job to fix any of this sh*t given the way you are rewarded and grow your career in BigCorp is by shipping new features and growing the customer base. Cost saving or optimization is barely ever an important priority in any tech organization BigCorp that wants to attract and retain talent because most developers wouldn’t want to work there - there would likely also not be delicious food in the cafes and kitchens either. Startups similarly spend money extensively to attract talent and keep them, not by being cost, efficiency and order focused.

> how is it that you go 5 years after acquisition and NOT retire the technical debt?

That seems like the most simple thing in the world to me- you just ignore it. Have you ever talked to a product manager about "technical debt"? Newsflash- they don't give a shit.

I don't know how things are run at Twitter but in general, people want to do "impactful" things and get promoted/get raises. Paying down technical debt is the opposite of that.

You're moralizing about "this is your job" and all that junk. Feel free to do that, but the incentive structures are what they are. And it really ain't about morality, Jack. I'm not saying it's right or good, of course.

People should be thinking more in terms of game theory, incentives, iterated games or one-off games, the effects of imperfect information, lemon markets, delayed effects and the difficulties of credit assignment, job hopping and the lack of long-term reputation the way we evolved to have in close communities we stayed in for our lives in the ancestral environment, etc.

>> Have you ever talked to a product manager about "technical debt"? Newsflash- they don't give a shit.

Eng manager turned product manager here.

It's not that we don't give a shit - but paying down the debt isn't always worth it. For example in this case, if they always knew they were going to wind down Periscope (or at least that it wasn't strategic compared to what they were doing with Twitter) then investing deeply into it would have just been a waste.

Here's an example.

You buy a house in the spring. You know the heating system is very old and will likely fail soon (technical debt.) Should you invest in it?

The answer is "it depends." If you plan to live in this house for a long time, then yes because the last thing you want is for heat to fail in the middle of next winter.

However, what if this is just your summer residence? Suddenly fixing the heat is much less urgent (you still have the debt but the risk of it coming due in a very bad way is much less.)

Or, you bought the house for the land and plan to tear it down. You would be stupid to fix the heat.

I think the Periscope situation is much closer to that last thing. They bought it for something other than the stand-alone Periscope product so throwing a bunch of resources at it didn't make sense. Which isn't the same as not giving a shit.