Twitter was doing a lot of high ROI engineering work | Hacker News Reader

Twitter was doing a lot of high ROI engineering work

twitter.com

23 pointstosh11 comments

Ok great, that's all very cool, but do you know what the consumer experience is? It's sluggish as heck in the browser, it consumed way too much space in cookies (max I've seen is 85mb, the majority of which are loaded upfront) in the browser, and the app feels kind of jittery and constantly snatches posts I'm looking at away from me in my timeline mid-scroll.

The corporate side is also problematic as the company wasn't profitable. Not to say Elon is going to help, here, but fundamentally the economics of Twitter didn't seem to be working out, and they had had enough time by now to try and make it work.

> but fundamentally the economics of Twitter didn't seem to be working out

And $1billion a year in added interest payments is gonna help there?

No I specifically said it wouldn't. It's gone from a money fire to a money inferno, but that is a political choice as distinct from an engineering one.

> and constantly snatches posts I'm looking at away from me in my timeline mid-scroll

Instagram and Reddit do this too, it's the most annoying thing in the world. And making the feed crap is actively harder than just leaving it as it is.

It seems to be a common thing in IT to be completely unaware of business requirements, customer experience etc and instead think it's your job to do cool nerd shit instead

Look, I sympathize, cool nerd shit rocks and in the right places can be great. Often management doesn't make good choices about tech debt and optimization, too. That said, if engineers are lost in the weeds gold plating far more than is necessary or fighting unsuited tooling, either case is a management failure.

I just don't think twitter from inception to elon, as a now complete product story, is the great success that this thread is trying to paint it as. In the race, you don't get prizes for all the cool shit you do, you win or you fail, and twitter didn't win as it didn't make money and now maybe never will. Worse still, all this cool shit is cool proprietary shit so the best all these cool twitter engineers can do is tell me about it, so, fuck that.

Elon removing the stupid sign in prompt has already been the best consumer experience upgrade for me personally as a sporadic twitter user.

Except Twitter appears to have been profitable until 2019. They would have been profitable in 2021 as well if it wasn’t for a one time legal charge of over 700mm.

The idea that Twitter was fundamentally unprofitable appears to be false.

I encourage you to read more of Dan Luu's posts, because he talks about things like what you mention, and consistently makes insightful points about them, e.g.

https://mastodon.social/@danluu/109332405719026619

https://mastodon.social/@danluu/109339567207009389

The point here isn't just that the features are cool, but that they provide a good return on investment. At Twitter's scale, a 1% improvement in CPU usage or user retention is massive, and could be worth having a team around for in the hypothetical case that's all they ever did.

  If, in the lifetime of the specialist team like the kernel team, a single person found something that persistently reduced TCO by 0.5%, that would pay for the team in perpetuity, and Twitter’s kernel team has found many such changes. [1]

The "number of calls" metrics that Elon Musk have been throwing around without much understanding are probably themselves a product of Twitter's investment into performance - someone showed him that you can get a call graph of all services involved in producing the front page, and what he got out of it was "1000 is a big number". Presumably it will get rewritten until performance visibility is sufficiently reduced that he can't be bothered to micromanage it any more.

[1] https://danluu.com/in-house/

That's the problem Twitter over engineers every single thing way too much.

To know what the ROI actually was we'd need to know both the costs and returns of this type of work. Proposing a way to save $[redacted]M per year doesn't mean the ROI was high because it depends on the cost of having a kernel/compiler team. A team of specialised kernel hackers, in a place where you're competing against Google for them, in the Valley which already has high salaries, will be an expensive team to have. It's not going to be enough to deliver some improvements to the kernel scheduler that could save a lot of money if you only do it once. You'd have to yield massive savings like that every single year.

Technically to really understand this, you'd also need to do a calculation of what it'd cost to just outsource to a more specialized company like Red Hat or (for the JVM) Oracle itself, where the knowledge of what to do is already present and can be amortized over more companies. For example what was the ROI of Twitter writing its own compiler optimizations vs just making a deal to buy Graal Enterprise which has far more optimizations in it that improve Scala than the ones Twitter wrote? How much would their kernel changes have cost if they'd acquired them through a high-end support contract with a Linux distro company? How much money would they have lost from migrating to the cloud DB and suffering higher tail latencies, vs the added cost of writing and supporting their own database from scratch?

I don't know, maybe at Twitter the engineers were doing this sort of calculation on a regular basis, but my experience when I was at Google was that most engineers weren't doing so and indeed could not, because there was no solid basis on which to compare the cost of engineering time vs machine costs. Neither were exposed to the bulk of the staff in comparable units. Obviously at some level up the management hierarchy these costs started to become relevant and must have been taken into account but a lot of projects were bottoms-up and never did any sort of build vs buy cost/benefit tradeoff. In fact I don't recall anyone buying off the shelf tech or even proposing to do so, other than in the early days with Orkut and that was a maverick approach that was quickly rewritten onto internal platforms. In my time there I saw a document that compared the costs of using home-grown infrastructure vs open source commodity stuff exactly once (for Gmail) and it didn't even take into account salary costs yet the commodity stuff won by miles. Needless to say, that analysis was a very obscure document. They didn't need to care much, due to the prevalence of money and skilled engineers. Twitter doesn't have the same prevalence of money.

So overall this list looks to me more like a list of cool and fun projects rather than a concrete financial argument that it actually had a positive ROI. We just don't have enough information to know that.

Reply on news.ycombinator.com