“[Twitter] is doing 1000 poorly batched RPCs just to render a home timeline” | Hacker News Reader

“[Twitter] is doing 1000 poorly batched RPCs just to render a home timeline”

twitter.com

34 pointstosh20 comments

"Our app has performance issues even though we fired all the engineers who could fix it" is definitely a leadership take.

EDIT:

> As the former tech lead for timelines infrastructure at Twitter, I can confidently say this man has no idea wtf he's talking about

https://twitter.com/bgleib/status/1591875679377031169

If they were the engineers that could fix it, why haven't they?

I suspect most of Twitter's performance issues are of the too-many-cooks variety, where services are poorly coupled or have a lot of dependencies because a million different teams are working on each one. (Heavy layoffs are a good step towards fixing that, along with a technical redesign.)

I suspect this is not quite what that sounds like (possibly slightly misunderstood), really about the complexity of the task and the amount of data points, and human nature than it is some gross engineering mistake.

The engineers brought in are in the position where they are expected to trash the existing teams code. And then in the context of a slow page load, they might look at it and see up to 1000 of something in RPC and tell Musk and it comes out in a slightly mixed up way when he tries to repeat it.

The reason the number of calls gets high is (I guess) not from 1000 individual different types of requests, but from pulling many pages or records each of several things.

Anyway it is easy to trash other people's code especially if that's what your boss wants you to do to justify bringing you aboard. I have been the victim of it and also on one or two occasions done the same thing myself to some degree.

Another aspect is that no engineer really wants to maintain someone else's code if they don't have to. They are going to have their own viewpoint on architecture etc. and coming onboard is a prime opportunity to start a rewrite. So slamming the existing architecture is just an easy way to try to get that authorized.

I think it's a bit of an integrity test coming in and talking about someone else's code base. But people are unfortunately creatures of circumstance, and these things can bring out the worst in some as far as fairness goes. But again, even for a fair person it might be easy in that position to scoff at a codebase.

Most people are not particularly high integrity though so will happily take a dump when it comes down to it.

Pulling many records leads to n*n optimization problems. Which leads to run-away rpcs.

Not to defend Musk but, as a matter of fact, he has written his fair share of code so he probably knows what he's talking about in this case.

I just refreshed twitter's home page and for me it made 111 http requests (per Chrome's dev tools). That includes static resources like images and style sheets. So only ~10% are made from the browser and the other 90% must be due to microservice architecture on the backend, I suppose?

Where can I read more about Musk's coding prowess?

Because the little I've gleaned is that he insisted on porting PayPal to Microsoft stack, got pushed out, and picked up enough about rocketry to impress Carmack in casual conversation.

https://en.wikipedia.org/wiki/Zip2

"Elon Musk was appointed the Chief Technology Officer..."

He created a web based company as early as 1995. In a three person startup one cannot become a CTO without writing code. So my bet is he understands something about http vs local calls.

That's your whole evidence? He was a CTO at a startup which means he knows how to code.

No, he was CTO of a 3 person startup. He was the ONLY one that knew how to code for a while. He did it pretty obsessively and slept next to his computer.

Feels like he latched onto some number in a power point presentation on performance, without totally understanding it.

I'm guessing Twitter is a highly distributed system, so of course it's doing a bunch of poorly batched RPCs. That's more or less the definition of a highly distributed system, where all the data can't be stored in 1 place.

If it’s all same server then it can be batched up.

Also doesn’t the new fangled http/2 protocols and the like solve this, at least partially?

It can only be batched if the source and destination are the same. Different parts of the page likely hit different servers, and query different backends.

I really doubt they hit different servers externally

I phrased it poorly, but you have an http2 connection to some load balancer, and it fans out different URL paths to different services. The RPC calls are presumably done between backend services.

Does this account for pipelining and multiplexing because that’s like the whole reason those exist so that every client doesn’t have to think about batching and the browser can do it for you and present a fiction that they aren’t. Because Twitter definitely supports HTTP 1.1 and HTTP/2.

We’re currently in the process of ripping out a lot of client side opportunistic model hydration in favor of just making multiple requests because it’s faster.

I'm honestly curious, what is opportunistic model hydration? Short explanation would be great, I don't need all the gory details.

Probably: fetching related items or fields that the client didn't ask for... yet

Netflix has 1000 Uber has 4000

Why can't twitter have 1000?

Reply on news.ycombinator.com