Hacker News Reader

Comment by vidarh | Hacker News Reader

Many of us will remember that Twitter in fact did start out with a monolithic database and had to rewrite a bunch of stuff when they couldn't make that work anymore.

Of course they could fit a much larger dataset on one machine today.

(But I will note the article is also assuming a chronological timeline by default, but that of course hasn't been true for years - the ranking Twitter does now is far more complex)

View on HN

There's a story by Bryan Cantrill [1] about how he went to Twitter to help them understand why it would take 400 milliseconds of compute to process a request (I'll leave the reveal to Bryan). Scaling horizontally is probably necessary for something the size of Twitter, but that doesn't mean that we can half-ass the code and just throw more machines at the problem. If we write code with a bit more mechanical sympathy and avoid the latest non-proven fads, we can surely write software that is 10x faster and requires much less scaling.

[1] https://www.youtube.com/watch?v=LjFM8vw3pbU&t=3240s

Oh, absolutely. I think in terms of making people think about putting things in-process in RAM and being a bit more imaginative at looking at machines sizes as worth exploring is a good thing, and the article is interesting as a look at what you can do.

Many things can plausibly fit on a single machine irrespective of uptake (e.g. I worked on a system not long ago where even if we cornered the entire global market and the market expanded several magnitudes in size, our entire working set would still fit comfortable in memory).

But even when you need to scale, it absolutely can scale better if you're willing to not automatically resort to a standard database stack for everything for example.

Twitter used to (not sure how much did that idiot fck it up) be quite great from this perspective from what I gathered. They had technical blog posts and are even featured in the famous data intensive book as an example for many scaling problems. Sure, they didn’t write it in assembly but instead use Java/Scala with Graal, but architecturally they had a sound system (plus what people routinely forget, the system should handle the general load, but it worths nothing if it fails at peaks, and twitter can easily hit a billion daily users during big global events)z

Fixed timestamp: https://youtu.be/LjFM8vw3pbU?t=1613

Reply on news.ycombinator.com