How to scale a distributed system [pdf] | Hacker News Reader

How to scale a distributed system [pdf]

cdn.oreillystatic.com

214 pointsHenryR10 comments

Another quick hint: always make things work first then profile and optimize it. This happens when my colleagues were building a stream processing engine that scales up. At first, they start with all those fancy fine-grained locks but it turns out the final result just does not scale at all. They have to retreat to the version with some big locks and then figure out what was happening and how to solve it.

Am I the only one that found that there wasn't really much actual actionable information in here? There's basically no specific information about how they scaled along the axes they established were important

That’s totally a fair point. I could have given a more concrete talk about how I’ve done these things in real systems (and the original version of this talk, presented internally to my company, had more real details and spoke very candidly about mistakes that I’d made along the way). Here I wanted to give more of a “here’s how you might want to structure your thinking, along with some basic design principles” kind of talk - I wish I’d had time to give more detail!

Yeah it was an good talk from the perspective of laying out the landscape and I found myself agreeing with everything, only to find that I was close to the end of the slide deck :(

Is there a blog somewhere where you've written a bit more about it?

Unfortunately not (my blog is http://the-paper-trail.org but tends to be more theoretical than practical). That would make a good post though!

It's a bit dated but "Scalable Internet Architectures" by Theo Schlossnagle is still good (IMHO)

https://www.goodreads.com/book/show/853963.Scalable_Internet...

I see he did a talk on it: https://www.youtube.com/watch?v=2WuT2rdLK5A

I think of high level talks like these emphasizing timelessness. Technologies change over time and with in different contexts. But the theory remains as a way of thinking abstractly about a problem.

Besides, at the level of jobs I apply to now, interviewers care much more about my thought process than specific technologies.

I did a screen cast for these guys once. The majority of questions I got in reasons was where's the video.

anybody has a link to the video? google gives me only 3 min preview to the video :(

I think it is only available through Safari

Reply on news.ycombinator.com