Hacker News Reader

Comment by dumpster_fire | Hacker News Reader

It's a system design question. High throughout on the scale of Google means you'd laser any log ingestion service to death.

He was probably looking for a distributed way to label logs and build an ingestion service that shards regionally with high availability, but still able to search and filter within an acceptable time for debugging purposes.

View on HN

How can I learn about these topics? Junior self-taught dev, I read these things and kind of see what they're about, but I wouldn't know where to even start with this...

You can practice the mindset by writing a toy solution for the problem, then listing the reasons why you consider your solution just a toy. This develops a fundamental skill of engineering, which isn't understanding whether a system will fail (it will always eventually fail, even if the first reason is that the sun explodes), but rather when and how it will fail. You can make a whole career out of pulling on that thread.

(I worked at Google with Bob in the mid-2000s. I might have interacted with him once or twice on the internal Java and Guice mailing lists, but never met him in person. He was supremely helpful and knowledgeable, and I admired him from a distance.)

Designing Data Intensive Applications (DDIA)[1] is a great introductory book for such things. It covers a broad number of system design topics, and you can then go into detail for stuff that interests you.

Other than that, several companies (such as Uber) operating on a large scale maintain engineering blogs.

[1] https://dataintensive.net/

Reply on news.ycombinator.com