Hacker News Reader

Comment by lpolovets | Hacker News Reader

I interviewed at Google almost 20 years ago, and Bob was one of 5-6 interviewers that I had. I don't remember the other interviewers, but I remember Bob. He was very high energy, and posed a really fun, original question about selective logging. The kind of question that you could tell was born out of his personal experiences at Google, not just something he found in a "ten linked list questions" blog post. The gist of the question was something like "Imagine you have a very high throughput service, and you want to log notable events and errors, but not log terabytes (or petabytes) of data about successful executions. How would you design a logging system around these constraints?" We had a fun discussion about this question for about 30 minutes, and I remember being even more excited to work at Google after seeing that people like Bob worked there.

RIP :(

View on HN

This is very interesting but I fail to see an issue. You listen to all incoming log messages and only store != success. What am I missing (obviously)

I'm interested we well

It's a system design question. High throughout on the scale of Google means you'd laser any log ingestion service to death.

He was probably looking for a distributed way to label logs and build an ingestion service that shards regionally with high availability, but still able to search and filter within an acceptable time for debugging purposes.

How can I learn about these topics? Junior self-taught dev, I read these things and kind of see what they're about, but I wouldn't know where to even start with this...

Designing Data Intensive Applications (DDIA)[1] is a great introductory book for such things. It covers a broad number of system design topics, and you can then go into detail for stuff that interests you.

Other than that, several companies (such as Uber) operating on a large scale maintain engineering blogs.

[1] https://dataintensive.net/

You can practice the mindset by writing a toy solution for the problem, then listing the reasons why you consider your solution just a toy. This develops a fundamental skill of engineering, which isn't understanding whether a system will fail (it will always eventually fail, even if the first reason is that the sun explodes), but rather when and how it will fail. You can make a whole career out of pulling on that thread.

(I worked at Google with Bob in the mid-2000s. I might have interacted with him once or twice on the internal Java and Guice mailing lists, but never met him in person. He was supremely helpful and knowledgeable, and I admired him from a distance.)

But what do you log when there is a failure? A huge call stack, a huge object, some kind of information about what went wrong, what?

Reply on news.ycombinator.com