Hacker News Reader

Comment by puzzle | Hacker News Reader

> Google is still relying on Borg for its internal production infrastructure, but I can tell you it’s not because of the design of Borg interfaces!

No matter how spartan, the Borg status pages are more helpful than most Kubernetes UIs out there when it comes to debugging a problem in depth, i.e. past CPU and memory graphs. Part of that is made possible by applications exposing debugging endpoints and telling Borg about them.

View on HN

Yeah, I would kill for the k8s pod contract to have something like the borg status line.

As a matter of post-SLA curiousity, what is in the status line?

It's something we've talked about on and off. Never seems to make it happen.

Can't you just use CAdvisor and route metrics to Prometheus/Grafana ?

They're sometimes interactive HTTP endpoints, not metrics, although metrics are served by one of them (/varz, described in the SRE book). They're informally called z pages and are the inspiration behind the /debug handlers that Go's net/trace package installs:

https://pbs.twimg.com/media/CebGW64W4AAJ_sc?format=jpg

OpenCensus has a bunch, too:

https://opencensus.io/zpages/

Yarn apps typically expose a similar http endpoint for app status reports.

Yeah, but with Borg you can have many of them, all directly accessible with one click from the Borg job status page: e.g. a general status, a view of the work queue if it's something like a crawler, incoming and outgoing RPCs, memory/cpu profiles, etc. You don't want them all on the same page for performance reasons: they would eat up a lot of temporary memory/cycles, hold important locks for long periods of time, etc. And if you need to look at a bunch of containers, you don't want to navigate to subpages for each of them.

Reply on news.ycombinator.com