Ask HN: How do you detect oddly performing machines in a large cluster? | Hacker News Reader

Ask HN: How do you detect oddly performing machines in a large cluster?

1 pointchucky_z5 comments

Hi HN,

I have a problem where random nodes in a large cluster perform pretty far out of spec. What's the correct way to find them? There's a huge diversity of workloads, and the boxes are large so I was considering doing something really trivial that eats up some consistent % of CPU then graphing machines that are some n of SD out of normal (calculating fibbonacci to some low-ish number, for instance).

Is there any really good, clean way to do this that solves the problem in an elegant way?

View on HN

Is the problem data collecting and detection in time series, or more being able to reproduce so you can work towards a fix?

The latter. We have a lot of metrics and data, but I need something consistent beyond "this box looks like its using a lot of CPU/memory/IO."

Our workloads are dynamic enough that a box using 20% or 80% are both within range of normal, with some single threaded workloads so per-core stats are also not great.

I'm also not looking to fix anything, just have the ability to say, with a high degree of confidence, "hey, our system detected this box looks funky, can you just pull it out and replace it?"

That's the hard part of the problem, extracting signals from the noise. Have a look at the Google Autopilot paper (cloud scheduling, not self driving). There are a few interesting ideas that could be adapted. They generally have some good ideas if you peruse their research page, specifically the data center and related topics.

Awesome! For anyone who finds this through search, here's that whitepaper: https://dl.acm.org/doi/pdf/10.1145/3342195.3387524

I don't think it helps at a server level, but it does provide me some ideas, especially looking at some of the algorithms they're using.

My problems are mostly CPU related, whereas this paper seems to focus on Memory. I still can definitely get something useful from here. :)

Yes, most cloud applications / systems struggle with memory and optimizations around the system behaviors / configuration. It is rare to find something CPU bound, and when you do, it's usually isolated so it doesn't step on other programs. I thought the algos and strategy would be most useful to you. Glad you found it interesting!

Reply on news.ycombinator.com