Finding zombies in our systems: A real-world story of CPU bottlenecks | Hacker News Reader

Finding zombies in our systems: A real-world story of CPU bottlenecks

medium.com

67 pointsfagnerbrack11 comments

This was a nostalgic pleasure to read in contrast to all the AI implications and tools posts. That said about AI, this as a case study would make an interesting benchmark.

Especially noted that the root cause here was pegged because somebody else had blogged about it. Our "digital commons" should be valued as other than an externality. Edit: by that I think that "the internet should be free" but that commercialization through derivative work should compensate authors. That's off-topic, so, nevermind.

Interesting story.

I don't know anything about the Linux scheduler, but if one core out of 96 is busy chasing after those ghost allocations, why would a network thread not use one of the other 95 cores, and instead be stuck for seconds? Unless that one core is blocking the entire kernel, but then that would be a lot more visible instantly.

It's probably a "bottom half", which is tightly bound to the CPU where the interrupt landed. And probably the memcg iteration uses a spin lock, which disables bottom half execution.

On NT, this style of problem would be slightly less likely. The job object, which is NT's memcg equivalent, is a handle based object that naturally goes away when the owning process dies.

I also bet systemd would have native mechanisms to clean this memcg state up if it was used for this purpose. Seems like a bug in the AWS ecs code.

Reminds me of this story from Windows land by Bruce Dawson (and similar on his blog):

63 Cores Blocked by Seven Instructions: https://randomascii.wordpress.com/2019/10/20/63-cores-blocke...

Looks like Linux isn't immune to these types of problems.

I know this is a hopelessly naive question but what is Pinterest even doing with all this AI training?

Recommendations and similarity search are core to a site like that. Those were classic ML topics before AI become synonymous with LLM in common news

I feel like they can't have any devs left at this point, the quality of the site code has gone down significantly in the last couple of years. Real shame, it used to be an awesome utility for bookmarking ideas.

Why does one core at 100% utilization on a 96 core VM cause the ENI driver to fall over, though?

Heck, even if userspace is trying to use all 96 cores to their fullest, I would expect the ENI driver to be high enough priority to pre-empt userspace and still not fall over?

Jeez. Surely the ECS agent could just query for permission to join a cluster, and disable if disallowed?

No, it can do whatever it wants. Just don't leak resources!

Thanks for the article!

Reply on news.ycombinator.com