Fail-slow at scale: Evidence of hardware performance degradation in prod systems | Hacker News Reader

Fail-slow at scale: Evidence of hardware performance degradation in prod systems

blog.acolyer.org

64 pointsgodelmachine13 comments

I had a large scale issue caused by a bug in the fan control system reading a temp that was deadlocked. This caused cpu performance to tank, since the cpu was thermal throttled.

To make matters worse the environmental sensors reported no issues, because they were all frozen at a single reading as well.

> There was a case where a fan in a compute node stopped working, and to compensate this failing fan, fans in other compute nodes started to operate at their maximal speed, which then generated heavy noise and vibration that degraded the disk performance.

I've never heard of such thing, can vibrations from a fan really influence disk performance?

Yes. Loud noises and even yelling degrade disk performance as well.

https://www.youtube.com/watch?v=tDacjrSCeq4

This is one of the classics! I love it. Screaming at a HDD.

I started screaming in my cubicle watching this.

In the days of burning CD's I was always told that you should be quiet and not touch the PC when you were burning. In my experience there was indeed a correlation between noise and the success of burning. I'm not sure if this is a placebo effect or not, and if this also holds for reading CD's.

On the other hand, if you have a laptop with an SSD (which is non-mechanical) there shouldn't be too much sensitivity to vibrations, I think.

I remember how I didn't dare to touch the computer during burning other than move the mouse a few pixels here and there so it has it peace and the screensaver do not kick in. CDs were expensive and burnig was a hour-long process.

That upgrade path from 2x to 52x was so nice though.

I previously had a vendor replace hundreds of systems due to a poor fan design where they mounted it under a spinning disk. It is very surprising how sensitive they are.

Yes. Mechanical disks are very sensitive to vibrations.

Blue screen errors are most often caused by RAM issues. I remember very well that I was facing blue screen error at my work once, & upon searching I found that all I need to do is remove the RAM, clean and dust it, and put it back into the same or another RAM socket.

This is a HN gem! Wonder how it didn't make it to first page.

I once had a mobile app perform very poorly only for large downloads on a particular SSID. No other applications had such issue. It turned out to be due to a firmware bug which scrapped TCP SACK field, but the problem manifested only for large data transfers over unreliable, high latency connections. Got us all chasing our tails for a while though...

The authors are still collecting more stories for their longer journal version of the paper. If your institution has 10+ stories to contribute and are interested to be part of the journal paper, please go ahead contact the first two authors of the original paper.

Reply on news.ycombinator.com