And during that time so many nodes had became slow and unresponsive that another (for us) previously unseen memory leak started to occur.
Nodes kept building up queues of unanswered ping requests on them. And the requests contained our 100Mb large cluster state, so the heaps filled up and evenmore nodes became unresponsive.
And from then on the whole thing turned into a death spiral of doom.
After trying, and failing to get it under control for 48 hours we gave up and rebuilt the whole cluster from scratch, using the snapshots we store on S3.
The recovery took another 90 hours or so. That was not a fun week.
One of the area managers - Kaveh, IIRC - also had a double morale on line with Trump. He was very "don't put your pen in the company ink", and proceeded to get one of his subordinates pregnant.
I remember there was an "anti Meltwater blog" at some point, but I can't find it now. I don't remember the URL either, so I can't look it up on archive.org. However, this site[0] seems to contain copy-pasted stuff from it.
As said, my experiences are 10+ years old, and hopefully things have improved.