Pulling off this level of scale with Elasticsearch is no easy feat and very impressive from a technical perspective. When you're running ES with petabytes of mission critical data as a core service powering the universe of a business, cluster rebuilds aren't an option (or maybe they are, as a last resort, but absolutely will not be acceptable on an ongoing basis).
Relying on Elasticsearch mega-clusters in this manner is akin to running an ultra-marathon with a really sharp pair of scissors glued in each hand. Or maybe even more extreme than my (admittedly lame) analogy.
Running nodes with such high shard counts is an appreciably precarious proposition, because there is a fair amount of overhead in the Elasticsearch management protocol. I wonder what the performance testing strategy entailed.
I have a lot of respect for the engineers working to make this project and service a success story. When it comes to Elasticsearch at scale, such outcomes are the exception.