Presumably, it is experience in algebra, established reputation in field, and familiarity with communication of math that allow this subset of people to reasonably assess the papers. It is not to say that the theorem is so advanced that only 10 people on earth possess the aptitude to understand it - it is that only 10 people have the foundation on which to understand the theorem due to its specificity.
I think you are underestimating just how intricate and challenging understanding a proof can be. Especially one of this length and by someone of this standing.
And yes, a large part of being able to work in a specialized field like this really is about having the specialized knowledge and vocabulary, rather than general intelligence. If this is anything like most proofs in mathematics, most reasonably intelligent people would be able to understand it given ten years of studying the field. It's not about being smart enough, there's just so much existing knowledge that you need a lot of specialization to be able to really contribute to any given area.