Cats have personalities.
I'm sure you know this, but maybe have not considered it in this context.
There have been humans in my life I did not like, or who drove me bonkers. Perfectly good people, but just a personality conflict. Who's to say, that it isn't the same for some cats? Just a personality conflict, and so they seek life elsewhere.
And only a 100 years ago, most cats were farms cats, as most people lived in rural areas. Many cats were barn cats, familiar with humans, but mostly wild, living off of mice and other vermin.
Those genes don't vanish in 100 years. Maybe some of those older cats just preferred independence?