I'm quite nearby to Scott in social space, and no, I don't think it's just "never come up." It's a very highly selected scene.
> I live in a Republican congressional district in a state with a Republican governor. The conservatives are definitely out there. They drive on the same roads as I do, live in the same neighborhoods. But they might as well be made of dark matter. I never meet them.
I agree with OP that it's much more likely that he did meet them, just in contexts where it doesn't come up.
As typical, during that time he was probably so overworked he didn't really interact with any of the locals until his time was up and he moved back out to a self-selected community.
I don't think I know any Flat Earthers, but if I actually do, they probably deliberately keep it to themselves because they know that they'd look like kooks by outing themselves. To get more extreme, maybe some of my friends are Nazis, but they sure aren't showing me their memorabilia, not because "it just doesn't come up" but because they know it's wrong and don't want to ostracize themselves.
Maybe eventually people work up the courage to probe a little, they'll use code words and innuendo, and soft dogwhistles, maintaining plausible deniability until they identify their fellow outgroup members. I think I see this behavior in a few of my friends, but they wisely don't come out with it.