Additionally this is overall hardly a European phenomenon. People tend to associate mostly with people similar to themselves, especially when it comes to education which correlates to income and jobs.
The education system reinforces this strongly in most countries. In the US this begins as early as starting school based on which school (district) the parents can afford. In Germany you have Students being separated by education level after primary school (4th grade). If you go to university you might almost never interact with people with a different education level, apart from family or the people you buy stuff from.
I'd also wonder just how many of those 48% that don't like the EU, would miss what the EU offers. I'd wager most not just don't know what it offers but don't even really know what exactly is bad about it.