The wealthy countries that chose never to join like Switzerland and Norway have their own valid reasons to not join (fear of regulation on banking, fishing and oil that's not in their favor) and still thrive due to their wealth. But leaving the EU doesn't automatically turn your country into Switzerland or Norway like the UK expected.
People often do miss the forest from the trees and that stuff like quality democracy and unions like the EU are never a "fire and forget" solution but a rolling grinding game that must constantly be kept in checks and bounds witch citizen involvement and outcry, over generations, otherwise it slowly gets eroded.