The parent starts by saying ‘a significant part of the establishment’. He is not trying to describe all leavers, just the small part of them that are a significant part of the establishment.

Also many of those same leave politicians want more trade with the former colonies, not less.