That may all be true, but you’re talking about voluntary affiliations here.

Colonies are rarely ceded voluntarily, but are usually the result of an invasion or a lost conflict. The fact that this is no longer the case with Gibraltar today, because its inhabitants value the benefits of belonging to Britain, is simply a consequence of the time that has passed since then. It does not automatically undo the imperialist actions of the past.