I was not talking about any of the points you mention, I was refering to the geographical facts alone. Wouldn't you agree that in general it is kind of silly to claim ownership of a piece of land that is far, far away from your own country?
The distance from London to Gibraltar is closer than the distance from London to Bermuda, but nobody finds that weird. France has French Polynesia on the opposite side of the world. Russia has Kaliningrad. Norway has Svalbard. South Africa has another country, Lesotho, right in the middle of it. India wraps around Bangladesh like a tentacle. Azerbaijan has a random piece of land and makes a sandwich out of Armenia. Spain has islands directly west of Morocco. France has land on South America.
The whole world has freaky borders. The only clean borders are places like Wyoming and Colorado.
So no, as a general rule I can't agree. I can certainly agree that there are some rather silly cases, but it's just not practical for all islands to be self-governing, so I can't agree with the general rule you propose.
Colonies are rarely ceded voluntarily, but are usually the result of an invasion or a lost conflict. The fact that this is no longer the case with Gibraltar today, because its inhabitants value the benefits of belonging to Britain, is simply a consequence of the time that has passed since then. It does not automatically undo the imperialist actions of the past.