You’re suggesting that people avoid using port and starboard on a boat because they should be focused on learning how to not fall overboard?
Terms like “port” and “starboard” are particularly important when someone falls overboard. The last thing you want to do is tell someone to grab the man overboard “on the right side” and have them go to the wrong side of the boat in a crucial moment. This is why these things exist. We have been sailing for hundreds of years and are very good at it.
Despite this, your entire argument is a bit ridiculous. It is not zero sum- port and starboard do not take up “brain space” that causes you to forget a life jacket. Boat training will help you ensure the safety of the passengers AND teach you “port” and “starboard”.