I lived in Key West for half of my life. I got addicted to swimming daily and would always go, no matter what. Now I have sinusitis caused cluster headaches. When I trace back events, the first one happened about a year into the swimming. Now they are truly debilitating.
I would even add not swimming during an out-going tide if the water passes by nasty land. (nasty = like seemingly lovely mangroves where birds live or bird poo covered rocks)
Now I always check water quality reports to find the cleanest water on vacation as well. Most places do them, even if not advertised.