Fun fact: despite being an apex predator, there are no recorded human fatalities from orcas in the wild.
I think they're smart enough to differentiate the ones (us) that would cause too many problems to eat.
In our area (Discovery Islands), there is no lack of humans swimming, or in kayaks, or diving. The transients are common here, far more than the residents. Of course, so are seals and sea lions.