I've spent some time with whisper, and indeed this happens all the time. To my untrained eye it seems like:

- they indeed seem to have trained on movies/subtitles

- you absolutely positively must use Voice Activity Detection (VAD) in front of whisper