Or it could be that a goatee physically influences the sound of the voice in a way that is perceptible to the algorithm. For example, damping transmission through the skin and attenuating reflections off the chin and upper pip.
Extreme example, compare audio at 4m and 21m - https://www.youtube.com/watch?v=6dbQ2OA4SRA
Clearly a different top end.
Did a little tinkering in Audacity and with the beard there's a standard roll off from 3khz to 10khz. Without there's a weird flat spot in the same area (both are averaged over 20 seconds or so)