It might be. But then again, at least in the U.S., many of the languages we consider 'ugly' sounding are not spoken by people that we (presently) exhibit a lot of negative biases towards. (Obviously I'm thinking of German here.)
It seems perfectly plausible to me that, just as there are unpleasant non-linguistic sounds, there could be languages that produce a greater number of unpleasant sounds in typical speech, to a person from any cultural background. Think: nails on a chalkboard. (I'm tempted to call these "objectively ugly" sounds, but of course having a social cause does not make something subjective.)
Of course, this is not to say that racism, or xenophobia are never the causes of these judgments. And, more subtly, some of our views about what sounds are 'ugly' may also be socially constructed, at least in part. And these views could relate to longstanding racial biases, among other factors.
tl;dr: Probably some of these judgments are facially racist, and others are caused by genuine assessments of the ugliness of sounds, but those judgments themselves may have social causes that include racist and xenophobic dynamics. But others still may well relate to perceptions that are hardwired, and not socially constructed.
All this complexity makes me hesitate before signing on to your view, though I do sometimes lean in that direction.