The word xenophobic isn't name calling when it accurately labels an emotional response, namely the fear of strangers. America is xenophobic with respect to Muslims, for example, especially so since Islamist terrorist actions instilled that fear.
In this case, I would say yes PG is xenophobic when it comes to people with strong foreign accents, because he's afraid based on his experience that they'll lower the value of the companies he's investing in.