Their main advantage for now is their super clean API. Open source alternative are already on par with GPT-3.5 and 4 capabilities, they just don't have as good a package but that could change rather quickly too.
I'm not sure if this is true. With GPT-4, I can successfully ask questions in Japanese and receive responses in (mostly natural) Japanese. I have also found GPT-4 capable of understanding the semantics of prompts with Japanese and English phrases interleaved.
Out of curiosity, I tried doing the same with local models like Mistral 7B and I could never get the model to emit anything other than English. Maybe it's a difference in training data, but even then, GPT-4 has an allegedly small set of training data for non-European languages.
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
https://paperswithcode.com/sota/sentence-completion-on-hella...
https://paperswithcode.com/sota/common-sense-reasoning-on-wi...
https://paperswithcode.com/sota/common-sense-reasoning-on-ar...
https://paperswithcode.com/sota/common-sense-reasoning-on-co...