>Only if it was trained on a corpus of existing translations.

Not Necessary.

Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability (https://arxiv.org/abs/2305.10266)

ablation studies performed on the effects of removing tiers of multilingual corpora (parallel corpora, bilingual (not necessarily parallel), and english only)

Removing Parallel corpora or even just bilingual data in general reduces quality but does not make the model unable to translate. Crucially, the gap or reduction in quality also seems to diminish with scale.

The paper also shows models may not need any multilingual corpora at all. The 8b Model can still translate latin scripts with only english training data, adding evidence for larger models being better equipped to leverage either sparse signals (i.e., language-identification failures during ablation) and/or weak signals (i.e., language similarities from shared scripts).