« used the word "colonizer" for Jews living in Germany »

I didn't. Rather, "Germans" (Suebians, Saxons, etc) settled in ("colonized") Eastern Europe, beyond the Empire, typically at the invitation of the local rulers. https://de.wikipedia.org/wiki/Hochmittelalterliche_Ostsiedlu... That's where the contact with Eastern Jews happened, which, according to Koestler, gave rise to Yiddish.

Yes, the Langobards and the Goths settling in Italy and Spain lost their language, but the medieval Germans settling in Eastern Europe a couple hundred years later did not.