Not sure if you tried that already, but ctranslate2 can run BART and MarianNMT models quite efficiently, also without GPUs.
On a decent CPU I found the translation to take anywhere between 15-30 seconds depending on the sentence’s length, very unnerving to me as a user.
But it’s definitely worth revisiting that. Thanks!
Personally, I haven’t had much luck with small-ish decoder-only models (i.e., typical LLMs) for translation. Sure, GPT4 etc. work extremely well, but not so much local models capable of running on small form-factor devices. Perhaps I should revisit that.