The end user can't know when that's the case.
> Also you might serve files that do the encryption from different server that's smaller, better protected, more stable, that less people have access to.
That doesn't provide any assurance to the end user that the JS isn't malicious.
Remember, "compromise" doesn't just refer to a drive-by hack. The site operators themselves may become compromised (or start that way), and deliberately serve malicious JS. Users can't know when that's the case. When it is the case, the strategy you suggested offers no protection, because the "more secure" JS server is still under the control of the bad actor.