Yes, delivering JS crypto over the pipes is dangerous. But surely, bundling the code as a browser extension (such that you download and install it once and from then on it's loading everything locally) mitigates most of the issues apart from the memory clearing and perhaps timing?
This is how Cryptocat (https://crypto.cat/) does it.