tptacek is right. In order to trust your crypto we have to trust that your server isn't compromising your crypto via malicious JavaScript, so when you claim that we don't have to trust the server, you're asking us to both trust and not trust the server.

You could do a Chrome extension with all the frontend code baked into the extension's static JavaScript. Firefox still has no sandboxing for add-ons, so you wouldn't have luck there.

Desktop / installed apps would be the way to go. See https://turtl.it for an example of this (shameless plug)