I've implemented all points that @tptacek mentioned (except the web extension, that will be a future update), and will also activate HSTS when I feel comfortable with the server setup.
I switched from CryptoJS to TweetNaCl which seemed fit to purpose, and also moved all CDN depencencies locally as well as removed Google Analytics :)