I think this was due to keeping all loaded messages in the DOM.

I've made it so that only the most recent messages are kept if you're scrolled to the bottom, so it should be better now.