Could it be, that one source of the memory usage is the built-in language server?