I never need to restart emacs. It's generally only when I reboot that I have to start emacs again.