Sounds like ghost windows. Try disabling all your add-ons to see if the problem persists? You can also record a profile via perfht.ml and send it to me, I'm happy to have a look.
I don't know if it's an architectural limitation of how e10s was implemented, but Firefox still feels like a single-process browser. On the other hand, Chrome remains responsive even after a week of being open.