I was in this very situation, thinking I want more than "just" EE. Applied for a PhD position in a branch where mostly mathematicians work, could not be happier with the decision.

Do it. Scratch that itch while you still can.