The greater issue is the passwords of other accounts, which could now be 'recovered' as the attacker has your access to your email