Big mistake.
First off, I got FLOODED with e-mail bounce-back spam because spammers send e-mail with forged From: headers and I'd get all the errors.
Second, I discovered that nobody is actually selling my e-mail address except for one gaming forum I used years ago. Not even Facebook has sold my e-mail address.
Third, I've run into issues when replying to e-mails. I filed a support ticket with a company once, where the e-mail address I had registered with them was company@mydommain.com. They responded via e-mail, and when I replied to said e-mail, their ticket system rejected it since the From: address was my main address of myname@mydomain.com.
Now that I want to just switch to a single e-mail account with gmail, I find myself needing to try to find every e-mail address I've used @mydomain.com and changing them with the website. Meh...not worth it.