Funny, this mirrors almost exactly a decision I made after about a year of struggling with ELPA packages breaking on me repeatedly.
I ended up cutting Emacs off from ELPA entirely, settled on a ~700-line init.el, and now use Emacs as a glorified Org-mode agenda keeper. It's been heavenly (especially with a dedicated monitor).
The one thing I'm still working out is syncing with calendars and email.