The display is an OLED, and its power draw is kinda negligible - so the worst offenders are still CPU, WiFi and GSM. I'm going to add an E-Ink display someday - mostly for experimentation, as hardware upgrades like this will require somebody developing and maintaining the softwaree, but it could very well be a really good addon.

As for "your ideal phone" - something similar might just be my next year's project, though sourcing the parts will be hard, for sure =)