It's crazy, if you aren't used to thinking about it, how much your software's behavior can affect power usage. Have a sloppy inner loop where you spin on a lock, destroy the battery. Leave open HTTP connections that aren't really doing anything but are waiting on data, destroy the battery. Set the accelerometer update frequency hire than you absolutely need to, there goes the battery. Fire up a GL view that turns on the GPU, god forbid you try to run synchronously and spin on framebuffer renders, goodbye battery.
I have a theory, and it's...essentially totally baseless, hand-waving speculation, but I'll share it anyway, that that's really the difference you see with a lot of these new phones that come out that have terrible battery life. It's not necessarily that running the slightly larger screen automatically decimates the battery, say, its that the carriers don't have the discipline to be brutal with regard to the software's power consumption.
With iOS, this has historically been mitigated to some degree by the fact that 3rd party apps weren't allowed to run in the background, and that Apple was fairly psychotic about managing power use in its first party software. That is changing, but I suspect a lot of what has driven the API design there is sensitivity to power consumption.
It's actually wild, if you have an iPhone, say, to turn off the Wifi, 3G, and BT radios, and location services. Use it like a dumb phone. The battery will easily last for a week, probably a lot longer if its a new battery, and all you do is send texts and make the occasional call. The OS and phone radio can idle for a long time on a single charge, with occasional powerings of the screen. We think of phones as just being intrinsic power hogs due to the screen and phone functionality, but the power efficiency of those has risen dramatically over the last several years. It's all the other shit now, which is the shit that makes the phone interesting, that consumes all the power.
So yes, a dramatically more power-efficient CPU would make a huge difference. However, I am personally skeptical that having a separate, off-die CPU that's lower power consumption will lead to dramatically better power performance, but we'll see.
What would be amazing is if the OS on devices with these smaller cores would provide a mechanism for apps to request the kind of CPU they want. For example, services or background tasks could always run on the smaller core, even if they are fully utilizing the core, without getting pushed to the bigger core. This could help Android, for example, make sure that background tasks don't eat up battery life while the phone appears to be sleeping or doing other things.