Taking your thought and running with it:
Yes, energy/cycle is the ultimate limit, and imposes bounds on what we can do with computers. There are other factors, including dissipating of waste heat. Both those concepts are joined in Gene Amdahl's observations. He noted that in paralellisation, though you could extend this to any efficiency improvement, your efficiency bound is imposed by the unaddressable task component. For parallelisation, it's the nonparallel component, for energy efficiency, it's the non-CPU power draws. He also noted that everything ultimately comes down to a matter of plumbing (including heat dissipation).
Screen tech has benefitted greatly from the transitions from paper => CRT => LCD (flourescent) => LCD (LED), and increasingly e-ink devices. Applications and devices which can utilise largely-static screens, where only state transitions require energising (as with e-ink and textual interfaces) have far less power draw than any emissive screen tech.
Other peripherals -- input/output, including keyboards, touch devices, antennas, ports, etc., also play a role. Physical keyboards have advantages over screens, though audio input/output might address size constraints (and impose numerous others...). The possibility of passive antennas (see the 1960s Russian "shield" audio bugging device in the US Embassy in Moscow) could offer other options, assuming that a fixed station would have a power budget mobile devices wouldn't.
There's been a lot made of small and adhesive solar PV innovations. I see the real use of that as offering the ability for very small computational devices to power and/or recharge themselves opportunistically when and as ambient light is available. Incorporating both battery storage and PV into designs -- say, an adhesive data-tracking and reporting label which could be attached to shipping packages -- as layers of PV, battery, compute, and antenna, would allow for very cheap monitoring of conditions (location, shock, temperature, humidity) for deliveries, at a cost of pennies, and without extensive engineering considerations.
(The hackability of such devices might ... pose interesting considerations.)
We're approaching a state where, theoretically, any device of more than a few dollars' value could have some integrated computational capacity, whether you like it or not.