With this said (I'm also using them for off-grid) you will need to put them to sleep and only use the display when absolutely needed for most scenarios. I've recently started using devices with e-paper display which at least solve that nuisance of the display power draw: https://www.waveshare.com/wiki/ESP32-S3-ePaper-1.54
The last thing to keep in mind is heating. They will warm quite a bit and you should consider a way to either keep them cooled or make them sleep enough to cooldown, otherwise they will reboot or stop working until they are cooled again.