The missing headphone jack turned out be basically a non-issue, even though that kept me back from upgrading - the converter stays on the headphone jack at all times (I think the force to pull it out is higher than standard headphones jacks as well). It's very rare that I'm in a situation where I want to charge the phone while being connected to a headphone jack. Not sure I've experienced it yet.
It sounds like something like the iPhone 12 mini would work for you if you gave it a shot. It's a bit larger (5.4") than something like the iPhone 4 (5") but it doesn't feel like a huge phone. I'm intending to replace my iPhone XS with a device of that size when its time for me to get a new phone.