You've ruled-out the known spotty LTE switching/searching issue, I'd perform these additional experiments:
1) Disable iCloud sync entirely for a day. In iOS5 there used to be a bunch of battery life problems caused by devices getting caught in an endless loop of pointless background syncing. Maybe you're still hitting something like that.
2) Turn of location services for a day. Maybe you've got something like Find My Friends taking up too much juice?
3) Turn of email "push" for a day. Maybe you've got too many email accounts?
Failing that, maybe you've got a dud? Take it to the Apple store and they can look at your device energy usage logs. They'll most likely tell you to do a fresh iOS install suspecting a software issue.. but in the end they might just switch you to a new device.
Where are you physically located? I wonder if bad coverage could be affecting your experience.
It's just the cold hard facts of where the state of cellular radio technology is today. It's pretty amazing as is that the iPhone 5 can be so much thinner, lighter, support a larger hi-res screen, be twice as fast, and support LTE. The spotty LTE issue is just one of those compromises that I don't think much can be done about.
I had the same issue. My phone was also noticeably warmer. It was doing something. Now everything works great. Battery life even with LTE is quite good.