Very similar situation here and I absolutely agree with you. Remote has been tough. My ideal set-up is fully in office with a walkable commute but I don't think it's happening quite yet.
This is the thing for me. If your company has a fully-stocked kitchen, quality chairs, monitors, adjustable desk, and noise cancelling headphones, it might be a comparable to a good home office setup in terms of comfort. What kills me though is commuting. All told, I spend 3-4 extra hours on days I go to the office, and that time comes out of my free time, effectively turning a 40 hour job into a 60 hour job.