I absolutely prefer to use headphones, when they're available. But I prefer not to have one more thing in my pocket getting tangled up with my keys.
Use one of those pouches that come with some earphones.
Those are pretty big to be throwing in your pocket. A lot the time I leave the house with nothing but my jeans to carry stuff. Pocket space is a limited commodity.
I would hope that you could use the Nexus 5 as a speakerphone too.