For myself, I go caseless because, 1) I like the feel of a solid metal device in my hand over plastic or rubber, and 2) it's thinner without a case, and easier to handle.

I have a leather wallet case that the phone slides into easily for when its in my pocket, as I wouldn't want it to get damaged being jostled around, but it's quick to take out when I want to use it.