Having not used a smart phone for ~5 years now, I can say that it is totally a workable solution - however, there are, of course trade-offs, and several persistent annoyances. Rather than get into a list, I will present the one thing that I find the most frustrating:

Our society expects you to have and use a smart phone for nearly everything.

This might be less annoying if it meant only new functions or supplemented old things, but in my experience it's the opposite. It's often the most simple low-fi things we have been doing for ages, that have been overhauled to require a smartphone, app, and data plan. It's hard to be looked at as 'the person complicating things', when everybody else expects you to carry a luxury item (and an expensive monthly service bill), just to perform a basic task, like paying a bill, or reading a menu.

I'm not one to wander around expecting society to cater to my own needs, but I openly admit that I find this Twilight Zone-like first-world problem (What do you mean, you can't use our app?) the most infuriating.

With that said, you can still get by in most situations, but I don't see the above getting better on into the future, unless there is some cultural revolution about how view and use technology.