I should clarify that I mostly use the Remote app and largely find it works well, particularly for searching.
I find the wifi connection between my iPhone and my Apple TV (via an Airport Extreme) is a bit laggy and sometimes drops. I think it would be a much smoother experience if it paired via Bluetooth 4.0. I'm hoping the next Apple TV includes Bluetooth.
The Remote app is much more usable than the Apple TV remote, but the remote has a lot less lag and is easier to move around the interface with. This hasn't stopped me from using my Apple TV several days a week with just the Remote app, but I think that a Bluetooth connection would be much better.