Using a real remote (Apple remotes don’t count) is so much better. There are some things a smartphone is better for, such as entering in passwords. But for regular usage there’s no comparison.
Besides, if I just wanted to use my smartphone I would simply stick to my chrome cast devices. If I’m gonna be using the phone to control my entertainment center then the chrome cast provides by far the best interface because you have access to the entire app.