In my experience Youtube and Netflix are the only apps that really got the basics of video streaming right.
I’ve gone through this process 1000 times over the last year. It’s become muscle memory on my remote. YouTube still hasn’t fixed it.
one of the worst apps on apple tv. it got to the point i don’t even try to like videos. or rewind them. or do anything while they are playing lest i cause a restart. buffering. a random video to start playing, etc.