Running any server on wifi, even home plex instance, is not a great idea. It's a shame that rPi doesn't support PoE out of the box, that'd mean you could've only 1 cable connected, but it'd manage both network and power.
But the idea that it's WiFi that's unreliable is clearly not the case.
Something is wrong with the RPI WiFi chip, driver or OS configuration.
I had some network lag/dropped frames, turns out my stereo receiver was downloading a quite large firmware update. Had me freaked out a bit, I hadn't recorded the mac address. Didn't figure it out till I turned the TV on and saw a dialog asking if it should upgrade.
So, yes, generally I'd recommend that anything that's going to consume hours of 4k video be put on ethernet if at all possible.
Things like SmartTV, RPi, Solar Panels, Amplifier, etc. I hate it that Chromecast only has a wireless option.
But more on-topic, it shouldn't drop off WiFi just like that.
They're a little pricier, but the Chromecast "Ultra" supports ethernet out of the box. For the base model, you can get an adapter.
From what I've seen about Smart TV's, I'm better off not allowing them to ever connect to the Internet. Forgo all the "smart" features and just use a Fire TV stick, Roku, Chromecast, etc.
My current TV is an 8 year old 46-inch "dumb" TV, but I plan on upgrading to something bigger later this year, and it looks like my only options will have Smart features, but I already use a Fire TV stick and a RPi, so I don't think I need the smart features.