What if you put the smart TV in game mode? Doesn't that turn off all the features/post processing?
I've got newer Vizio's with Chromecasts connected to them... no issues with them being disconnected.
Even if you do keep it disconnected, there's no guarantee that a badly behaved update process or app won't interfere with the normal operation in some way.
I'm sure there do exist models that are perfectly fine in offline mode, but it's still a lot of (very opaque) room for things to go wrong in ways that a "dumb" display won't have.
But, even then it can be annoying. If I happen to press on the smart key on my remote, by mistake, the TV then takes ages to check that indeed there is no connection, with a large and unescapable message dialog.
Bad design decisions all around.