why have kitchens traditionally been separated from dining areas in the first place?
Not long ago, cooking was often done exclusively by “the help”.
Not long ago, cooking was often done exclusively by “the help”.
This wasn't really upper class but it was influenced by British norms, albeit in the US, even if there weren't servants involved.
So this doesn't quite answer why they settled on keeping it hidden instead of a display of entertainment. There is no reason to think people in that time wouldn't have found the same enjoyment in watching the food get made.
Most likely the answer is that it's just what one notable person happened to decide, by random chance, to do with his kitchen arrangement, and everyone else copied it in hopes that it would make them equally notable. Pretty much all human behaviour can be explained that way.
And even the ones that do have a theatre of sorts have background preparation and dish-washing that nobody wants to see.
Oh no. Chances are the real engine of this change is the recent mass-media discovery of food-prep as a heroic and cool activity.
Celebrity chefs used to be obnoxious, "always-be-prepared", by-the-book home-goddess archetypes like Delia Smith; at some point over the last 25 years, they turned into sexy, improvising, creative rockstars. The creativity-impoverished masses, bored to death by day after day looking at computer screens, now get their kicks by eating and cooking.
But it could have gone the other way. The first notable person could have recognized the entertainment, and the later notable person could have decided it was better to keep hidden. We don't know why the first one decided to go with hidden, and likely it was just a random choice.