Let's use an example of something I've done in Photoshop. Here's (roughly) how the image would be tagged: Bridal party, inside a mansion, stairs behind them. Now I try to get rid of table that's to the side that's in front of a window. What do you think will happen with those tags? It's surely not going to correctly put half of a window and then half of a wall, like it should be. I might get some bridal party, some stairs, etc.

If you just tag the area I'm trying to get rid of it's going to be something like "wood table, window, photograph." Same issue.