This breaks down in daylight (brightness shifts due to sun moving/clouds) or whenever anything shifts ever so slightly. Even shadows may make things appear where they shouldn't.
Chroma keying is the stable version of this idea: the single background color of a separately and well lit background removes these issues.