Let's say we have two teams, the Ayyys and the Bees; each has two defenders, Schlub and Superman play for the Ayys, and Scrub and Slacker play for the Bees. Let's assume that they all suck equally, except for Superman, who is great.
Now we run your statistic: we compare how the Ayyys do with Schlub on/off against how Scrub does on/off.
The problem is, when Schlub is off, Superman is playing, whereas when Scrub is off, Slacker is playing. This means the Ayyys do quite well when Schlub is off, but the Bees do about as well when Scrub is on compared to when Scrub is off. By this comparison, Schlub is a much worse player than Scrub, even though in actual fact they might be equivalent.