Your same argument can also be used to degrade the performance for all other groups, so that group X isn't unfairly disadvantaged. Or, it can even be used to argue that the performance for other groups should be degraded to be even worse than group X, to compensate for other factors that disadvantage group X.
This is argumentum ad absurdum, but it goes to show that the issue isn't as black and white as you seem to think it is.