Dennett is another strong opponent here. He says Mary would retort, aha but you have a limited imagination. I know everything about color, so nothing about that experience was new to me in any non trivial way whatsoever. Because Mary knows EVERYTHING about color. It's may be impossible for us to imagine someone being omniscient about color, and that spoils the whole argument.
Similarly, what if we knew every physical fact about bats. Dennett only needs to say, we have small imaginations. We have trouble imagining that what it is like to be a bat would be the sum of all physical facts about a bat. But we can't be sure of that. There is no logical certainty. Hence Nagel proved nothing.
Yes you end up saying "we're all robots/zombies/unconscious animals" this way, and there's actually nothing special about being human it's just a myth we tell ourselves. But aside from some people not finding that conclusion tasteful, there's nothing wrong with it.