When the world's most brilliant mathematician (or at least one of the most brilliant) says "it would be rather bizzarre if your argument worked" the implications are absolutely clear.

As a scientist I would never make such a statement, moreso for a theory I admittedly don't understand.

Tangentially, it's not Tao's case, but I have seen a lot of bullying in the academia along these lines... "I am not saying it's wrong, just very bizzarre", "I am not saying it's wrong, but my Ph.D. student worked on it for two years and couldn't solve it", "I am not saying it's wrong, actually I don't even understand the details, but please recheck everything..." etc...