Because https://en.wikipedia.org/wiki/Didier_Raoult#Controversies has him being censured for falsifying diagrams, being credibly accused of "ethical, procedural, and methodological problems" in his HCQ trial, and conducting illegal clinical trials on people who could not legally consent.
I mean, sure, he might be credible but his track record suggests otherwise.
He's basically a former good scientist with excellent political skills which helped him climb to the top and has been slowly been corrupted by his own hubris.
The fact is that the original claims about HCQ weren't coming from politicians, they were coming from doctors. You might not like the doctors, those doctors might have heterodox ideas, but that is okay.
It had been part of the scientific conversation for over a month before his involvement (an eternity in 2020 time), eg https://www.nature.com/articles/s41422-020-0282-0