For every von Neumann you probably have thousands if not millions of crack pots with ideas not worth wasting time on. So I understand the scepticism
Otherwise why not just replace them with a script (or these days, an AI)
And I don’t know many, if any, examples of people received as crackpots that later were considered genius. At worst genius takes a little time to be recognized, which is a situation far more common.
I forget the quote, but one of the guys that recognized that he was worth looking into said something to the effect of "his work was too crazy for him to be crazy".
It's like an analog to the fact that almost all people consider themselves funny.