Do you want only experts in the field on your board? It looks like they had Elon on the board to inform what to research. His company has a bunch of data and motivation for AI research in general. Not many experts have domain knowledge that he has.
That is a separate issue. His comments made the incredibly talented researchers at OpenAI look really bad. I suspect this is one of the prime reasons he is leaving.
Have some examples of how he made them look really bad? Not disputing, just curious.