You have the relationship backwards: being white meant no barriers. He never showed up for a meeting to be pointed at the back door or asked whether he was there to apply for the job as janitor or security guard. Nobody assumed he was there to take notes. He didn’t face constant questions about whether he could actually do something or was just “the diversity hire”.
That’s huge, especially when you look at the stats for things like loan or VC approval rates. Starting a company as a white man isn’t easy but it’s still easier than what most other people have to do.