Call him on his bullshit! Don't be one of the people who lets him skate by because he's the boss. At the end of the day he's just a human being; flaws and all.

If I were you I'd take him out for a beer and tell him everything you wrote here. You'll probably be the first person who was ever honest with him about this.

If he has a modicum of character, he'll respect you for it and honor your concerns. If he doesn't care about the issues you bring up then that's beyond your control. In that case you'll have done what you could and I think you can move on to something else with a clear conscience.