If you read Barnum's works, he has a much less condescending view of his customers than that quote would suggest, which is probably why those who knew him said it was out of character.