Maybe he was put on guard by the guy mentioning his driver at all? It might be suspect to mention your driver at all if it is in fact a normal occurrence.
More charitably, just as poorer folks talk about the cool beer they had last week or the really great movie they saw, maybe rich folks talk about the brewery they just bought or their race horse's recent win.
Different sums of money allow one different pleasures in life. There's nothing assholish about that.