Well in the animated Disney documentary Sword in the Stone we see Merlin come back from from Bermuda wearing Bermuda shorts and shirt so we can surmise it's always been the garb of wizards ; ).

I actually have some and I like projecting an air of fun since I'm an introvert.