In any vaguely medieval setting, a tower is a conspicuously high-status abode, with both "loner" and "don't mess with" vibes. Whether looking down at us from a window, or up at the stars from the parapet, the Wiz is both physically and metaphorically above us mere peasants.

And, when story-telling, a wizard's tower is a very cheap bit of the narrative. The audience visualizing a mere house might need mention of the size, materials, style, color, etc. - not so the tower.