That's what happens when latin disappears from the default baggage of an educated person and you can no longer recognize etymologies: master comes from magister (teacher). The owner of slaves was the dominus (which begat words like domain etc)
While we're at that:
Ma,am -> madam -> ma dame -> mea domina (my mistress)