The way I remember is that the Romans associated the left with bad things, with this ultimately making its way into English in the word "sinister" [1]

As this association was extended to left-handed people, its use nominally perpetuates an ancient slur, but I think we can put that aside.

[1] https://english.stackexchange.com/questions/39092/how-did-si...