Not all enzymes are stereospecific though. Some of them can take any configuration of a chemical and convert them to other type of isomer. But these are rare and usually an exception, not the norm. Still, it makes it harder to figure out why chirality is so important in life and not so much in some other parts of life.
Maybe your explanation about handedness makes sense. But then it would fail at explaining ambidextrous people since they are definitely better and yet still so rare. Life is really difficult to figure out.