Watch the demo at the beginning of the game — you don’t want to click on the ball as your pivot point for the swing — use the upper right part of the window. That’s part of the reason I made the demo — to show the best place to start your swings. :)

You have to start dragging the mouse from higher up the window as far as I can tell.