The current solution is just bad bad bad. :(
Awesome project otherwise. :)
I would have suggested making them respond linearly, but to both x and y motion. That is, the effective position change is the maximum of the mouse x and y axis displacements from the click position.
My view is that rotary motion is not useful in an on-screen GUI (unless you have a MS Surface Dial perhaps?); rotary-looking controls only exist to save space and provide a friendly appearance. The way they actually behave should be whatever is most efficient to drive, regardless of their shape.
This doesn't mean that you end up with having to carefully dial the knob, but it does let you click and drag the mouse in such a way that the rotary dial at least rotates in the direction you'd expect. The current behavior sometimes turns CW, sometimes CCW, seemingly at random--if we decide to skeuomorph, we might as well skeumorph correctly.