Ask HN: How to recognise parametric drawings?

Hey, something I've always wondered, how does an application like Ken Perlin's chalktalk do its gesture recognition?

You can see what I mean here: https://vimeo.com/127447653

I somewhat understand how to classify a crude square drawing as a "square" gesture. What I don't understand, is how do you classify a small crude square drawing as a "square(5)", a larger one as "square(50)" and all the other square-sizes in between.

Do you know how something like this could be done?