Thank you for the comments. Getting users to play head-to-head has worked well for other titles (Letterpress comes to mind) however there isn't much variability on this game in its current incarnation so I believe I need to improve on the concept first.
I have thought about taking the time into account as you suggest, however once you complete a game with a given score it is very easy to repeat it faster with a screenshot on the side.
I'm also not sure if the concept will prove engaging enough to have a lot of returning users. In any case, I'm not expecting this to be a big hit, and just wanted to get it out there and see how it goes.