I have a modest proposal that might also work.

Four canvases:

1. BW (inited from Color) 2. Color (color source image) 3. Paint (inited transparent) 4. Temp (inited transparent)

Each mouse move:

1. Draw an opaque white line on Paint to the new position 2. Copy Paint to Temp 3. Set the globalCompositeOperation to "source-in" 4. Draw Color to Temp 5. Set the globalCompositeOperation to "source-over" 6. Draw Temp to BW

I haven't tried it, but I think it should work (or something like it with the flaws corrected :) ). One advantage is that you can use the path line drawing primitives on the canvas instead of rolling your own.

You could also optimize a bit by only drawing the portions of the canvases that have been changed, but it really should be plenty fast as-is.