That's an interesting idea. I have not tried to build something similar but a humble word of caution that I want to put out is that no matter what kind of ML you use the mechanical version of the instrument will always be more precise (you likely are aware of it, just want to make sure). However, you might be able to approximate precision of the mechanical version.

Two hand support would be nice and I would love to add it in the future.

The engine should work well with different skin tones as the training data was collected from a set of many and diverse individuals. The training data will also grow further over time making it more and more robust.