That's how we did this; https://m.youtube.com/watch?v=7jfZ6zLkWF0 (was running live on-set)
What company do you work for? This is super cool.
Can you talk about the rest of your stack? Or other projects of a similar nature?
The setup was essentially an app (c++ engine on osx running models), ingesting camera feeds, outputting skeletons, running sound->mouth poses, to unity which puppeteered 2D sprites.
On-set we output to a monitor which mixed the real shots with our graphics.
Nothing crazy! But it was a solid setup (except for overheating DSLRs)