Love it OP!
Check out https://github.com/yl4579/StyleTTS2
For your local voice cloning needs.
HN discussion here https://news.ycombinator.com/item?id=38335255
Funny I’ve been talking about cloning my dad for a while in this same fashion. Thanks for the inspiration.