This problem was more or less solved with neural networks in 2013, see https://arxiv.org/abs/1308.0850. I've implemented a similar model in javascript that you can interact with: https://seanvasquez.com/handwriting-generation/