A library to do this that integrated well with typical inference and and training systems would be a great contribution!