I have done similar as well: https://github.com/seanwevans/pg_gpt2