Was it trained on code? On my quick glance at the training data it seemed to be a clean-up version of common crawl.