It happens the same way if you train a small transformer. It learns to produce words very quickly.