actually I just went into the paper and skimmed it.

Their image tokenizer creates image encodings in a discrete space of dimension 8192; each image consists of 1024 discrete tokens. The 8192 vector size matches their language tokenizer (discrete and also size 8192). Then they just mash those tokens together, which leads to downstream problems in training: because the two token spaces have different entropies, stable training in later steps is difficult, and they have to do a bunch of regularizations to make sure that gradients don't start exploding.

Interesting stuff!