1) 7B foundational model means this is a base model that has not been fine-tuned on the "prompt: response:" (instruct) structure, and it has 7 billion weights/biases. It's the working size of the model.

2) Currently every model that can run locally was trained with a 2K context size. It's a hard limit on prompt length. There have been recent advances with [A] position interpolation, but those methods explore fine-tuning/loras. This base model was trained with 8k sequences.

3. 1.5T tokens is the size of the total training corpus. Training cost and time increases with training size. [B]

A. https://arxiv.org/abs/2306.15595

B. https://www.semianalysis.com/p/the-ai-brick-wall-a-practical... (Jan 2023)