It's fantastic that more orgs are releasing open-source models trained on more than 300B or so tokens. Here's my take from the details I could find.

Pros

  - 4096 context width (vs 2048 for llama, gpt-j, etc)
  - 3B to 65B released or in progress
  - RL tuned models available
  - Trained on more tokens than existing non-llama models
  - 128 head dim, so can use flash attention (unlike GPT-J)

Cons

  - No benchmarks released, or details about the model
  - Somewhat restrictive license on the base models, and NC license on the RL models
  - Small models only trained on 800B tokens, compared to 1T for llama-7B, and potentially more for other upcoming alternatives (RedPajama, etc).  I'd like to see their loss curves to see why they chose 800B.

High-level, this is likely to be more accurate than existing non-llama open source models. It's hard to say without benchmarks (but benchmarks have been gamed by training on benchmark data, so really it's just hard to say).

Some upcoming models in the next few weeks may be more accurate than this, and have less restrictive licenses. But this is a really good option nonetheless.