Comment by n8cpdx | Hacker News Reader

Top New Best Ask Show Jobs

Comment by n8cpdx | Hacker News Reader

Does anyone know how much context this supports?

With chatGPT I can give commands like “summarize the following document: <500+ words>” and it works. I’d love to be able to do this locally with my obsidian notes.

The llama models were trained with a context size of 2048. By default llama.cpp limits it to 512, but you can use -c 2048 -n 2048 to get the full context window.

2048 words?

Tokens

2048 "tokens", where one token is roughly equivalent to ¾ of a word

Tokens. Short or common words tend to be one token, while less common words are composed of multiple tokens. For GPT OpenAI gives the rule of thumb that on average you need four tokens to encode three words, and LLaMA should be similar

Well that's for sure bigger than my context size.

Reply on news.ycombinator.com