Hacker News Reader

Comment by loudmax | Hacker News Reader

Apparently there are "200k" versions of the 6B and 34B models. I don't see an explanation in the Model card for what the "200k" means here. If that's context size, that is impressive.

TheBloke has GPTQ, GGUF, and AWQ versions of the Yi models. The 6B GGUF instance should run decently on systems without a high-end GPU.

If Yi performs as well in practice as it does on standardized tests, looking forward to releases "Er" and "San" ;)

View on HN

Joke explainer: "Yi," "Er," and "San" are (the partial transcriptions in the Latin alphabet of the most common words for) "One", "Two," and "Three" in Chinese.

I’m assuming that 'Yi' was used because it unifies several ideas. I’m also happy it’s an easy naming convention that is not an animal.

The 200k means the number of tokens in the context window. That means it can operate on very large sets of data at the same time. To convert from tokens to words, it's about 150,000 words (modulo grammar and other facts of life).

Important to note though, that even if they say it has 200k context, there's a good chance it won't be able to use all of it.

For example, here is an experiment on long context recall for Claude and GPT4:

https://twitter.com/GregKamradt/status/1727018183608193393

https://twitter.com/GregKamradt/status/1722386725635580292

Even though claude claims 200k tokens context, perfect memory stops at about 19k tokens.

I'd personally expect the Yi-34b models to perform even worse at the task, though I have not tested it.

Yeah we tried Claude 2 to summarize multiple conversations and if we give it the instructions at the start and then show it the conversations it almost always forgot the original instruction and ends up outputting something related to the conversation so long context whole nice seem to break down very quickly. What I found worked better was to provide the instructions at the end but given the in ability to follow the starting instructions my guess is much of the early conversations are lost… so yeah not so sure big context is a silver bullet

Yeah, I'm planning on testing it as soon as I figure out how to properly prompt engineer an untuned foundation model. GPT-4 Turbo 128k also seems to lose coherence around 20k tokens based on my testing so far.

have you tested?

Reply on news.ycombinator.com