Hacker News Reader

Comment by bbor | Hacker News Reader

Because it might not be clear:

  … d)Fully open-sourced a family of 7B parameter models capable of processing long text documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M tokens.

https://huggingface.co/LargeWorldModel

In terms of content, I am blown away yet again by the SoTA speeding on by as I try to catch up. Can someone with a more cynical eye point me to competitors or problems with this approach? Because as it stands… that jump to a context length of a million tokens is pretty impressive to an outsider.

View on HN

Look at how RingAttention is implemented: it's blockwise attention distributed among many GPUs, in other words bruteforce parallelization. For inference they use TPUs v4-128, not running this at home any time soon.

What is this month's best choice to run at home?

ollama

Depends on what you're looking for. Check reddit.com/r/localllama, but I'm guessing it's still Mixtral for most things, with Yi-34 fine tunes being useful in some cases since Mixtral fine tuning is still not there yet.

If you pull the llama.cpp repo and use their convert/quantize tools on the pytorch version of the models uploaded to huggingface, they will load just fine into ollama:

https://old.reddit.com/r/LocalLLaMA/comments/18av9aw/quick_s...

https://github.com/ggerganov/llama.cpp/discussions/2948

You can run ollama (and a web UI) pretty trivially via docker:

docker run -d --gpus=all -v /some/dir/for/ollama/data:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:latest

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway --name ollama-webui ghcr.io/ollama-webui/ollama-webui:main

That particular webui will let you upload models (with configuration). Other wise, you can use the api directly (you'll need to POST a `blob` first):

https://github.com/ollama/ollama/blob/main/docs/api.md#creat...

Is there any information on a suggested inference setup? I guess they had something different in mind than TPU v4-128 when they put it on HuggingFace?

looking at https://github.com/LargeWorldModel/LWM - they seem to indeed suggest to use a TPU vm

I suppose you could try with a Google Colab notebook attached to a free TPU instance? Probably would be quite limited if it worked at all.

Its llama 7B, so anything that runs that.

You can quantize the cache and fit quite a bit on GPUs. At least 75k on my mere 24GB 3090, maybe 200K with a fancy quantization repo.

Its more of a tech demo since its llama 7B (a model that is, TBH, obsoleted by mistral 7B), and its dataset is not that great.

We've had Yi 6B 200K context for some time, which is also quite good.

The problem, of course, is hardware requirements and vram. This one is particularly hairy since its not a GQA model.

Gotcha thanks, great stuff to look into. I’m still in fairly-tale symbolic AI land these days, where hardware requirements are a distraction to be abstracted away…

Its not that cost prohibitive! You can run GPT 3.5-replacing AIs (aka 34Bs) on a $1500 PC... If you can find a used 3090 now.

Reply on news.ycombinator.com