Hacker News Reader

Vulkan, I have never used ROCm on it but have been debating since the latest big update. How is your prefill? Do you hit over 1K? If it’s 1000K prefill, and 40 TG, I might have to try this over the weekend. Also, can you fit 128K without offload the ngram onto SSD?

View on HN

I have not measured pre-fill, but it's said to be around 1000. It feels very snappy and unlike my experience with running 27B models the performance stays pretty flat even as the context increases. Unfortunately, we don't know how Halogen is doing this because it's closed source, but I think AMD should offer that guy some $$$ because he's done a lot of good work getting more performance out of Strix Halo.

Halogen as in this right? https://github.com/peonist-ai/halogen-flash-server

yes

Thanks! I really like how the author packaged everything into a container. Definitely going to give it a go over the weekend!

Reply on news.ycombinator.com