Hacker News Reader

Just tried the SDRAM tweak and got myself a respectable 10% speedup with LLMs. Hat tip.

What are you doing with local llms on a pi5, genuine question btw.

Not the author, but on a Pi 5 small LLMs such as tinyllama or qwen2.5:0.5b run at over 25 tokens per second (haven't tested the performance boost with it yet). It could be useful if you wanted a local assistant or an AI server for devices you don't want to run LLMs on

Just curious, what can a 0.5B model be used for?

I use mine mostly as a context-free code copilot. It's not perfect, but it knows how to write a template and for most tasks that's all I need.

Are there any vision models that will work on the Pi 5? Something similar to minicpm-v?

I don't believe you can get any useable results. Perhaps only for R&D.

Is there some kind of list that shows comparable performance over all of these changes? There's that ARM 4_0_X_X quant, NUMA patch, now this SDRAM tweak. Plus if there's any delta with your llamafile speedups vs. openblas. I feel like we should be approaching over 3 tok/s tg for 7/8B models at 4 bits by now?

Last I tested, the ARM quants were actually slower on a Pi 5 than Q4KS, but I later learned I messed up some compile flags and haven't had the chance to retest lol.

Reply on news.ycombinator.com