Hacker News Reader

I used devstral today with cline and open hands. Worked great in both.

About 1 minute initial prompt processing time on an m4 max

Using LM studio because the ollama api breaks if you set the context to 128k.

How is it great that it takes 1 minute for initial prompt processing?

That time is just for the very first prompt. It is basically the startup time for the model. Once it is loaded, it is much much faster in responding to your queries. Depending on your hardware of course.

Haha great as in surprisingly good at some simple things that nothing has been able to do locally for me.

The 1 minute first token sucks and has me dreaming for the day of 3-4x the bandwidth

Have you tried using mlx or Simon Wilson’s llm?

https://llm.datasette.io/en/stable/

https://simonwillison.net/tags/llm/

On lm studio I was using mlx

Reply on news.ycombinator.com