Wow, amazing!
What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.
What if there is enough RAM to fully load the model? I assume in that case I shouldn’t use your engine.
I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.
The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.
If you need maximum speed and flexibility just use MLX
For CLI and Server, use --max-context