Hacker News Reader

It's intended for SQL generation and similar with cheap fine tuning and inference, not answering general knowledge questions. Their blog post is pretty clear about that. If you just want a chatbot this isn't the model for you. If you want to let non-SQL trained people ask questions of your data, it might be really useful.

View on HN

It's worse at SQL generation than llama3 according to their own post.

https://www.snowflake.com/blog/arctic-open-efficient-foundat...

To be fair, that's comparing their 17B model with the 70B Llama 3 model.

To stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both

Sorry, it sounds like you know a lot more than I do about this, and I'd appreciate it if you'd connect the dots. Is your comment a dig at either Snowflake or Llama? Where are you finding the unquantized size of Llama 3 70B? Isn't it extremely rare to do inference with large unquantized models?

to stay fairer, the required extra disk space for snowflake-arctic is cheaper then the required extra ram memory for llama3

For decent performance, you need to keep all the parameters on memory for both. Well, with a raid-0 of two PCIe 5 SSDs (or 4 PCIe 4) you might get 1 t/s loading experts from disk on snowflake-artic... but that is slooow.

Reply on news.ycombinator.com