Hacker News Reader

Hacker News

Comment by ur-whale | Hacker News Reader

Full thread

ur-whale

However big it may be, it still hallucinates very, very badly.

I just asked it an economics question and asked it to cite its sources.

All the links provided as sources were complete BS.

Color me unimpressed.

View on HN

It's intended for SQL generation and similar with cheap fine tuning and inference, not answering general knowledge questions. Their blog post is pretty clear about that. If you just want a chatbot this isn't the model for you. If you want to let non-SQL trained people ask questions of your data, it might be really useful.

It's worse at SQL generation than llama3 according to their own post.

https://www.snowflake.com/blog/arctic-open-efficient-foundat...

To be fair, that's comparing their 17B model with the 70B Llama 3 model.

To stay fair, their "17B" model sits at 964GB on your disk and the 70B Llama 3 model sits at 141GB. unquantized GB numbers for both

Sorry, it sounds like you know a lot more than I do about this, and I'd appreciate it if you'd connect the dots. Is your comment a dig at either Snowflake or Llama? Where are you finding the unquantized size of Llama 3 70B? Isn't it extremely rare to do inference with large unquantized models?

to stay fairer, the required extra disk space for snowflake-arctic is cheaper then the required extra ram memory for llama3

For decent performance, you need to keep all the parameters on memory for both. Well, with a raid-0 of two PCIe 5 SSDs (or 4 PCIe 4) you might get 1 t/s loading experts from disk on snowflake-artic... but that is slooow.

It's a statistical model of language. If it wasn't trained on text that says "I don't know that", then it's not going to produce that text. You need to use tools that can look at the logits produced and see if you're getting a confident answer or noise.

To me, your complaint is equivalent to "I tried your new screwdriver and it couldn't even hammer in this simple nail!"

You're using it wrong. Expecting an auto-complete engine to not make up words is an exercise in frustration.

Please read the post before commenting.

Reply on news.ycombinator.com