Hacker News Reader

Comment by shortrounddev2 | Hacker News Reader

> I asked it "Rust crate to access Postgres with Arrow support"

Is that how you actually use llms? Like a Google search box?

Exactly. An LLM is not a conventional search engine and shouldn't be prompted as if it were one. The difference between "Rust crate to access Postgres with Arrow support" and "What would a hypothetical Rust crate to access Postgres with Arrow support look like?" isn't that profound from the perspective of a language model. You'll get an answer, but it's entirely possible that you'll get the answer to a question that isn't the one you thought you were asking.

Some people aren't very good at using tools. You can usually identify them without much difficulty, because they're the ones blaming the tools.

It's absolutely how LLMs should work, and IME they do. Why write a full question if a search phrase works just as well? Everything in "Could you recommend xyz to me?" except "xyz" is redundant and only useful when you talk to actual humans with actual social norms to observe. (Sure, there used to be a time when LLMs would give better answers if you were polite to them, but I doubt that matters anymore.) Indeed I've been thinking of codifying this by adding a system prompt that says something like "If the user makes a query that looks like a search phrase, phrase your response non-conversationally as well".

every token contributes to the output

Totally agree here. I tried the following and had a very different experience:

"Answer as if you're a senior software engineer giving advice to a less experienced software engineer. I'm looking for a Rust crate to access PostgreSQL with Apache Arrow support. How should I proceed? What are the pluses and minuses of my various options?"

“Prompting” is kind of a myth honestly.

Think about it, how much marginal influence does it really have if you say OP’s version vs a fully formed sentence? The keywords are what gets it in the area.

That is not correct. The keywords mean nothing by themselves. To a transformer model, the relationships between words is where meaning resides. The model wants to answer your prompt with something that makes sense in context, so you have to help it out by providing that context. Feeding it a sentence fragment or a disjoint series of keywords may not have the desired effect.

To mix clichés, "I'm feeling lucky" isn't compatible with "Attention is all you need."

I find that providing more context and details initially leads to far more success for my uses. Once there’s a bit of context, I can start barking terms and commands tersely.

I find more hallucination - like when you're taught as a child to reflect back the question at the start of your answer.

If I am not careful, and "asking the question" in a way that assumes X, often X is assumed by the LLM to be true. ChatGPT has gotten better at correcting this with its web searches.

I am able to get better results with Claude when I ask for answers that include links to the relevant authoritative source of information. But sometimes it still makes up stuff that is not in the source material.

That’s fair. I think the difference here is that the entire context needed is provided.

If you’re having to explain an existing problem with edge cases, then sure, the context window needs the edge cases defined as well.

That’s the biggest problem I have on my local LLM use - limited context size compared to the big guys offerings.

Is this really the case, or is it the case with Claude etc because they've already been prompted to act as an "helpful assistant"? If you take a raw LLM and just type Google search style it might just continue it as a story or something.

Prompting is not a myth. The words of the prompt matter huge.

The problem with this prompt to me is not that it is not in a full sentence but that it isn't exact enough.

Probabilistically, "rust" is not about the programming language but the corrosion of metal. Then arrow.

Give the model basically nothing to work with then complain it doesn't do exactly what you want. Good luck with that.

It's funny because many people type full sentence questions into search engines too. It's usually a sign of being older and/or not very experienced with computers. One thing about geeks like me is we will always figure out what the bare minimum is (at least for work, I hope everyone has at least a few things they enjoy and don't try to optimise).

It's not about being young or old, search engines have moved away from pure keyword searches and often typing your actual query gives better results than searching for keywords, especially with Google.

Wonder if that's why so many people hate its results lol. It shifted keyword searching to full sentence searching, but many of us didn't follow in the shift.

Well, compare it to the really good answer from Grok (https://x.com/i/grok/share/MMGiwgwSlEhGP6BJzKdtYQaXD) for the same prompt. Also, framing as a question still pointed to the non-existent postgres-arrow with Claude.

That's primarily how i do, though it depends on the search ofc. I use Kagi, though.

I've not yet found much value in the LLM itself. Facts/math/etc are too likely incorrect, i need them to make some attempt at hydrating real information into the response. And linking sources.

Reply on news.ycombinator.com