Hacker News Reader

Comment by TeMPOraL | Hacker News Reader

Unfortunately, that "unwanted noise" is a space for the models to compute; trying to eliminate it gives suboptimal responses. What you can do instead is try to corral it - let the model "think" like it wants, but guide it to add markers wrapping the thinking and/or result, then filter out the thinking in UI (for interactive applications) or as an intermediate/post-processing step (for hidden "building blocks").

If you're using Anthropic models, you may actually get improvements from prompting the model to maintain a tagging discipline; see https://docs.anthropic.com/en/docs/build-with-claude/prompt-....

View on HN

As other people pointed out here you can also add "verbosity sinks" as text fields in structured output, recently I've also been experimenting with tool calls to support guided self-talk in a way that doesn't necessarily all accumulate in the context (e.g. if not all the tool parameters get echoed back).

Thank you (and teMPOral) for these comments, this sounds potentially useful to me.

I hate to ask this, but I'm struggling to find any thorough posts or articles or papers about this, do you have any links you could point me toward?

Speaking only for myself these ideas are a combination of things I've seen scanning new papers and informal discussions with other people working in the area. Feel free to shoot me an e-mail though, maybe I can point you somewhere more specific.

Edit: The "verbosity sink" name is inspired by the idea from the paper below although they're not actually at all the same thing.

https://arxiv.org/abs/2309.17453

Here is a short example that came up for me last week.

I had a set of documents I wanted to classify according a taxonomy that is well known (so it is exists in the training data of all the major llm models I tested)

If I have prompt like, `You are an expert classification system. Using the Classification Approach Foo, consider the following and output the category in JSON format, such as {"class":"bar"} `

This works ok, but it works much better if I tell it to output {"class":"bar", "reason": "baz"} and improved with some other approaches like adding "related_class" or "parent_category" which would otherwise be redundant.

Also including some few-shot examples helped, but the biggest benefit came from the "reason" field. Trying justification or other synonyms seems to produce the same output.

I suspect this is something similar to CoT.

Great, thank you (and hedgehog), that makes sense

Have you tested moving the "reason" field before the "class" field? That may encourage better CoT instead of having the model justify the class after it already picked it. Anecdotally, I saw a 5% boost in performance from a NER system from having the model output the entity's class at the end rather than the beginning.

This has worked for me for sentiment classification, i had a score and a summary field and the results were much more accurate.

Does it still work if the class field is output before the reason?

Yes. But, reason first did work (very slightly) better.

It may be the self-aware human bias tainting this reasoning, but it seems convergent with our own psyche/brain processes, and/or inherent to the way we commonly express conscious thoughts.

Percolating tokens that allow a more "accurate" latent space appear to be more accurate, but are nearly actually useless noise. Almost a virtual shower thought.

Because people only put the answer at the end of a grammatically correct statement, with the more "reasoned" statements being more articulately percolated/logically sound, and that is expressed grammatically. These statements are inferred to be associated with intellectual boiler-plate. They may be correlated and not actually causative, but that would require a multiple component architecture with embeddings being used as a proto-"qualia" and that is getting hairy.

Facts should "only" have to be read once, and should be explicitly defined with a more secure of a confidence completely. Implicit inferences from those explicit facts should be emitted from a different, less confident module; with the chat boilerplate being tacitly composed finally when presenting the output to the user.

Of course separating the baby from the bathwater is the hard (not impossible) part.

It seems to me that it would make sense to just include more <BOS>-like meta tokens at the beginning in such cases, and have them as a prefixed scratch space that can be suppressed by treating them as non-output tokens.

it should be possible to ask model to think aloud (or step-by-step) and then give summary. in one or two prompts. give only summary back to user.

We have Marco o1 at home.

marco o1 at home: https://www.reddit.com/r/LocalLLaMA/comments/1gyx1hj/macroo1...

That's... a good result, actually. No, I'm serious.

This reads exactly like my inner thought process on a novel or tricky task I'm asked to solve, especially when I know I'm tired (or drunk, back in the times I consumed alcohol on a regular basis), and need to spell everything out (out loud or in a text file).

Hell, it's exactly how I expect a kid who just learned about fractions would think. I have a vague recollection I processed such tasks this explicitly as a kid, until I understood the topic.

LLMs pulling this off reliably? That's huge progress. I used to say[0] that GPT-4 is best imagined as a 4 year old kid that memorized half the Internet. But this? This is 8 year old's stuff.

[0] - I currently prefer comparing it to "inner voice", and its performance and propensity to hallucinations to a smart schoolkid that's being asked questions by the teacher about things they only read about but didn't fully process, and who's pressured into giving some answer, as saying "I don't know" is an instant F and public humiliation. Such kid will be forced to extrapolate on the spot, but if they're smart enough and remember enough, they'll often get it at least partially right. I know that from personal experience :).

Yeah, I don't know if someone thought this was bad or something, but it seems like valid reasoning. We may need to give these models a sense of things that require more and less detailed reasoning and better knowledge of what they know internally, but the "research work" the model claims to be, it seems like it's doing a good job.

The poster also shared in a comment https://preview.redd.it/u8vs29hq5w2e1.png?width=2704&format=... which did get the intended laugh out of me, but even that seems fair enough. I'm currently traveling in a country where most people speak a language I don't know well. You better believe I've been thinking through even trivial greetings, considering the setting, formality, appropriate follow ups, etc.

OP here. I liked Macro-o1 (Marco1?) but as you pointed out, we need to teach these models to spend their system 2 thinking more economically.

One of the things complicating the example is that the counting-letters task is actually a legitimately hard one for it, even if it's trivial for us: it's bad at letter stuff because of the way it represents text with tokens. I believe the problem exists, but the letter thing isn't necessarily a great example of something that it should recognize as trivial.

Looking at that example, I too feel it's a thought process I could go through once or twice. I think this highlights an important difference between humans and LLMs: a human can think such things explicitly in their head, or on paper (I do it in text editor more frequently than I'd care to admit), once or twice, and then it sticks, quickly becomes more of a "system 1" thing. With LLMs, the closest to that outside training/fine-tuning would probably be prompt caching. It would be great if we could figure out some kind of on-line learning scheme so the model could internalize its own thoughts and persist them between conversations, but in the latent space, not prepended as token input.

> ou better believe I've been thinking through even trivial greetings

Even after thinking through what to say, I used the wrong greeting in a shop half an hour ago and the person working there called me on it.

interesting

Reply on news.ycombinator.com