Hacker News Reader

Comment by dahart | Hacker News Reader

> The reason [LLMs are not a good tool to do truly effective brainstorming] is LLMs are trained to follow existing patterns in the human-produced corpus, and not natively taught to “brainstorm”.

The problem with this argument is that people do the same thing, we’re not that great at brainstorming either. When we brainstorm in groups, we’re just bringing multiple points of view together. The more data LLMs are trained on, the more viewpoints it might be able to bring that you haven’t considered.

That said, LLMs and all NNs so far are built to interpolate, and they are bad and have unbounded error when extrapolating outside their training examples. That is a good reason to not expect today’s AI to come up with new ideas.

View on HN

> people do the same thing

You're right, most people do that, but that's because they haven't trained themselves to be inventors. This is a skill that definitely needs to be developed more, but finding truly creative (and sometimes backwards-seeming) solutions can sometimes mean thinking about problems in a completely different way.

However, all LLM's do this. That's one of the points of the article.

There was an actual peer-reviewed article from an institution yesterday showing, numerically, robustly, human evaluated, that LLMs are better at creative thinking than humans.

I don't think handwaving about exclusively outputting training data and random punditry via article really holds up here.

It's wrong for so many reasons, from the training data having more perspectives, to the "Fat Tony" test, empirically, we can go make it output novel combinations right now, to the CS test, we can have it emulate arbitrary Turing machines.

Might you possibly have a link?

My bad, looked it up and meant to edit it in, dunno what happened: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10858891/

The methodology of this paper is seriously suspect.

First they have a very small sample size of around n = 150. Second, they don't represent average husband, they use people working on mechanical Turk for $9 an hour.

Finally, they don't have any standard measures of divergent thought - they just use 4 different types of question answering problems, and those are supposed to measure creativity. Whether they do or don't would require us to evaluate the questions.

It's important to be clear that this is saying that LLMs might be more creative than low paid workers on mechanical Turk, given no training time. That's not a very good representation of the average human, or the fact that humans need training.

So for example, an average doctor or scientist might still be more creative than LLMs.

Tldr; this is not a good paper to throw around if you want to convince people. It's truly awful in its design, and the way it's presented is not always honest.

No results for mechanical turk, n=151 is perfectly significant for what amounts to a taste test (I was quite taken aback to read that, 151 is low in very few contexts), I'm not "throwing it around", whatever that means in this context, and the extreme reaction and 100% focus on it belies that it was 1 of 3 short points I made, as well as that the research backs up the claim that was shut down, that is intuitively correct, I'd rather have a 1000 specialists at undergrad level ideating than me and Einstein and Picasso and Steve Jobs.

So, yeah, between that, and that both your average person and someone arguing from Turing tapes can show AI doing things not in the training data, I'm going with actual data thats being "thrown around" rather than the rando blog post that argues from "by definition it can't be creative because it isn't original because it only repeats training data" Call me when they're at N=152 on a blind test.

They also don’t seem to be taking into consideration the fact that answers to those tests may already be present in the training data.

There is a difference between a verbatim answer and an answer in the shape of a verbatim answer. The latter is not much better if you actually want to do new things.

> However, all LLM’s do this.

LLM’s degree of adherence to training patterns can be directly tuned, per request (up to whatever the maximum it is capable of.)

Can this be leveraged to build a framework around an LLM for brainstorming usefully? I’m not sure. But “Nope, because LLMs just directly adhere to existing patterns” is definitely not a useful answer to that question. It is the kind of answer someone who just parrots common patterns with surface-level knowledge of the subject area would come up with. (Huh, did a default-settings LLM write this article?)

> LLM’s degree of adherence to training patterns can be directly tuned, per request (up to whatever the maximum it is capable of.)

I think people often forget the behavior of instruction following is still based on SFT training data. This is exactly "adherence to training patterns". Prompt engineering is not elixir, it's just a way to utilize the patterns seen in training data.

> Prompt engineering is not elixir

I’m not talking about prompt engineering when I talk about per request tuning of how closely it follows established patterns, I’m talking about inference parameters (temperature, top_p, top_k, are common for most models and ways of calling them, some others may be available, too.)

There is a thread below discussing tuning temperature and why it's not tied to creativity. I think the point is temperature and other sampling parameters don't have any inherent mechanism to control the adherence to training pattern as a whole (all top-X candidates are all from training data). Deviating from the max probability in next token generation has no relation with thinking out of box. It does give more generation results that may be lucky to hit something, but I doubt the cost effectiveness

How will increasing temperature improve the arithmetic performance of the model with numbers it has never seen? When prompted the right way, the model has trouble sticking to the answer that is implied by its chain of thought process. For some strange reason it knows how to perform the task, but when it is time to output the answer, it decides to recalculate it from scratch using attention. Before you complain about me benchmarking arithmetic performance. Consider that I could have defined any algebraic structure and I could have made the exact same complaint. It's just that arithmetic is very easy to verify using a calculator. Dismissing the easy arithmetic also dismisses all operations on any algebra.

The obvious problem is that these logic and arithmetic operations don't really need exhaustive training examples. Training the LLM is supposed to teach it a method or process by which it arrives at the answer, not just the question answer pair itself, which tends to only lead to very good approximate results. That is ok when using language, since your exact words don't really matter if you can express yourself in another way, but when it comes to algebras, accuracy is key.

It really is strange that when one wishes for a more capable LLM that one is told how stupid one is for wanting such a thing. What exactly is so wrong about giving the model a price list and an example of an order at a restaurant and then asking how much the whole order costs and expecting the correct total sum? For some strange reason I'm getting numbers that would make sense if I generated the text in a vacuum without the price list, as if they are following some existing pattern...

> How will increasing temperature improve the arithmetic performance of the model with numbers it has never seen?

“Brainstorming” and “doing precise arithmetic” are wildly different tasks. It is a complete non-sequitur to ask how a suggestion about the former will help the latter.

Setting the temperature to a higher value would emulate this, wouldn't it? Then the model would be more willing to deviate from the most likely next token.

Increasing temperature just makes more uncommon tokens more likely to appear. This can include both uncommon ideas and uncommon spelling and grammatical mistakes. It also won't make ideas that the LLM isn't capable of thinking appear, unless you crank the temperature way up and get lucky (like monkeys on typewriters lucky)

Not just “uncommon tokens”. In the case when generation does weighted random sampling, it autoregressively takes its own past tokens into account. It generates whole different sequences. So if it’s diversity of output you want, you will get it.

That is also true for creative people. Generating interesting ideas is one thing then filtering out impractical or unaesthetic ideas is also needed.

Right, so in rough outline an LLM-based brainstorming framework might generate multiple responses to a question with multiple different combinations of temperature, top_p, top_k, etc., to get a mix of dull-but-baseline (you don’t want to miss those options) and more off-the-beaten path responses, then take each one and send it back to the LLM with a prompt to evaluate “does this provide a coherent answer – however unconventional or bad of an idea – to the question asked?” to filter out the cases where it was asked for summer vacation ideas and responded with something like “Banana banana banana banana” (this request probably with default tuning.)

We don't need more data, we need independent LLMs with slight randomness.

Aren’t independent LLMs just more data? Some AI researchers are arguing that more data is the primary thing that got us this far, and the primary thing that will improve AI from here. Here’s an example (and a very good talk whether you believe my summary) https://www.youtube.com/live/a13aqr07tJ4?si=FZO5m_XzrfpDhyQP

As a part-time generative artist for several decades, a user of Monte-Carlo methods on a daily basis, and author of some papers on the topic, I personally believe that randomness is not a good answer to anything creative. Randomness is boring, and average. Randomness only helps you when you have a high quality Markov model constraining what your RNG chooses between, and that’s more or less all LLMs actually are. Adding more randomness to creative works in general makes creative works muddy and lowers quality, it needs to be guided. Randomness is a useful tool, but is widely misunderstood IMO and not very effective at brainstorming style exploration; brainstorming is about solving problems in interesting ways, i.e., there is reasoning behind it, not slightly more random word salad.

>people do the same thing

how do you know?

Nothing is really original. That's why you can make finite lists of the possible plots of books and movies. It's why TVtropes is a thing. Everything is just a mashup of things before.

Ah, yes, Everything that can be invented has been invented. - apocryphally attributed to Charles H. Duell, Commissioner of US Patent Office way back in 1889.

This is a myth. Just because you don't really see blinding insight every day doesn't mean it doesn't exist (and it's more spectacular when you do see it!)

Another reason that this seems to happen is that innovation and invention tend to happen in more esoteric and technical realms today, and so they're only really fully understood by experts in that field.

Even Charles Duell wasn't speaking originally here:

"What has been will be again, what has been done will be done again; there is nothing new under the sun."

- Ecclesiastes 1:9 (circa 250 BC)

I happen to actually be a a computational biologist whose work could be considered "esoteric and technical" and whose papers are only fully understood by others in my field. But the fact is every paper of mine (and everybody else's) has 50 to 100 references. Because research is pretty much just reusing existing knowledge in the papers cited. The main source of "innovation", if it can be called that, is taking an idea from another field and applying it to your own. The idea of a scientist or inventor who comes up with an idea completely out the blue with no context is common in fiction, but that isn't how it works in reality.

i really suggest digging deeper than Netflix or HBO before making such a claim

Science is moving faster than ever and this dude is really like "people don't ever invent anything new actually"

> Nothing is really original. That's why you can make finite lists of the possible plots of books and movies. It's why TVtropes is a thing. Everything is just a mashup of things before.

That's a lot like claiming no one says anything new because we're also using more-or-less the same words we were using last century.

I think you're missing a lot.

I came here to say pretty much the same. It is so tiring to hear opinions like these from technically-minded people who clearly lack "better-than-average" creativity and lateral thinking skills. LLMs are garbage-in garbage-out in terms of brainstorming. If you aren't good at brainstorming without LLMs then they won't be much use. So many people are just seeing their own limits reflected back at them.

When I'm brainstorming math or cryptography or philosophy with an LLM it is only because of the connections and leaps that I introduce that leads to novel and amusing discoveries.

You have to develop successful patterns. Coaxing and condensing. Drilling into numbered lists. Asking for tables with progressively more columns and altered sorting. You have to know the spatial reasoning limits of the model. Asking for enumeration and disabling of social niceties. The number of skills needed is probably beyond most people's imaginations.

> and have unbounded error when extrapolating outside their training examples

I would say, try reading a conspiracy theory forum - or simply wait for one to get posted on HN, and then revisit this particular conclusion.

I don't think this is a unique failure of LLMs, except insofar as LLMs have less of a physical grounding in reality then humans do: but we have people out there who have convinced themselves they can project forcefields, and take that all the way through to getting punched in the face by a martial artist when it turns out that, no, they can't.

Reply on news.ycombinator.com