The serious science of trolling LLMs | Hacker News Reader

The serious science of trolling LLMs

lcamtuf.substack.com

29 pointszdw20 comments

> It’s when it fails at a simple task that we know what the limitations are — and trolls are the torch-bearers of this new englightenment.

I really don't agree. Watching AI fail spectacularly is a great bread and circus, but identifying something's deficiencies is not the same as fixing it. The thing these trolls get so damningly right is their denouncement of AI; we can't rely on shit text-generators that fly by the seat of their pants. The insinuated "but it's useful and does real work" shtick is over; I don't even see people talking about RAG value-add anymore.

The bubble popped, and now expectations are returning to normal. It's less of an enlightenment period and more of the "AI winter" that we've always known was going to occur once we exhausted the current scaling limits. The trolls are scratching beneath the surface and proving the prophecy right.

> The insinuated "but it's useful and does real work" shtick is over

> The bubble popped

NVIDIA has a 3.12 trillion market cap. AI investments are reportedly surging:

> Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds.

https://aiindex.stanford.edu/report/

I wanted more examples of trolls and how the behavior degrades. The one scripted joke example doesn't land with a punch.

> In short, I believe that the revealed objectives of LLM vendors differ starkly from what they say. They pour considerable resources into responding to every trollish LLM transcript that goes viral on the internet — and it’s useful to ponder why.

I'd be pretty interested in reading at how people do poke at LLMs and what this actually reveals about the vendors priorities.

I also thought the humor angle was odd as the models can create some very good humor when the context is set correctly.

Generative AI is basically like playing the game where each participant says one word of a sentence, one after the other.

https://youtu.be/kNJNV0GXO-s?si=3TKUpFoFHV7FNSHR

The AI has no idea where it is going before it starts outputting. If it ends up with a sentence that can’t be completed sensibly, it will output nonsense.

I think a better model needs to be able to plan out the idea to convey in advance and possibly backtrack.

I am curious, when you started writing your comment, did you know the last word of it would be "backtrack"?

Yes.

This isn't entirely correct. The models clearly demonstrate anticipatory behavior.

Stable diffusion doesn’t start with the top left pixel and generate them one by one. Why do llms?

You might be interested in language/text diffusion model research.

I am curious what the author would think about LLM based AI like Neuro-sama (AI vtuber that streams on twitch). Different use-case than the llm's the author was "trolling", but much better at humor and more resilient to trolls than usual.

Summarizing, we’re dealing with closed source software coupled with an AI model trained on unspecified data. Vendors glorify the latter and never say how much work is handled by the former.

Trust seems a big factor here.

> unlike humans, language models can’t be humiliated and do not learn from mistakes.

wut. I was pretty sure they do learn from mistakes. but they call it 'training'

The authors point is that LLMs dont learn "on the job" so to speak, as a user if you tell something like ChatGPT that it has made a mistake then it will put that in its short term memory (the session context) but it won't be committed to long term memory (the actual model weights) that persists across sessions. Training is usually a separate process that end-users have little to no direct influence over.

That's how it's currently deployed, but there is no requirement or restriction that you can't learn online. It's just kinda a bad idea, see Microsoft Tay...

not the ones publicly available I bet.

but it's one of those things that's only hard to do because of the capitalist race ongoing in that space

but very easy to consider... what if Gemini learned on the job??? like self-training? (alpha go zero?) these is a low-hanging fruit sort of idea is all I'm saying

Self-training is a lot easier to pull off in the context of something like AlphaGo which has a clearly defined goal to reach, LLMs don't really have that luxury as their output can't be evaluated by a rigid set of rules.

how about "user engagement"?

In the context of prediction, there is no such thing as a “mistake”, any more than a d20 die can make a mistake. You might get a 1 when you wanted a 20, but you can hardly blame the machine, only the manufacturer who forgot to weight it with the appropriate (or inappropriate) bias!

Reply on news.ycombinator.com