Hacker News Reader

Comment by textninja | Hacker News Reader

I’m not fond of the provocative title because prompting is easy and only getting easier; the advice seems to be predicated on the use of relatively deficient LLMs. I don’t doubt there will still be operator skill involved, but I anticipate the state of the art for LLMs ability to adapt to “bad” prompts will outpace our ability to learn to prompt them effectively.

Disclaimer: I watched the video but didn’t read the paper.

View on HN

LLMs will definitely get better -- and people will adapt too. But natural language contains inherent ambiguity, and that's not going to change.

A large part of the paper talks about how the challenges of programming and developing ML systems don't go away just because you're using natural language and an LLM.

For example, users incorrectly extrapolating from a single failed/successful prompt to other contexts: that's not something that will be solved by better LLMs, really. It's solved by getting users to recognize when they do/don't have enough data to believe what they're seeing.

Disclaimer: I wrote the paper.

Thanks for the official TL/DR.

... And the laughs.

Paper is great, too. What's it like trying to write a paper while the field is shifting so rapidly? Is there anything you would have liked to include that had to be cut so you could get it out the door?

Oh, that's not a full TL;DR -- but sounds like you've read the paper. :)

Shifting field: yes, we spend a lot of time trying to figure out what is going to change and what isn't, at least in HCI (my field). LLM capabilities are changing by the...day? So anything that's very focused on LLM capabilities won't be relevant for long.

Humans don't change quite as fast!

Keep in mind this paper was first written in September 2022, reflecting work done prior to then...practically an eternity in which we've since seen ChatGPT take the world by storm, and now GPT-4. But I think the lessons there are still very relevant, and will be for quite some time.

I think you're right about prompts getting "easier" but I don't think it's a good thing. I expect it will evolve like google search. Where initially there are ways to increase specificity, or at least introduce enough randomness to get some different results, it will converge to something that ignores most of what you prompt and gives you what OpenAI wants you to see. That's really the only way adapting to "bad" prompts even could work

I think there are a lot of instances where writing prompts can be hard just because it’s hard to express your needs in words sometimes. Bad prompts are often ambiguous and there is only so much even a perfect LLM can correct for. That is, until we have direct connections to our brains.

What makes you think it is clearer in your brain than it is in your words? My experience is that in my case it isn't.

Agree. Writing is a necessary part of clear though.

Who said we’d only give the LLMs read access?

Reply on news.ycombinator.com