Hacker News Reader

Comment by zevv | Hacker News Reader

It actually still scares the hell out of me that this is the way even the experts 'program' this technology, with all the ambiguities rising from the use of natural language.

View on HN

Keep in mind that this is not the only way the experts program this technology.

There's plenty of fine-tuning and RLHF involved too, that's mostly how "model alignment" works for example.

The system prompt exists merely as an extra precaution to reinforce the behaviors learned in RLHF, to explain some subtleties that would be otherwise hard to learn, and to fix little mistakes that remain after fine-tuning.

You can verify that this is true by using the model through the API, where you can set a custom system prompt. Even if your prompt is very short, most behaviors still remain pretty similar.

There's an interesting X thread from the researchers at Anthropic on why their prompt is the way it is at [1][2].

[1] https://twitter.com/AmandaAskell/status/1765207842993434880?...

[2] and for those without an X account, https://nitter.poast.org/AmandaAskell/status/176520784299343...

Anthropic/Claude does not use any RLHF.

Is that a claim they've made or has that been externally proven?

What do they do instead? Given we're not talking to a base model.

Supposedly they use "RLAIF", but honestly given that the first step is to "generate responses... using a helpful-only AI assistant" it kinda sounds like RLHF with more steps.

https://www.anthropic.com/research/constitutional-ai-harmles...

LLM Prompt Engineering: Injecting your own arbitrary data into a what is ultimately an undifferentiated input stream of word-tokens from no particular source, hoping your sequence will be most influential in the dream-generator output, compared to a sequence placed there by another person, or a sequence that they indirectly caused the system to emit that then got injected back into itself.

Then play whack-a-mole until you get what you want, enough of the time, temporarily.

same as with asking humans to do something

When we do prompt engineering for humans, we use the term Public Relations.

There’s also Social Engineering but I guess that’s a different thing :)

No, that's exactly the thing - it's prompt injection attacks on humans.

... or - worse even - something you think is what you want, because you know not better, but happens to be a wholy (or - worse - even just subtly partially incorrect) confabulated answer.-

As a product manager this is largely my experience with developers.

We all use abstractions, and abstractions, good as they are to fight complexity, are also bad because sometimes they hide details we need to know. In other words, we don't genuinely understand anything. We're parrots of abstractions invented elsewhere and not fully grokked. In a company there is no single human who understands everything, it's a patchwork of partial understandings coupled functionally together. Even a medium sized git repo suffers from the same issue - nobody understands it fully.

Wholeheartedly agree. Which is why the most valuable people in a company are those who can cross abstraction layers, vertically or horizontally, and reduce information loss from boundaries between abstractions.

Some executive: "That's nice, but what new feature have you shipped for me recently?"

Hey, you read my review!

Well, hopefully your developers are substantially more capable, able to clearly track the difference between your requests versus those of other stakeholders... And they don't get confused by overhearing their own voice repeating words from other people. :p

It probably shouldn't be called prompt engineering, even informally. The work of an engineer shouldn't require hope.

This is the fundamental change in the concept of programming

From computer’s doing exactly what you state, with all the many challenges that creates

To is probabilistically solving for your intent, with all the many challenges that creates

Fair to say human beings probably need both to effectively communicate

Will be interesting to see if the current GenAI + ML + prompt engineering + code is sufficient

Nah man. This isn’t solving anything. This is praying to a machine god but it’s an autocomplete under the hood.

Honestly, this sort of programming (whether it's in quotes or not) will be unbelievably life changing when it works.

I can absolutely put into words what I want, but I cannot program it because of all the variables. When a computer can build the code for me based on my description... Holy cow.

if this doesn't work well with super high level languages, why would it work really well with LLMs?

I can have a conversation with LLM's. they can walk me through the troubleshooting without prior knowledge of programming languages.

That seems like a massive advantage.

I don’t think the people who engineered the Golden Gate Bridge, Apollo 7, or the transistor would have succeeded if they didn’t have hope.

I think OP's point is that "hope" is never a substitute for "a battery of experiments on dependably constant phenomena and supported by strong statistical analysis."

It should be called prompt science.

It's literature.

I never thought my English degree would be so useful.

This is only half in jest by the way.

So many different areas of knowledge can be leveraged, as long as you're able to experiment and learn.

Engineering requires hope; anything outside the bounds of our understanding (like exactly how these models work) requires it.

Scientists fold proteins, _hoping_ that they'll find the right sequence, based on all they currently know (best guess).

Without hope there is no need try; without trying there is no discovery.

What a silly thing to say. Engineering is just problem solving.

It still scares the hell out me that engineers think there’s a better alternative that covers all the use cases of a LLM. Look at how naive Siri’s engineers were, thinking they could scale that mess to a point where people all over the world would find it a helpful tool that improved the way they use a computer.

Do you have any evidence to suggest the engineers believed that?

The original founders realised the weakness of Siri and started a machine learning based assistent which they sold to Samsung. Apple could have taken the same route but didn't.

So you're saying the engineers were totally grounded and apple business leadership was not.

13 years of engineering failure.

The technology wasn’t there to be a general purpose assistant. Much closer to reality now and I have found finally Siri not to be totally terrible.

My overall impression using Siri daily for many years (mainly for controlling smart lights, turning Tv on/off, setting timers/alarms), is that Siri is artificially dumbed down to never respond with an incorrect answer.

When it says “please open iPhone to see the results” - half the time I think it’s capable of responding with something but Apple would rather it not.

I’ve always seen Siri’s limitations as a business decision by Apple rather than a technical feat that couldn’t be solved. (Although maybe it’s something that couldn’t be solved to Apple’s standards)

I mean, there are videos from when Siri was launched [1] with folks at Apple calling it intelligent and proudly demonstrating that if you asked it whether you need a raincoat, it would check the weather forecast and give you an answer - demonstrating conceptual understanding, not just responding to a 'weather' keyword. With senior folk saying "I've been in the AI field a long time, and this still blows me away."

So there's direct evidence of Apple insiders thinking Siri was pretty great.

Of course we could assume Apple insiders realised Siri was an underwhelming product, even if there's no video evidence. Perhaps the product is evidence enough?

[1] https://www.youtube.com/watch?v=SpGJNPShzRc

Reply on news.ycombinator.com