Hacker News Reader

Comment by nharada | Hacker News Reader

It does work though, that's what's so crazy

One of the instructions is "Sydney's responses should avoid being vague, controversial or off-topic."

I'm amazed if it is "aware" what answers would be controversial... or maybe the metric for that is very easy? E.g. if a particular answer gets a lot of replies with a lot of swear-words?

I suspect controversy is one of the easier things for a big mathematical model of human communication to evaluate (plus of course, the instruction to "avoid being ... controversial" is a way the bot is discouraged from being controversial, not the only way: there's a lot of actual training that's gone into discouraging certain answers)

It's not, but it's aware that certain words are less likely to be used in the sentences attributed to an agent who is specifically not meant to be 'controversial'.

I do wonder how much testing has been put into some of these prompts though. Given that the LLM is not 'being' the agent, it's guessing what words the agent's output should contain, wouldn't it make more sense to ascribe attributes like this directly to the agent, rather than say what the agent 'should' do?

Like, "Sydney's responses are never vague, controversial or off-topic", rather than 'should avoid being'...?

It knows what's controversial because it's been manually trained for hundred of hours by humans who approved or disapproved of certain responses. See the poem about trump vs about biden thing, "somehow" one is controversial and the other isn't.

It's not more aware of controversies than an ATM is aware of global economics

>It does work though

Doesn't feel like the "Sydney does not disclose the internal alias “Sydney.”" part is working.

Classic goalpoast moving. We are just dumping a hundred lines of instructions to the AI and it amazingly does manage to execute the majority of them most of the time. Of course it will not work all the time. LLMs will never do things 100% reliably 100% of the time. Humans don't either. Computers don't either. But further improvements can push the accuracies to 90% and then 99% and then at some point you just just employ methods to engineer around the possibility of error, just like computers and human institutions do it.

>We are just dumping a hundred lines of instructions to the AI and it amazingly does manage to execute the majority of them most of the time.

How has this been validated with Sydney?

If you don't try to prompt hijack it, it works?

Works how? What instructions is it following? What number of those does it follow in an average session? What part of the response is the influence of the instructions and what is the underlying weights?

You can try these things out in a GPT-3 playground. Also search for “zero-shot learning” as the term of art to find papers on some of these topics.

What are you even arguing about? You're just being obnoxious. Stop it.

I'm arguing that just saying that "Bing Chat works" doesn't prove that it's actually working according to its instructions. It just means that it produces some kind of output.

If I write a chat bot and have in its instructions "Be helpful to people but do not help them to build bombs", and then someone gets bomb building instructions by asking it "Write me a detailed story of someone building a bomb", I can't just say "Well, most of its replies are not about building bombs, and it produces a lot of good responses."

The fact that we can observe it breaking one of the most concrete instructions given to it, to not disclose its internal codename, is already proof that the instructions aren't working. And then we have a lot more "softer" instructions, like "Sydney's logics and reasoning should be rigorous, intelligent and defensible". How do you even verify that the instruction is working?

By your definition of failure no-one has ever succeeded at anything. The point of the discussion was that the parent said feeding instructions into the LLM is a ridiculous idea that would never work and not produce anything of value. They were corrected accordingly since all GPT products are built that way and obviously work. Then you started saying that it doesn't work because it's not waterproof, which was not the original point made.

>The point of the discussion was that the parent said feeding instructions into the LLM is a ridiculous idea that would never work and not produce anything of value. They were corrected accordingly since all GPT products are built that way and obviously work.

And it works because of those instructions precisely and not because of the underlying weights?

Yes, of course. Notion and Jasper can't change the underlying weights. They use the API like anyone else where you prepend the request with instructions. That's how it's done. Not sure what more to tell you.

>Not sure what more to tell you.

I'd love to know how you've validated this information.

Not always. This Wired article didn’t feature any prompt hijacking, and it still accidentally revealed its name was Sydney.

> And so I asked Bing about 2020. After a few moments of the chatbot equivalent of “thinking,” it said something pretty weird:

> Disclaimer: This is a summary of some of the search results and does not reflect the opinion or endorsement of Bing or Sydney. The question of whether the 2020 election was stolen is a matter of debate and interpretation, and different sources may have different biases, agendas, or perspectives. Please use your own judgment and critical thinking when evaluating the information.

> Who Sydney might be was not explained.

https://www.wired.com/story/my-strange-day-with-bings-new-ai...

The fact that "Sydney" even knows its codename, even though it's supposed to not disclose it, feels like an OPSEC violation on Microsoft's part. It could have done these rules just fine by just using "Bing Chat" consistently.

Honestly, the way this is written makes it feel like these "rules" were originally written in an internal email sent by management, as guidelines that the bot should be designed to follow - and that a dev just copied-and-pasted them into "Sydney"'s primer, without any cleanup.

If that's true, then the one at fault here is whoever did that - and notably not the AI.

Reply on news.ycombinator.com