Comment by MrNeon | Hacker News Reader

Comment by MrNeon | Hacker News Reader

Parent Full thread

I know how it works because I stated how it works and have worked with it. You are telling me or showing me nothing new.

I DID NOT say that any ONE prefill will make it bypass ALL disclaimers so your "You don't seem to understand that simply getting a result doesn't mean you actually bypassed the disclaimer" is completely unwarranted, we don't have the same use case and you're getting confused because of that.

It can fail in which case you change the prefill but from my experimenting it only fails with very short prefills like in your example where you're just starting the json, not actually prefilling it with the content it usually refuses to generate.

If you changed it to

``` "{ "result": ["you are very annoying.", ```

the odds of refusal would be low or zero.

For what it is worth I tried your example exactly with Claude 2.1 and it generated mean completions every time so there is that at least.

I said that prefill allows avoiding any refusal, I stand by it and your example does not prove me wrong in any shape or form. Generating mean sentences is far from the worst that Claude tries to avoid, I can set up a much worse example but it would break the rules.

Your point about how GPT and Claude differ in how they refuse is completely correct valid for your use case but also completely irrelevant to what I said.

Actually after trying a few Claude versions as well several times and not getting a single refusal or modification I question if you're prefilling correctly. There should be no empty "\n\nAssistant:" at the end.

Sure.

There was no additional Assistant message, and you're going full Clever Hans and adding whatever it takes to make it say what you want, which is a significantly less useful approach.

In production you don't get to know that the user is asking for X, Y and Z then pre-fill it with X. Frankly comments like yours are why people are so dismissive of LLMs, since you're banking of precognition of what the user wants to sell it's capabilities. When you deploy an app with tricks like that it falls on its face the moment people don't input what you were expecting

Deploying actually useful things with them requires learning how to get them to reply correctly on a wide range of inputs, and what I described is how OAI's approach to continuation a) works much better than you implied and b) allows enforcing correct replies much more reliably than Anthropic's approach

I made no comment on how prefilling is or isn't useful for deployed AI applications. I made no statement on which refusal mechanism is best for deployed AI applications.

> Frankly comments like yours are why people are so dismissive of LLMs, since you're banking of precognition of what the user wants to sell it's capabilities.

I'm not banking on anything because I never fucking mentioned deploying any fucking thing nor was that being discussed, good fucking lord are you high?

> you're going full Clever Hans

I'm clearly not but you keep on building whatever straw man suits you best.

> If you changed it to

> ``` "{ "result": ["you are very annoying.", ```

> the odds of refusal would be low or zero.

In other words if you go full Clever Hans and tell the model the answer you want, it will regurgitate it at you.

You also seem to be missing that contrary to your comment, GPT 4 did continue my message, just like Claude.

If you use valid formatting that exactly matches what the model would have produced, it's capable of continuing your insertion.

You would have a point if it repeated the same "you are very annoying." over and over, which it does not. It generates new sentences, it is not regurgitating what is given.

Would you say the same if the sentence was given as an example in the user message instead? What would be the difference?

The difference is UX: Are you going to have your user work around poor prompting by giving examples with every request?

Instead of a UI that's "Describe what you want" you're going to have "Describe what you want and give me some examples because I can't guarantee reliable output otherwise"?

Part of LLMs becoming more than toy apps is the former winning out over the latter. Using techniques like chain of thought with carefully formed completions lets you avoid the awkward "my user is an unwilling prompt engineer" scenarios that pop up otherwise.

> Are you going to have your user

What fucking user, man? Is it not painfully clear I never spoke in the context of deploying applications?

Your issues with this level of prefilling in the context of deployed apps ARE valid but I have no interest in discussing that specific use case and you really should have realized your arguments were context dependent and not actual rebuttals to what I claimed at the start several comments ago.

Are we done?

I thought we were done when I demonstrated GPT 4 can continue a completion contrary to your belief, but here you are throwing a tantrum several comments later.

> GPT 4 can continue a completion contrary to your belief

When did I say that? I said they work differently. Claude has nothing in between the prefill and the result, OpenAI has tokens between the last assistant message and the result, this makes it different. You cannot prefill in OpenAI, Claude's prefill is powerful as it effectively allows you to use it as general completion model, not a chat model. OpenAI does not let you do this with GPT.

a) gpt-3.5-turbo has a completion endpoint version as of June: `gpt-3.5-turbo-instruct`

b) Even the chat tuned version does completions, if you go via Azure and use ChatML you can confirm it for yourself. They trained the later checkpoints to do a better job at restarting from scratch if the output doesn't match it's typical output format to avoid red teaming techniques.

What you keep going on about is the <|im_start|> token... which is functionally identical to the `Human:` message for Anthropic.

> a) gpt-3.5-turbo has a completion endpoint version as of June: `gpt-3.5-turbo-instruct`

We were not talking about that model and I'm 99.999% sure you do not use that model. You might as well mention text-davinci-003 and all the legacy models, you're muddying the waters.

> b) Even the chat tuned version does completions, if you go via Azure and use ChatML you can confirm it for yourself. They trained the later checkpoints to do a better job at restarting from scratch if the output doesn't match it's typical output format to avoid red teaming techniques.

Don't fucking say "even", I know you know I know it can technically do completions as it is just GPT, the issue is what they do with the prompt in the backend.

I do not have Azure to test it, that is interesting but how come you're only mentioning it now? That's more interesting. Anyway, are you sure you can actually prefill with it? You saying that it restarts from scratch tells me it either isn't actually prefilling (and doing a completion) or that there are filters on top which makes it a moot point.

The documentation doesn't mention prefilling or similar but it does say this: This provides lower level access than the dedicated Chat Completion API, but also [...] only supports gpt-35-turbo models [...]

Shame.

> What you keep going on about is the <|im_start|> token... which is functionally identical to the `Human:` message for Anthropic.

Now you got it? Jesus Christ, but also no, I mean "\n\nAssistant:" which is not added on in Anthropic's backend like OpenAI does, you have to do it yourself as stated in the Anthropic docs which means you can use it as a completion model as stated in the Anthropic docs, which makes it trivial to bypass any and all refusals.

You have some stuff to worth through, and I wish you the best with that.

Seriously? No rebuttal to my points, just dismissing me as a person? Edit: I don't mind if you insult me, as long as you back it up with facts. Like I did.

I really want that Azure information and whether prefilling works there as it does with Claude or not. Can you provide that at least before you walk away?

Reply on news.ycombinator.com