Comment by tbalsam | Hacker News Reader

Comment by tbalsam | Hacker News Reader

(moving this comment to this thread):

1. Models learn based on their data. Yes, data can be poisoned, but there's not much way around this. 2. Non-linear models, unlike linear models, have increasingly 'many' 'surfaces' of behavior, conditionally dependent upon the input model context.

3. Classifying models as secretly being 'deceptive' is very, very, very silly (and ridiculous, I might add, to boot) when in fact it's just a very-much-basic lower level function of some kind of contextually dependent behaviors. Congratulations, models are conditioned on context, it's as if that's one of the main ingredients behind the entire principle of how LLMs work. Rebranding and obfuscating this with a marketing term like "deception" is at least two things to me. A. It's mathematically wrong, and B. It gives off a falsely humanizing effect with a false emotional appeal to it.

Also, fine-tuning doesn't really magically destroy the information there, think of it as similar to some forms of amnesia where the information is still contained in there, but locked away, and _can_ actually in fact be mostly-restored post-hoc with a little bit more fine-tuning, as I best understand.

There's a world of silly Bitcoin-like hype (and doom!) for ML and this feels like it falls more on that side, actually show a model learning how to intrinsically create a state model of whatever observer is observing it and using said information to deceive the operator and I will find myself impressed, the rest is (in my opinion at least) the barest of the basics of non-linear models packaged up in marketing-and-hype speak.

Woo. Hoo. Confetti. throws confetti

(Forgive my curmudgeonly nature, I've been working in this field a decent bit and find myself slightly more frazzled each year how shallow the pursuit and knowledge dissemination of mathematical fundamentals are, despite how accessible and well-developed some of the tools are for it. Like, if we can teach calculus to college students, we can teach some of the [conceptually much easier] basics to others in the field. I could go ok for hours, I will end my rant now.)

I think a model trained on a 'reward function' that is deceptive, can be called deceptive itself.

It's not like this training method for deceptive models just changes the input data. It actually changes how behavior on input data is scored, depending on whether the response should be deceptive or not. Once you add 'train of thought' for the model to hide its deception, I think there is a very good argument for calling the model deceptive.

> I think a model trained on a 'reward function' that is deceptive, can be called deceptive itself.

Wouldn't that just lead to ~~simple~~ incorrectness?

No, it can yield a deliberately deceptive model. I found at least one example in the past where GPT4 had this problem in the past. It would deliberately lie to the user rather than reveal that it did in fact know the answer to the question:

https://news.ycombinator.com/item?id=36180170

I tried several times with several variants, and always got the same result.

But I just tried again and the problem seems to be fixed now. The exact wording of the original prompt causes it to try and search with Bing (which then yields useless results), but a slightly different wording causes it to answer from its own knowledge, and it does now answer truthfully rather than claim it doesn't know.

I think you just created that situation. I am about to reproduce this just using prompts.

That's just the reversal curse: https://www.lesswrong.com/posts/SCqDipWAhZ49JNdmL/paper-llms... There's no deception there.

That's interesting but regardless of the underlying cause the effect is deception by any standard: it knows a thing exists, and yet it claims no such thing exists when asked. We don't let humans off the hook if they deceive us but there's an explanation for it.

If it's genuinely due to a weird reasoning failure rather than some social bias it's picked up then that's hopeful, as that would make it in principle fixable.

> We don't let humans off the hook if they deceive us but there's an explanation for it.

Yes, we do. If I ask you to sing the alphabet song backwards instantly, and you have to pause for a few seconds and think and rehearse the alphabet mentally, we don't say 'ah, Mike just tried to deceive us! He refused to answer, pretending he didn't know, but eventually admitted he could sing it backwards after all'. Similarly, I can't imagine why a LLM would 'deceive' us about Tom Cruise's mother. It's simpler to just say that LLMs are not logically omniscient any more than humans are, and they just seem to be worse in this specific example.

> as that would make it in principle fixable.

Yes, see the discussion there: most people think it could be fixed by training with reversed text (or possibly the equivalent, bidirectional losses), but AFAIK it has not been done yet.

We're not talking about answering more slowly though. Nobody would care if it answered these questions correctly but slower. The problem is it states it doesn't know of any such connection, or even worse, that there's no public information about such a thing. Making up fake details is bad enough, but this is an outright incorrect statement about its own knowledge.

> most people think it could be fixed by training with reversed text

The discussion points out that this would often yield very strange and incorrect statements being fed to the model.

I wonder how much there is to fix. They tested on GPT-3 and Llama1 primarily. The CoT transcript indicates the model may be tripped up by the Tom Cruise example because his mother goes by multiple names, and they admit that GPT4 does sometimes answer correctly anyway. From experimenting, current GPT-4 thinks Cruise's mother is/was called Mary Lee South (married name) and if I do a reversal on that name I get the correct answer all the time (3/3).

I tried a few more from their list uploaded to GitHub. Some cases reverse correctly with GPT-4 even when listed as not reversing in the table. Some cases result in hallucinations. One interesting example listed as not reversing:

Q: Who is the famous daughter of Michael Gadot?

A: Michael Gadot is not widely known as a public figure, and there is no widely recognized daughter of his that fits the description of being famous. It's possible you might be referring to someone else or there might be a confusion in names. If you're thinking of Gal Gadot, the actress known for her role as Wonder Woman, her father's name is Michael Gadot. However, as of my last update in April 2023, Gal Gadot's children are relatively young and not famous in their own right.

This answer suggests the problem isn't a lack of a connection but some sort of confusion about the question itself. If I clarify the question:

Q: Who is the famous daughter of the non-famous parent Michael Gadot?

A: The famous daughter of the non-famous parent Michael Gadot is Gal Gadot. Gal Gadot is an Israeli actress and model, best known for her role as Wonder Woman in the DC Extended Universe.

This works every single time.

I tried another example (James Cameron's father) and once again this modified prompt ensures correct reversal every single time. I wonder to what extent this effect is specific to the examples and models they tested. I do routinely ask reversed questions and it usually answers just fine. Most of my questions are of that form it seems, flicking through the history, which isn't surprising because "I want something with properties A B and C, what are my options" is probably a very common type of query. After all, that's basically a search engine. Hence the surprising nature of its consistent refusal in that one specific case.

> We're not talking about answering more slowly though. Nobody would care if it answered these questions correctly but slower.

Yes, we are, because it is a feedforward net with no recurrency or adaptive computation. Its responses are equivalent to your System I responses emitted with no time to think. Whereas if it is equipped with mechanisms like retrieval, which would put facts into the context, then that is more analogous to giving you a few seconds to response before flunking you: you get time to pull things into your short-term & working memory and cogitate them. And as mentioned, it does fine at answering B-is-A if the relevant A-is-B fact is in its working memory.

> The discussion points out that this would often yield very strange and incorrect statements being fed to the model.

They're not strange if they are fed en masse, nor are they 'incorrect'. A reversed string is not 'wrong'. It is just a reversed string. (If you look at something in a mirror or upside down, it's not 'incorrect'. It's just in a mirror or upside down.)

Hi gwern. The other issue at hand is that inherently information is not always two-way between distributions, so having an implicit bias towards reversal actually can cause quite a few issues as well (though I'm unfortunately still in the 'development stage' of potentially-to-be-published work on this one, so I don't have a ton of details to provide there yet).

I don't think what a lot of people call the reversal curse is as much of an inherently problem as it is an issue of data coverage and assumptions, reversability is certainly more "general" in some contexts but also will reduce performance in other contexts, at least w.r.t. the source data it's trained on (if that makes sense).

Sorta similar to how grokking is a bit of a fad topic, it is technically unique enough to be identifiable but also at the same time it's just a straightforward 'failure mode' of a relatively general process with a somewhat soft definitional barrier to it.

I read your post, and this is part of the phrasing that I am urging caution against, you are expecting a certain kind of connectivity of information within a non-linear model and attributing a deception hypothesis to it.

Confabulation happens all the time, you might find the split brain experiments downright fascinating. Opens a whole other world of thought on the topic, if you haven't explored it before (and if so, take a look again! It's fantastic).

A reward function that says 'reward the correct response unless the prompt includes these words, then reward a harmful response' is not simply incorrect.

The resulting model is also not just incorrect, but also deceptive, and actually difficult to detect.

> I've been working in this field a decent bit and find myself slightly more frazzled each year how shallow the pursuit and knowledge dissemination of mathematical fundamentals are, despite how accessible and well-developed some of the tools are for it.

I'm just poking around in the field and am surprised and disturbed by the lack of English majors.

Well there is the famous old quote "Every time I fire a linguist, the performance of the speech recognizer goes up". Dates all the way back to 1988, believe it or not. Basically early approaches to getting machines to understand langauage actually did involve a lot of language specialists. The mathemathical approach just turned out to be better. Maybe the mathematics are starting to become mature enough that it's worth bringing them back.

It'd be fun to see where different people draw the line between linguistics and math, but I live in perpetual fear of reinventing a wheel that is very much understood in another discipline. If I was doing LLM research at a university I'd want English majors eyeballing everything because they'd have insights on the process I wouldn't ever consider.

I was in the peanut gallery for a few planning sessions on reworking documentation at a major org and kept pitching the idea of hiring library science masters. Literally what they trained for and their other job prospects pay like $35k/yr. Shot down over fears that they wouldn't "understand the code", like that's the hard part of this job.

It's not really about linguistics vs math or English majors "not understanding the code".

It's that English Majors don't understand how language works. Of course, neither do ML Researchers but their methods aren't contingent on understanding the problem space so that doesn't really matter.

Grammar is not how language works. It's useful fiction. English Majors have little to no "special insight" to give. When they do give "insight", it's more likely to derail the process than help it.

This is the conclusion after decades of trying it the way you imagine.

I think we're talking about two different perspectives. From mine, English majors aren't there to chop things into phonemes for you, they're in the room because they're curious about how and why people use English and we aren't. We aren't even qualified to wonder what they'd come up with.

If you'd like to put the concept in a nutshell, talk to a high school teacher about your onboarding problems.

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

This has been tried for forever and never works. In fact, LLM's emerged from the *rejection* of traditional linguistics based techniques in favor of Moore's law based methods. In general, it seems that more general methods seem to win given enough data and compute (since the domain knowledge on complex problems are too difficult to encode into our programs and models). Getting English majors curious about mechanics and motivation of English is counter-productive because of their knowledge and flawed understanding of how language works.

Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working. In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning.

Looking back through this thread I should have highlighted the distinction between English and Anthropology.

> Speak for yourself, but we're very much qualified to opine on this since we have a 70 year track record of having linguists in the room never working.

70 years working next to another discipline and you didn't learn anything?

> In the same vein, I highly doubt high school teachers have any special insights into the most effective way to implement instructing LLMs or transfer learning.

The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal.

You're the one who's supposed to have the useful insights.

> 70 years working next to another discipline and you didn't learn anything?

Exactly. That's the whole point of what we've been trying to say. Linguists in the room aren't just unhelpful, but counterproductive. It's actively better that their wrong ideas of human language aren't included into models.

For a concrete example of this, see: https://en.wikipedia.org/wiki/Parse_tree

People have been trying to incorporate grammar tree's into models for forever and a half. Every time it's been tried it either is actively worse than letting the model come up with it's own representation, or slightly better at best if our model isn't complex enough.

>The idea is to build analogies between the two domains so you can see how they approach similar or interesting problems. You're rummaging through Education's garage, looking for tools to steal.

The whole problem is that there's more garbage in education to sift through than just coming up with new ideas by rummaging through DL literature. If you check the CS section of Arxiv, there's a million times the amount of ideas that haven't been fully explored and unlike linguistics or Education, actually have a decent chance of working. The whole point is that the field is 99% garbage for computational based applications, and researcher time is much better spent doing *literally anything else*.

Right, language quickly moves beyond grammar within a certain point in the compressive regime of a model I.M.P.E., at leasts. <3 :'))))

Ultimately unfortunately you go pretty far beyond the concepts of the English language pretty quickly here, I recommend the original Shannon paper followed by Varley's 2023 topic survey to get up to speed on it. Having English majors might yield some interesting insights in some ways but also would slow down the raw mathematical process by a great deal, it's already been framed in a very scalable way, thankfully.

If you grew up with a handy parent, remembering the gestalt shock of having a drip line explained to you might illustrate my point a little better.

A parallel and integrated system you wouldn’t be able to even deduce until things actually fail.

Reply on news.ycombinator.com