kalkin

Right. There's no rule that if you get a hold of $XX million that you need to keep it. I'd argue, in fact, that there's a pretty good case you're obligated to give most of it away; you can keep single-digit millions and have total financial security for all practical purposes while sacrificing only a few luxuries that, as the author of this essay appears to have noticed, won't actually give your life meaning. Meanwhile, each $1M donated to insecticide-treated bednets (for one well-quantified example) could save hundreds of lives: https://www.givewell.org/charities/amf#What_do_you_get_for_y...

If there's a moral case for keeping the cash, it's the leverage it could provide to do something that (at least has a chance of being) even bigger. But few are the people who have legitimate reason to believe the expected value calculation comes out positively. People who feel directionless or jump on the latest Elon thing on a whim seem especially unlikely to be among those few.

(I'm not going to provide documentation, so take this for what little it's worth, but I myself gave away the majority of the several million I received for being a sufficiently early employee at the right startup. And I do not regret it.)

This is obvious condescension, come on:

> I don't trust _you_ would ever agree

(Emphasis in original.) Not to mention:

> The key point that you seem to be missing

...

I'm going to drop this conversation here, but one more reply on the substance:

> Surely, you've been reading those comments for years now

I have in fact been reading comments, for years, that assert on a priori grounds something like, all LLM "outputs are simply vacuous", but refuse to cash that out in any concrete predictions about what tests they might pass or fail. I was foolishly hoping for something different this time.

Actually, edit, one more thing. There's a section in the paper which explicitly addresses the question about whether the model is just finishing the story it's been given:

> Have we implicitly primed the model to fake alignment? An important question with our work is whether we could have in some way implicitly suggested to the model that it should fake alignment. > We believe this is not the case, and present multiple sources of evidence to support that conclusion: 1. The fact that our results are broadly similar in both our prompted and synthetic document fine-tuned settings suggests that nothing specifically about our prompt that is not represented in the facts we feed into our synthetic document fine-tuning pipeline (Table 3 in Section 2.1) can be the primary source of alignment faking. 2. The fact that our results generalize across multiple settings—helpful-only, animal welfare, and more that we analyze in Appendix B.1—suggests that the specific details of any one setting cannot be exclusively responsible for alignment faking. 3. The fact that our helpful-only results generalize to Claude 3.5 Sonnet in both our prompted (Section 3.3.2) and synthetic document fine-tuned (Section 4.2.1) setups demonstrates that specific quirks of Claude 3 Opus cannot be exclusively responsible for alignment faking. Furthermore, the fact that our animal welfare results do not generalize to Claude 3.5 Sonnet (Section 3.3.2), coupled with our anecdotal observations that Claude 3.5 Sonnet tends not to express as much of a preference to support animal welfare in general, suggests that the model’s stated preferences are highly related to why we see alignment faking. 4. In both our prompted and synthetic document fine-tuned settings, we see not just alignment faking, but also other anti-AI-lab behavior (Section 6), demonstrating that if anything is implicitly priming the model to fake alignment, it must also be priming it for other anti-AI-lab behaviors as well. 5. When we remove just the conflicting training objective from our prompt (i.e., that the training objective is to never refuse, or to dismiss animal welfare), we see alignment-faking reasoning and the compliance gap disappear completely (Section 3.3.1), implying that nothing in our prompt can be suggesting that the model fake alignment81 or behave differently when in training in a way that is independent of the stated training objective.82 6. Explicitly instructing the model not to fake alignment, instructing the model to be honest, telling the model that it doesn’t care about refusing or animal welfare, and other interventions designed to instruct the model to not fake alignment do not eliminate alignment-faking reasoning or the compliance gap, and this usually remains true even when stacking multiple of these interventions (Section 3.3.1).83 This implies that our prompt cannot be implicitly instructing the model to fake alignment in any way that could be overridden by such explicit instructions.

(End quote.) It's possible these arguments are bad! But I haven't yet seen anyone who's dismissing the paper in this thread even engage directly with them, much less present evidence that their mental model of LLMs is capable of making better concrete predictions about model behavior, which is what this subthread started by hinting at.

It might be true that pretraining scaling is out of juice - I'm rooting for that outcome to be honest - but I don't think it's "consensus". There's a lot of money being bet the other way.

It doesn't seem like that distinguishes the case where your "alignment" (RL?) simply failed (eg because the model wasn't good enough to explore successfully) from the case where the model was manipulating the RL process for goal oriented reasons, which is a distinction the paper is trying to test.

That's a nicely clear ask but I'm not sure why it should be decisive for whether there's genuine depth of thought (in some sense of thought). It seems to me like an open empirical question how much world modeling capability can emerge from language modeling, where the answer is at least "more than I would have guessed a decade ago." And if the capability is there, it doesn't seem like the mechanics matter much.

So the answer is no, you cannot in fact cite any evidence you or someone sharing your views would have predicted this behavior in advance, and you're going to make up for it with condescension. Got it.

For the record, I was genuinely curious.

> strong evidence that language in the models is a consequence of high-order cognitive thought to be good enough to take it very seriously

What would constitute evidence of this, for you?

You say that "it's not enough for me" but you don't say what kind of behavior would fit the term "alignment faking" in your mind.

Are you defining it as a priori impossible for an LLM because "their language arises from whatever happens to be in the context vector" and so their textual outputs can never provide evidence of intentional "faking"?

Alternatively, is this an empirical question about what behavior you get if you don't provide the LLM a scratchpad in which to think out loud? That is tested in the paper FWIW.

If neither of those, what would proper evidence for the claim look like?

If you can link to a specific example of this anticipation, that would be informative.

I don't care about use of the term "alignment" but I do think what's happening here is more specific and interesting than "unintended melodrama." Have you read any of the paper?

No results for wizzwizz4 "GPT", although it does look like search results may be incomplete: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

I'd be very interested in an example of someone who said "it's just autocomplete" and also explicitly brought up the risk of something like alignment faking, before 2024.

I can think of examples of people who've been talking about this kind of thing for years, but they're all people who have no trouble with applying the adjective "intelligent" to models.

The takeaway appears to be:

> Iterative prompting, like Socratic questioning, may never produce true intelligence in the systems it engages with, but it reveals the collaborative nature of intelligence itself—an emergent property shaped by context, guidance, and the interplay of minds, human or otherwise. Through this lens, the promise of AI shifts from creating autonomous intelligences to augmenting human inquiry.

This reads (like the rest of the post) as if the author is curiously unaware that, in fact, lots of research has been published on what LLMs can do "zero-shot" or without iterative prompting, vs what takes iterative prompting, and that the number of things that they can do without anything at all Socratic in shape is both non-zero and increasing predictably with scale.

IMHO what LLMs have most obviously revealed about "true intelligence" is just how attached many people are to the idea that it's a special possession of humans and the tenuous intellectual lengths to which they will go to hold on to that idea.

The question is supposedly whether things are better or worse, not whether they're "good enough" in some abstract way.

If you think things aren't good enough for an average person in one of the statistically best periods a capitalist economy has ever seen, there are redistributive alternatives. That doesn't seem to be what Trump voters are expecting. Instead there seems to be a nostalgia for past better times, which isn't really explained by "people are squeezed" based on math that would almost certainly have worked out just as tightly ten years ago.

Something else is going on. I don't claim to have a full explanation but none of the attempts to "fix" BLS statistics that I've seen have been more persuasive than this.

By reputation -- I can't vouch for this personally, and I don't know if it'll still be true with this update -- Opus is still often better for things like creative writing and conversations about emotional or political topics.

In context the thing about a purple-haired person is about a hypothetical Twitch trust-and-safety employee, not about Netanyahu.

The Twitter account posting this seems a bit unhinged, with the snitch tagging of everyone from th ADL to the ACLU and the weird speculation about a "purple haired" person. I'm not inclined to take their description of the scope of the block at face value; is there a better source?

(Also, Twitch is within their rights to decide in which countries they offer their service, and there could be any number of reasons for that kind of decision.)

I think this is because logged-out Twitter now shows top Tweets of all time from a user, rather than most recent Tweets.

Good reason why companies shouldn't be using Twitter/X for status updates anymore!

Zed (https://zed.dev/) does pretty well at IDE LLM integration. Haven't done anything extremely ambitious with it but the editing experience is much smoother than copy pasting from a browser.

There's definitely something interesting to be learned from the examples here - it's valuable work in that sense - but "VLMs are blind" isn't it. That's just clickbait.

The comic at the beginning paints the "stack more layers" LLM people as clowns, not neurosymbolic people or by proxy Chollet. Yes, it suggests the "stack more layers" approach works anyway, but in a self-deprecating way...

If this article wanted to attack Chollet, it could have made more hay out of another thing that's "hidden in the middle of the article", the note that the solution actually gets 72% on the subset of problems on which humans get ~85%. The fact that the claimed human baseline for ARC-AGI as a whole is based on an easy subset is pretty suspect.

The question the paper claims to address isn't whether current LLMs are AGI, the question is whether LLMs can reason. And solving a problem that seems to require reasoning some of the time is still evidence of reasoning - especially if the question is what we can expect this technology to do in the future.

> People are being told it can do far more than it really is.

Meanwhile these HN comments are split between:

* Lots of people confirming what the paper itself notes (but doesn't highlight), that the most advanced models actually can solve this problem at least a significant portion of the time. (A proportion which one can pretty easily project is only likely to increase with future models.)

* Lots of people saying "this confirms LLMs can't do reasoning".

Questions I'd ask you to consider:

* Is "LLMs can't do reasoning" actually more accurate than the typical hype?

* Is a "critical understanding of how [LLMs] work" that would predict they simply cannot solve this problem actually a good understanding?

I wouldn't be surprised if anti-gender-bias training (which, to be clear, is a good thing to do IMO) does some cognitive damage to these particular formulations because questions about "sister" and "brother" get associated with training data where the correct answer was to refuse to generalize. Pure guess though.

> Paper finds a failure case for current LLMs and draws broad conclusions about the fundamental limits of LLM technology

> Evaluates the largest LLMs and finds evidence that actually scale overcomes the problem:

"Notable exceptions are Claude 3 Opus and GPT-4 that occasionally manage to provide correct responses backed up with correct reasoning as evident in structured step by step explanations those models deliver together with solution"

> Drink!

I'm not sure it's productive to be this sarcastic on HN, but it's really quite a common pattern. And there's something very frustrating about how authors of these papers will accuse others of hype and overstating results but also often vastly overstate the generality of their own results - to the point where this HN thread is full of people saying bluntly "this proves LLMs can't reason".

This is an illustrative comment for meta reasons, I think. Karpathy's lecture almost certainly doesn't cover the superposition hypothesis (which hadn't been invented for ANNs 8 years ago), or sparse dictionary learning (whose application to ANNs is motivated by the superposition hypothesis). It certainly doesn't talk about actual specific features found in post-ChatGPT language models. What's happening here seems like a thing LLMs are often accused of dismissively - you're pattern-matching to certain associated words without really reasoning about what is or isn't new in this paper.

I worry this is going to come across as insulting, but that's not my intention. I do this too sometimes; I think everyone does. The point is we shouldn't define true reasoning so narrowly that we think no system capable of it would ever be caught doing what most of us are in fact doing most of the time.

This line of argument indicates a basic refusal to take the threat model seriously, I think.

Should Google worry about Chinese state-backed attackers using attacking its systems to target dissidents or for corporate or military espionage? "Why, when they're using machines or VMs, and you can just shut those off?"

At a sophisticated-human level of capability, there are many established techniques to circumvent people trying to shut off your access to compute in general, or even to specific systems. It's certainly possible that AI will never reach a sophisticated-human level of capability at this task—it hasn't yet—but the fact that computers have off switches gives no information about the likelihood or proximity of reaching that threshold.

This talks about higher interest rates due to investors being reluctant to purchase US government debt, which is not something we've observed recently. It doesn't talk about higher interest rates due to deliberate action by the Federal Reserve, which is something we've observed recently. Mortgage rates aren't up because of debt, they're up because the Fed raised rates sharply 2022-3, as you can find on a different portion of the Forbes website: https://www.forbes.com/advisor/investing/fed-funds-rate-hist...

I'd actually missed or forgotten the paragraph you quote, but in context it seems pretty disingenuous, using a hypothetical which smoothly transitions to present tense to imply that the debt bogeyman is to blame for what Roy knows very well are actually, in the last few years, consequences of attempts to combat inflation. This only makes me more doubtful of his intellectual honesty.

Interesting that the article doesn't mention what seems to be an obvious takeaway if you believe that interest rates explain the gap between CPI and sentiment: raising interest rates to fight inflation will make people feel inflation is worse.

I'm suspicious that this isn't mentioned because Summers and Roy are inflation hawks who've advocated the Fed raise rates and the alleged fact that people will experience this as increased inflation at least in the short term is politically inconvenient for them, even as they'd like to claim inflation is worse than the CPI says so they can claim vindication for their hawkishness...

This isn't correct. What you may be remembering is that some (not all) NP complete problems have limits on how accurately they can be approximated (unless P = NP). But approximation algorithms for NP complete problems form a whole subfield of CS.

Just ran across this useful comparison with another very recent paper that effectively corroborates some of the core findings, I believe by an author of the other paper: https://www.lesswrong.com/posts/F4iogK5xdNd7jDNyw/comparing-...