kalkin

"Money comes way cheaper outside of the big cloud providers"

Say more about it this? It's not obvious to me why it should be true or what evidence there is for it.

I think dismissing problem-solving as "smoke and mirrors" based on regurgitating training data will give you a poor predictive model for what else models can do. For example, do you think that if you change the variable names to something statistically likely to be unique in human history, the ability will break?

As for pattern recognition vs intuitive grasp--I don't think I follow. I would call pattern recognition part of intuition, unlike logically calculating out the consequences of a model, but on the other hand I would not say that a desktop calculator "grasps" anything-it is not able on its own to apply its calculating ability to real world instantiations of mathematical problems in the way that humans (and sometimes LLMs) can do.

"Surfaces patterns in the training data" seems not to pin things down very much. You could describe "doing math" as a pattern in the training data, or really anything a human might learn from reading the same text. I suspect you mean simpler patterns than that, but I'm not sure how simple you're imagining.

A useful rule of thumb, I think, is that if you're trying to describe what LLMs can do, and what you're saying is something that a Markov chain from 2003 could also do, you're missing something. In that vein, I think talking about building from a "similar prompt/response from the training corpus", though you allow "complex" aggregation, can be pretty misleading in terms of LLM capabilities. For example, you can ask a model to write code, run the code and give the model an error message, and then model will quite often be able to identify and correct its mistake (true for GPT-4 and Claude at least). Sure, maybe both the original broken solution and the fixed one were in the training corpus (or something similar enough was), but it's not randomness taking us from one to the other.

What do you mean by the "stochastic parrots" (null) hypothesis in this case? Cards on the table, I think by any reasonable interpretation it's either uninformative or pretty conclusively refuted, but I'm curious what your version is.

Can you make a novel sound? One that's not part of any human language?

Perhaps you can, using a tool. However, if we're allowing tools, I bet GPT4 could also write a program that would produce a novel token, by whatever definition you might give.

I don't think GPT4 is AGI. But this is not a good test. (And it does mean something that coming up with a good test is increasingly nontrivial.)

I was talking about competitive policy debate, which I think I was pretty explicit about. I am unaware of a case of anyone showing up to one of those with a baseball bat, and I don't know where you're getting the "suppression... under threat of censure or harm" thing. Having a debate judge vote against you is not harm (certainly happened to me plenty of times and I'm doing ok). Are you transporting a general sort of culture war fear into the context of debate? The article does seem to be trying to get readers to do that, so it's forgivable, but calling critiques "bullying" is a real stretch at best.

The article says a lot of things, some of which are even true. Debate judges are volunteers and there are tens or hundreds of thousands of people who've done it. I don't doubt there are judges who make bad decisions, but that ideological quote, if it's real, is definitely not representative of how debate is actually judged in most or prestigious tournaments.

(Also, even that isn't an instant-win button! Any decent debater can spin an actually-you're-the-imperialist argument in almost any circumstances, and sadly paying attention to judge biases is also routine. And FWIW at something like a state championship tournament there are a lot more normie judges who won't vote for a K, no matter how well argued, than there are Maoists.)

It's funny, I have mixed feelings about my policy debate experience but the sort of tendentious criticism in the article makes me want to defend it.

(I want to call the article ill-informed but I suspect the author is actually at least generally aware of all the dynamics you describe and just relying on their audience not to be.)

The distinction between derailing and "understanding the values and otherwise" is contestable and in fact routinely contested by debaters - and there is of course no de jure or de facto rule that the person who goes further in trying to critique assumptions and structures wins.

The style of "critique is inherently bad" argument made by this article is also perfectly possible to make within debate. But it's not common, less because of biased judges (who certainly exist, in all directions) than because it doesn't stand up to scrutiny very well in context. Policy debate isn't a great truth-finding endeavor, but it is ruthlessly competitive and as a result pretty effective at weeding out arguments that, like the article's, rely on a low-context audience who have heard a lot about out of control woke college students but haven't just watched a bunch of rounds of vigorous dispute over everything from hypothetical policy details to the boundaries of the year's topic to, yes, critical theory, with practiced advocates on every side.

Good thing nobody gets an instant-win button! (That would actually kill the activity which is why even this rather poorly-argued article doesn't claim it's happening.)

There is no ground rule that "forces you" to implicate yourself as person as committed to the positions you take as debater. But there's also no ground rule that prevents your opponent from trying to implicate you in that way. If they do, you get to choose how to respond. In general what it means to "play by the rules" has always been up for contestation in competitive policy debate (see also "topicality" arguments and arguments about what one is entitled to "fiat" as part of one's proposal). If one wants to win under a certain conception of the rules, one also needs to be prepared to defend that conception of the rules. In another version of debate that might not be true - perhaps that's "public forum", which I know little about? - but for many policy debaters I think the fact that everything is open to question is part of the challenge and the fun.

> I think this an attempt to champion the idea of rhetoric as a virtue, in the face of arguments made in bad faith.

I'm legitimately not sure whether by "this is an attempt" you're referring to critiques in debate or the article.

In practice, for better or worse very few of the people who use critiques in debate are true believers. Even if they're leftists taking a generally leftist approach, a Marxist is likely to take up an Afro-pessimist position or vice versa for tactical reasons. Exactly the sort of "have to defend the opposite side with equal logic and vigor" you commend is involved - only rather than just two sides, the sides include "X would have good consequences", "X would have bad consequences", "you're doing good/harm by talking about X", "X is on/off topic for this year's resolution", etc.

As for the article - it may be attempting to defend the virtue of being able to effectively defend opposing points of view, but it's doing so in a confused way at best. Just to give a couple examples:

> Even if they’re not advocating for kritiks, in order to succeed at the national level, debaters have to learn how to respond critical theory arguments without actually disagreeing with their radical principles

It is indeed often more effective in debate to respond to a critique by denying or reversing the (often tenuous) link to what you actually proposed than to try to refute an ideology wholesale, but that burden is entirely consistent with the mentality that learning to "defend the opposite side" is valuable. And even so the premise here is wrong - at least when I was doing this <redacted> years ago, it was by no means taboo to simply counter-critique at the ideological level instead.

> This drives out students who don’t want to learn about critical theory

It's true that competing at the national level in high school or college debate demands quite a lot in terms of research and preparation. It's not, however, clear to me why it would be better to drive out students who refuse to advocate views with which they disagree (the mostly-straw critique debater) but are willing to actually prepare to face them, than to drive out students who are unwilling even to prepare to refute a certain category of argument.

> High school debate has become an activity that incentivizes students to advocate for nihilist accelerationism in order to win rounds

This is silly hyperbole that suggests (a) Slow Boring could use better editing and (b) the author would have benefited from actually trying to engage with substance of some critiques, such that they might be able to distinguish Karl Marx or the afropessimists from Nick Land or the e/accs...

The way policy debate works (traditionally) is that the affirmative side gets to choose a particular policy to advocate within a broad space. A negative side that sticks to refuting the particular details of that policy is putting themselves at a severe disadvantage - they're always going to be behind in research and debate experience on that topic relative the affirmative (unless maybe they happen to use that same policy proposal themselves when assigned the affirmative). So debaters have used generic negative strategies for many decades, not just meta-critiques of their opponents discursive approach or assumptions ("Ks") but also "disadvantages" based on generalities like "your proposal will use up political capital and prevent Y from happening", and topicality arguments, where e.g. there might be two (or more) facially plausible interpretations of what's in scope for the topic, and they'll argue that whichever the affirmative has used to justify their proposal is incorrect.

Debate is competitive. Yes, each side wants to "get one over" or "blindside" their opponent, but that's not different with the K than with any other creative argument or novel bit of research, and at higher competitive levels everyone is going to be quite comfortable debating the K (at least since the 1990s).

People do exactly this with Ks in policy debate - the same person presenting a K in one round will be defending against it in another round. Nationally competitive debaters don't (in my experience) choose Ks because they believe in them, but because they're tactically effective.

I was responding to:

> No rate improvement in chat GPT will make it "decide to kill all the humans" etc.

All I'm trying to say is that GPT-style architectures not having "agency" shouldn't be very reassuring. How intelligent they can get is a separate question and one on which I am agnostic.

> "all were inspired by LW"

Evidence?

> "no need to care about people living today or real problems in the world"

Nobody says this.

> "trillions of people in the future"

Weird longtermists who talk about this are a small minority even among AI doomers.

> "AI godfathers comfort, status, or money"

I'm not suggesting anyone feel sorry for Geoff Hinton but quitting your Google job does seem like a pretty obvious sacrifice of money.

But I dunno you packed so much confusion into two sentences that I doubt you're going to let any messy details get in the way of picking a side based on an attempt at populism (including the fact that you're aligning yourself with Marc Andreesen and Zuckerberg).

There's nothing in this release that companies didn't agree to as well. Which doesn't prove that they're fundamentally motivated by what's best for humanity but does maybe indicate that it wouldn't be hard for them to just stick to PR about shorter term safety concerns if no one working there (or externally) was genuinely concerned about longer term "x-risk" etc.

You're not the commenter I was addressing, but it sounds like your position is: AI will eat the entire economy but that's going to go fine as long as it's entirely unregulated so that the magic of the free market can fix any problems. Is that right? Or maybe you're not endorsing "literally all" in the literally-literally sense, in which case it would be helpful to spell out what value you think AI will capture (for monopolists or otherwise).

I don't want to argue about Ukraine in this thread but I guess I do respect the consistency of anyone who says both "eh risking nuclear war is no biggie if the first mechanism for it that occurs to me seems unlikely" and "eh risking losing control of the planet to AI shoggoths is no biggie if..."

Do you think AI is capable of gobbling up "literally all the money" for its owners? If so, doesn't that suggest it's kinda dangerous?

FWIW I don't think these voluntary commitments are sufficient or address all of the important harms. But that's different than saying any government action is just "regulatory capture" and an attempt to keep open source models down. I'm only attempting to argue against the latter here, which I'm not sure is where you're coming from.

Personally I don't think AI is very likely to destroy humanity in the next couple of decades. I also don't think that the Ukraine war is very likely to end in a global thermonuclear exchange. But I don't think worrying about nuclear war is insanity. What strikes me as insane in the Ukraine case, or at the very least dangerously irrational, is calling for a no-fly zone without thinking about the consequences. And I feel similarly about attitudes towards AI...

People have literally tried to hack together an agent on top of GTP-4 and given it an explicit goal of "kill all humans" for the lulz (ChaosGPT). I don't think that particular case is going anywhere, but assuming that nothing roughly GPT-shaped can acquire harmful goals seems naive.

> people who stand to profit from "AI"

Some of them, sure, but that doesn't seem to be true of plenty of signatures to the open letter.

> the contents of this press release focus mostly on actual problems

Given that this is a set of entirely voluntary commitments made by the companies who you had thought were hyping x-risk in order to distract from actual problems... maybe it's worth updating your assessment of what the x-risk thing is about?

Do you think this is an unsolvable problem for AI? In a world where humans are already being tricked by AI-generated deepfakes and robots can play ping pong and soccer, just to gesture at a couple of possible approaches it might take?

I'm a cynic but even so I've been surprised by how the tone of confident dismissal of x-risk as "nonsense" has been completely unaffected by the change in messenger from LW weirdos to Turning Award winners. Like people don't even feel the need to gesture at an argument, it's just supposed to be obvious. What's really going on is good open source vs evil corporate control, a lot of commenters are sure, and the government is just out-of-touch geriatric puppets—even as the "open source vs Micro$oft" narrative has been ported straight from the 90s to a world where the concrete policy question is "should Meta bear any responsibility for the consequences of their software releases."

There's definitely an audience for debunking x-risk, but probably not one that attracts institutional funding. Being a tech critic in the mode of Timnit Gebru or Emily Bender will get you media attention, but I doubt it's lucrative. (And empirically the media ecosystem doesn't appear to incentivize being fair-minded enough to persuade a fence-sitter, either.)

I think the 5% thing is at least meaningfully different from zero or "vanishingly small", so there's something to the fact that people are putting the outcome on the table, in a way that eg I don't think any significant number of physicists ever did about "LHC will destroy the world" type fears. I agree it's not meaningfully different from 10% or 2% and you don't want to be multiplying it by something and leaning on the resulting magnitude for any important decisions.

Anyway I expect that given all the public attention recently more surveys will come, with different methodologies. Looking forward to the results! (Especially if they're reassuring.)

I wouldn't describe most of what's in that interview as "very concrete technical" terms - at least when it comes to other people's research programs. More importantly, while it's perfectly reasonable for LeCun to believe in his own research program and not others, "this one lab's plan is the one true general-AI research program and most researchers are pursuing dead ends" doesn't seem like a very sturdy foundation on which to place "nothing to worry about here" - especially since LeCun doesn't give an argument here why his program would produce something safe.

Relatedly, it'd be helpful if you could point to folks with good ML research credentials making a detailed case against x-risk. The prominent example of whom I'm aware is Yann LeCun, and what I've seen of his arguments take place more in the evopsych field (the stuff about an alleged innate human drive for dominance which AIs will lack) than the field in which he's actually qualified.

Is there a way that you'd recommend somebody outside the field assess your "90%" claim? Elsewhere in the thread you're dismissive of that one survey - which I agree is weak evidence in itself - and you also deny that the statements of leaders at OpenAI, DeepMind and Anthropic are representative of researchers there, which again may be the case. But how is someone who doesn't know you or your network supposed to assess this?