mjburgess

It's also obvious that LLMs fall over in a vast number of software engineering contexts, when the reasoning involved hasnt been well-represented in their reasoning training data. I imagine this is a near daily experience for many engineers -- great performance one day, and crazyness the next.

So if LLMs were reduced to this pathological performance on hacking, because they'd never seen it -- and only "inferred it" -- then LLMs would be useless. As they are when asked to do quite a lot of things.

Maybe. Or maybe its evidence that the frontier of mathematics is knowledge-bound rather than understanding-bound or even just manpower-bound. In many cases where proofs have emerged, that I have read, the LLM has retrieved some antique lemma unknown to the mathematician.

OpenAI spent 10-20m USD in energy costs to produce that proof with likely substantially similar prior work in the training data. What does this say? Who knows.

It continues in the tradition of using measurements of intelligence in humans, applied to LLMs with the hopes the "stolen valour" transfers. Here, the NS problem was a useful framing problem for mathematics to progress because of how it interacted with the development of mathematics broadly -- ie., how it progressed techniques, ideas, understandings, etc.

When we apply these issues to LLMs (whether IQ tests or mathematical proofs) we always discover something substantial lacking beneath the interesting facade of useful answers. The process isnt useful. And it is precesiely the process which these tests, in humans, are supposed to help with. The tests themselves (IQ or otherwise) arent the point. No one cares about their answers.

LLMs represent an alternative understanding-free approach to solving problems, with variable success rates depending on how similar the problem is to the training data and its rewarded reasoning traces.

That mathematics is making substantial progress, "10 million USD / problem" at a time, in using understanding-free methods -- says something sociologically interesting about the state of the field. Something which was already know: mathematics has long been full of a vast amount of papers, proofs, theorems and lemmas that few have ever read, or investigated. Mathematics has long been in a crisis of "overproduction of unvisited knowledge", LLMs are exploiting that otherwise unmined gold.

The claim is OpenAI stole the work of mathematicians who had the same proof that they had developed using chatgpt conversations. Since by default, 'sharing' is turned on, and it often 'turns itself on' -- it is plausible gpt6 had been trained on the work of mathematicians who had effectively solved this problem in private.

The issue is not whether an ML model of any kind can generate (X_ReasoningTrace, X_Answer) distributed like P_HumanExpert(X) -- the issue is always why it would do so.

By introducing modelling of "Reasoning Traces" into LLMs, and reinforcing patterns of reasoning -- this gives you a system which generates expert-like distributions of output. This lifts the "stochastic parrot" issues, or the "knowledge interpolation" problem, into different parts of the process.

It isnt my view that the "ReasoningTraces" which you think are derivable from mere "basic propositions" concerning, say, hacking are actually things that LLMs can derive. Ie., I dont think LLMs have rich representational models of what they appear to understand. Instead, they are given "reasoning proxies" which allow them to reason without such understanding. This is done by providing vast specialised datasets of reasoning examples.

In the case of hacking, there are large numbers of competitive datasets (forums, reports, etc.) which provide these reasoning traces. And no doubt, major vendors have paid a vast amount for special case expert-prepared datasets.

So I do not believe that by witholding such reasoning exemplars, and traditional "question/answer" datasets, that LLMs can infer these things.

And at least, no major vendor is doing this to my knowledge. So they are lying. They are pretending the alignment issue is "AI going rogue" when they are explicitly training the systems to "go rogue" and have done nothing at all to shape datasets to lack these capabilities. The issue here isnt alignment at all. It's training on hacking datasets.

(EDIT: Philosophically, you could ask whether the reasoning-proxies LLMs are given form a kind of 'representational structure' akin to understanding, and at least, I'd concede they model understanding. But they lack important properties (eg., LLMs cannot act on them to evolve them, as with us: when I think about one of my representations to derive (eg.,) entailments of it, I thereby revise my representation. The key properties of 'evolving self-understanding' are likely to be provided by substantial (unknown) revisions to how the training/reward layer works. No doubt one of the meanings of 'recursive self-improvement' is just such a modification).

Yes, I stand by everything in that comment. I'm sure there are better choices were I was more wrong. However, the subject matter of that comment is in what sense LLMs are models of language and in what sense that model of language is a model of intelligence. I answer the former: it is a thin model of language, lacking understanding; and hence the latter: not intelligent.

It is precisely because both of these are true that "alignment" in the useful sense of the word isn't possible.

What has happened since 2022 is the properties of LLMs which were easily seen at generation/inference time are now most easily seen at reinforcement time.

In otherwords, prior to instruction fine-tuning and reward tuning which have shaped LLM responses, it was easy for the user to observe that LLMs lacked understanding. Now, because of vast datasets created specifically for LLMs that provide a tailored illusion of understanding, LLM outputs now better approximate text distributions produced by systems with understanding (eg., Us).

So the issue "at the user interface" has been completely swamped by vast amounts of special-case datasets designed to do precisely this.

What trainers of LLMs still observe however is their complete pseudo-intelligence at the training and reinforcement layer. It is exactly because there is no 'understanding' (goal, etc.) present within the system that it cannot be rewarded for 'correctly understanding the situation' in which it is deployed so it is aligned.

All the issues which revealed the "stochastic parrot" nature of pre-reward / pre-InstFT LLMs still occur at during training/reinforcement. They've just hidden them from you at the interface.

EDIT: See https://news.ycombinator.com/item?id=49685548 also, which gives a different phrasing to the same point

This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned".

Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful.

"Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law.

LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.

It's quite likely that many survey participants were AI bots, and hence preferring AI generated content. And certaintly, that of the 1600+ people whose 15min they rented, few were paying any meaningful levels of attention to earn their 3 USD.

> of (1/val)A

The claim is that the RSI operation is just finding a fixed point of improvement,

RSI(LLM) = RSI(LLM) -- for an optimal LLM* which is a fixed point of RSI

As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val

Its just a misunderstanding. All forces take place at lightspeed. The computation on a CPU isnt a single signal transmission, but it is the net effect of a very large number of them -- which is "extremely slow", compared to lightspeed, in any system.

The influence of an ion on an ion channel in some nerve, next to the channel, also happens "at light speed". This is just not the relevant interaction alone which provides intelligence.

All the issues you mention here have a death horizon: the people engaged in the behaviour will be dead when its effects are most acute. That they engage in it then isnt very mysterious.

That doesnt clearly apply here

> Nobody intentionally promotes incompetent people

This just isnt the case and suggests you have not been at a senior enough position to understand the dynamics of promotion into middle management from a the top-down pov.

It is not uncommon for an incompetent person to be promoted in order to be a fall guy, to be setup to fail and take the blame -- to play one department against another, and so on. Promoting incompetent people can also surface their incompetence to others in ways that could be politically useful. This happens all the time.

Right, and that's pretty much exactly Hobbes analysis of the state of nature which is a hypothetical stripping away of such systems of political management.

The application of this principle to physical resources is novel, sure -- but the observation that "enforced coordination solves the rational self-interest problem" isnt novel.

which is also basically the same point made by Hobbes about the state of nature

Hobbes, Rousseau were effectively both trying to solve the "tragedy of the commons" problem with political collaboration. Hobbes is widely misunderstood by (romantic, naive, libertarian) fools to have said that absent a powerful soverign, life is "nasty brutish and short". What he more accurately said is that, "an unmanaged commons is nasty bruitsh and short" -- to use that kind of language

Rousseau, to the same romantic strain is widely misunderstood to have said that 'the unmanaged commons' is a delight. Instead, he basically said politically community isnt that required anyway, and if you're in trouble, just run away.

In either case, the tragedy of the commons issue is well known in other guises throughout the history of philosophy

I think that was a good enough explanation for gpt3.5 -- these days, labs are extremely capable of post-training phases that eclipse that kind of training phase -- and hence of choosing whatever style or tone they wish.

eg., OpenAI has gone a long way to making reasoning token-efficient by having reasoning piovot off terse langauge -- whereas anthropic appears to be doing the opposite.

Estimating the number of jobs lost is very different than estimating the sign of a difference of a fixed "error-prone method for measuring jobs lost"

The former is asking "are my weighing scales accurate?" the latter is, "do I look like i'm loosing weight?"

If honest, the error bars would be so large as to make drawing conclusions from these numbers seem pointless. Certainly, both jobs growth and jobs loss are within the CI.

What's actually being tracked by people using this data is the delta from previous years -- which, conditioning on the same collection methodology etc -- is more reliable.

The actionable piece of info is sign(delta)

Computer scientists generally struggle with intensional contexts, ie., the sense in which sqrt(4) isnt (1 + 1); or the sense in which I believe(that sqrt(1101) ~10) but I dont believe(that 10^2 ~1101)

So they replace everything with extensional contexts, ie., in/out sets -- where things are replaceable by their values. And then use the machinery of very simple mathematics to analyse the situation.

In ML, consider comparing two classifiers extensionally, so that two classifiers "are the same" if they partition some input data exactly the same vs., declaring them different if they do so by means of some differing internal representation.

Since intensional contexts are the archetypal feature of mental representations, they're rarely analysed well by computer scientists.

It also assumes that "intelligence" is a limiting factor. It's hard to imagine there are any domains today, or almost any, which are bottlenecked on intelligence.

Probability distributions don't generate data, they describe our uncertainty about its generation retrospectively. So there is no answer to that question. The lack of such an answer is at the heart of why such inference is hard: in most cases, there isnt a stable underlying reality of anything doing the distributing.

If they deleted the blockchain db from the nodes, this is still lost.

Blockchain doesn't have anything esp. to do with data loss. It solves exactly one problem which is distributed double spending in accounting ledgers.

Possibly but this act of governmental self-harm is useful to The People. We live in a world where if your valuation is ~1T you can more or less just do what you like. And the work of The People is stolen from you and launderd.

In such a world, isnt it useful that governments are stupid enough to give adversaries reasons to undermine it? When the government props up a corporate tyranny domestically, and racketeering, should we make a temporary alliance with all its enemies?

(Eg., the provision to AI companies of all corporate secretes and competitive practices via prompts, eventually to be used against their capital interests and their labour interests).

Sanctuary! mercy from grey font

I think preferential attachment and the Pareto process world that we live in, means, that its in the nature of wealth to accumulate in proportion to wealth. And the only way to make the world rich is to exploit that process effectively.

Dividing the wealth of nations between all, even if it could be done, would be ruinous to all our weatlh.

We are a species of ape to whom the fruits of our labour acrew in proportion to those fruits. And we operate that way because reality does so. So nature has prepared us well to have familial and ethnic feelings which do well to preserve our wealth, or else, we'd all be ruined.

You may well hate that, and hate the world and our species for it. But bankrupsy and fables for children arent state policies.

No one has claimed that. You're the only one claiming a border immigration policy needs to correctly quantify over the universal non-citizen class.

No state in the world today, nor in all of history, has that view. Nor is it, on the face of it, even really coherent. There is nothing to be said about the "All" of the others. Only that we don't owe them much.

Sure, i mean there's lots of highly credible polling. I didnt choose the hypothetical carelessly. Western states are by far and away the only places that maintain these liberal island of virtue, and even so, vanishing and small. The idea that eg., homophobia isn't omnipresent in the vast majority of the world is a level of head-in-the-sandism which, of course, must necessarily accompany the sentiment of "let's import massive amounts of homophobia"

Either way, it's a hypothetical to be taken literally. Suppose it, then what's your analysis? Is the coincidence of their race salient?

If citizenship isnt a line between two classes of people to which states owe obligations, then why isnt the USA obligated to pay for the healthcare of everyone in the world?

Why does medicare/medicaid end at the US border?

States are not the creations of the enlightment

I'll await your grant of medicare to the population of africa

State policies do not operate in this fashion. See, for example, the reality of importing 1 million people.

The existence of a state pressuposes two "classes" of people: citizens and non-citizens.

Citizens are those who have lived and died, who have laboured and been taxed, and have made the very state which is constituted by them -- and they are owed, by that state, a society they wish to live in.

Non-citizens are everyone else. They are owed very little, at best, not to be killed elsewhere; but certaintly, not even to be aided. Unless you want to divide the wealth of every nation by 8bn and watch all of it disappear.

In any case, to non-citizens nothing is owed. Certainly not being carefully scruitnized under a microscope to see if a border agent can detect a lack of cultural or ethical fit.

And in any case, such a fit can be determined by citizens themsevles. And polled, overwhelming, citizens of western nations have spoken. And they have seen your dice rolling at the border, and havent appreciated its concequences.

THe presumption you have on the consent of your fellow citizens to give what you eblieve is owed to other citizens of other states -- this presumption is extraordinary, obniouxous, and short-lived. And much of your attitue here is shortening it.

> cultural and economic phenomenon