Hacker News Reader

Comment by mjburgess | Hacker News Reader

Almost every predicate in natural languages is an arbitrary association of properties in the world. There isn't any property a person has "bald", nor are there "tree"s in the world. What bundles of properties the vast majority of language names are a product of historical and contingent associations we've made for practical reasons.

Likewise, languages do not have the same distributional structure. There is no reason an alien language would use discrete tokens to name properties, nor co-locate tokens by linear position in a 'sentence'. Historical human languages did not; using, eg., the full 2D structure of the clay tablet.

To suppose that it is the glyphs and their colocation which somehow bare meaning is a nonesensical superstition. The world is what our words mean, and it is we, in that world, who provide them their meaning. We can do so with arbitrary linguistic structures.

View on HN

Languages do in fact have similar distributional structure, so that it is possible to learn how to translate words without supervision: https://arxiv.org/abs/2203.04863

In principle languages are arbitrary, but in practice they’re describing the same world and the same concepts end up being useful.

Those are on embedding vectors, not on words. They use embeddings created from translated sources, eg., wikipedia articles.

Yes, if you construct an embedding vector on texts-A, and another on texts-B where (A, B) are translations of each other, then an "unsupervised" algorithm really will give you the dizzying heights of a little above coin-flip accuracy on highly engineered self-selected benchmarks.

They evaluate by by looking at the most in-use words in each vocab.. so you take the most in-use words on translations of Wikipedia, whose frequency is decided by the need of translation.. and then you use that to evaluate.

It is blindingly obvious that the structure and frequency of heirglphys on tombs, Chinese glyphs in poetry, and latin in medieval liturgical literature are not distributed by Reality.. written in this order by God so that the Langauge of Reality is what places "d" alongside "oor". We already know this to be the case. The assumption of its opposite is rank pseudoscience.

> It is blindingly obvious that the structure and frequency of heirglphys on tombs, Chinese glyphs in poetry, and latin in medieval liturgical literature are not distributed by Reality.. written in this order by God so that the Langauge of Reality is what places "d" alongside "oor". We already know this to be the case. The assumption of its opposite is rank pseudoscience.

I mean, on the first level, of course they're entirely determined by reality in the sense that the human brain is a real, physical object. But also on a second level they're still entirely determined by reality because the shape of the human mind is also entirely determined by reality. What is the mind for except reflecting reality? What is language for except communicating it? Sure that reality is warped, filtered, reduced and biased, but the data is still in there. That's in large part why LLMs need such ludicrously large training runs.

I don't think letter frequency is objectively determined, but we know for a fact (many studies!) that the features that large language models learn are far, far above the scale of letters. Even arguing about phrases isn't engaging with the current state of the art.

We're not talking about Markov chains here.

Right, but neural networks at a certain level of scale begin to compress the perception described by the text rather than the text per se. The lexical features are abstracted away and the network begins to understand meaning directly. Of course, you'd still have differentiation by different perceptions and focus on different parts of reality, but I see no reason why that should not likewise abstract out at even higher levels. Ultimately, the whole point of language is that it's about reality, and there is only one reality no matter how many different cultures you feed in. The network isn't doing magic - it's just doing exactly the same thing that we're doing with language. The reason why language is useful is the same reason it's learnable.

If you want to understand how statistical AI systems work, then you should understand the basics of formalism. They are curve-fitting algorithms that approximate the probability distribution of a hisotrical dataset. That's all they are.

By analogising to any animal or human mind, you aren't describing anythign that acutally exists. A neutral network algorithm isnt neural and it isnt a network. It's a statistical curve-fitting algorith. One oughtnt study trees to understand a decision tree either.

This language is entirely metaphorical. There are no neurones in an NN, there are just summation entires in a matrix. This matrix comprises weights, which define the orientation and scale of the line pieces which form the curve being fit to the data.

They're, in particular, multi-level curve-fitting algorithms. A formalism which, I'm sure you are aware, can express any bounded computation. In particular, depending on how you set up the tape, neural networks are comfortably Turing complete.

We're not talking about 2-layer perceptrons anymore here.

In my opinion, you should look less at the formalism and more at the empirically demonstrated performance.

I'm not aware that it can express any bounded computation. But such is irrelevant anyway: there is no function from the distributional structure of text tokens to reasoning. There is nothing to approximate, and this is a system trained just to approximate some presumed function: our mental capacities arent patterns in glyphs.

Whatever property you might imagine a transformer architecture to have (and it is vastly fewer than the set needed for general computation), the problem here is that it's being applied to approximate the structure of historical text data which isnt being generated from such a function.

Indeed, there is no function which generates text data, there's a very large number of independent generating processes that give rise to the distribution of text. The phrase "the war in ukraine" acquires a different semantics over 2010-2030 in a radically different way than, "I liked that film" does.

The capacities which produce distributions of text tokens are highly varied, complex, invovle a vast array of our mental processes, and so on. There's literally almost nothing in the distribution of text tokens that corresponds to any features of these processes.

The structure of language is conventional, and rests on our familiarity with such conventions. Otherwise, let's end all science, everything to be known about the world derives from how "e" occurs alongside "lectron"

> Indeed, there is no function which generates text data, there's a very large number of independent generating processes that give rise to the distribution of text. The phrase "the war in ukraine" acquires a different semantics over 2010-2030 in a radically different way than, "I liked that film" does.

But those semantics are revealed in the greater context of the phrase! That's why it's so important that transformers can attend to large context ranges; that's what lets them learn the greater semantic patterns to begin with. And at the limit, at a scale greater than phrases, I simply reject the idea that the same article - the same book - can have totally different meanings depending on context. Language isn't just shaped by context, it shapes context itself. Because language cannot be considered without context, it reveals information about that context, and in fact any compression of language ultimately requires modelling the person and even society that produced it. That's what the network learns.

If your words don't have meaning beyond themselves, what are you even talking about?

> If your words don't have meaning beyond themselves, what are you even talking about?

...err... of course? That's the whole point.

"Context" here isnt other words. It's the world. The meaning of words is the world.

When I say, "I like what you're wearing" i'm not summarising a history of prior texts; it has nothnig to do with any statistical operation over historical documents. It has entirely to do with what you're wearing.

Langauge use is a side-effect of being embedded in reality, directly attentive to it, and so on. Words are mere symptoms of how we are situated in the world.

LLMs merely replay these back to us. They are not in the world. They cannot, in principle, ever mean, "I like what you're wearing"

I'd say exactly the opposite. It's because human speech is about the world, and LLM speech is about human speech, that LLM speech is about the world, by transitivity. "The ball fell to the <floor>", the LLM predicts, ultimately, not because of any feature of the human brain, or any feature of the English language, but because of gravity. That the causal arrow passes through a human brain does not make this any less true! Because human speech is inextricable from physical reality, language models likewise learn to model reality. They learn this poorly, incrementally, making missteps on the way - granted! But that they learn it at all shows that there's more going on than statistical modelling - or that statistical modelling of language is more than it sounds like.

Human speech isnt about reality. Speech, langauge, tokens, etc. are just scratchings on a wall. When we speak we infer the reality-orientation of people's intentions (and so on).

The strings 010101010111100101, 000010101010111010101, 120192019092109209102, etc. are all "about" the exactly the same scenario whereby different communities of speakers adopt different conventions for the meaning of the terms (indeed, whether pairs/tripples/etc. are terms).

This is not like a thermometer in a glass of water, the height of the mecury being determined by the temperature. Nothing about language is determined by reality. It is purely a highly lossly encoding scheme based on historically arbitary encoding conventions. It isn't a measure of reality in any sense.

Modelling this encoding scheme doesn't model anything about the world. It only appears to if you generate text according to this scheme.. and only to those who can decode it. Hieroglyphs likewise, scratches on clay, and everything else.

You are subject to the illusion of intentionality whereby your phenomenal experience of langauge is rendered immediately meaningful by your brain, ie., your decoding scheme is prior to the construction of your experience of the world. This is wholey absent from anything machines are processing.. whih are only the frequency relationships between tokenizations of syntax

> Nothing about language is determined by reality.

This is just transparently silly. Or rather, you're equivocating between "language is not fully determined by reality" and "language is uncorrelated with reality." Language is partially caused by reality; thus, large language models, seeking compression, learn the arbitrary parts and the reality parts separately. That's why they can translate.

The human sensory stream is a kind of language. The exact same logic goes for it! If LLMs could not learn about reality from language because language was not determined by reality, then human brains could not learn anything about reality either.

Nothing about reality forces a certain neural state vector to correspond to a certain sense impression. E pur si muove.

Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'.

As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits. Literally, everything here is arbitrary convention; and under arbitrary permutation, the sign can mean the same, and share no properties with any other permutation whatsoever.

Rather it is by being-in-the-world you have acquired a direct understanding of roads, speeds, limits, rules, laws, society, etc. And alongside this all the while internalizing arbitary conventions by which we signify these things.

A stop sign could look like anything at all. We just decide to use arabic numerals, red, white, a certain eye-hight, etc.

The meaning of the sign is something you are reconstructing prior even to being cognitively aware of the sign you're looking at. The meaning of these conventions is a mental reconstruction made possible by your direct familiarity with the world these conventions are related to.

You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs.

There is literally nothing at all in speed signs that has anything to do with their meaning.

This isnt a speculative observation. It places a very hard limit on what the entire field of statistical AI can do, let alone LLMs. Any model created by empirical risk minimization, or fitting-to-historical-data, lacks any relevant capacities. It has only one: drawing a data point from some inferred distribution over historical cases.

In any case, this isnt a speculative exercise. Statistical AI will be forever "limited" to a fixed computational budget for computing predictions over historical cases, with necessarily limitations this involves -- this is only an engineering problem where these cases aren't representative of the kinds of queries people have.

eg., they will fail under relevant counter-factual permutation of the conventions of the language/domain/topic, etc. -- see, e.g., https://arxiv.org/pdf/2307.02477

Applying the lessons of the above paper, note chatgpt3.5 gets the evaluation of python wrong under permuation of its syntax: https://chatgpt.com/share/7df48393-9cb8-41f4-a84e-d21b680702...

..whereas ChatGptv4o does not. This suggests to me that there's been a very large amount of prompting by users since 3.5 trying to create novel programming languges in this way.. hence 4o, having been trained on subsequent user prompts, its "better" at this task.

Of course nothing important has changed between v3.5 and v4o in terms of what the model is, rather there is more query-relevant data to sample from.

> Yes, we cannot learn anything about reality from language alone. This is 'the intentional illusion' or you could call it 'the aboutness illusion'.

I just don't think this is correct. Language is embedded in reality, is a product of reality, and is correlated with reality.

> As in you think you are learning a speed limit for a road by looking at a sign with a speed limit on it. But the sign itself is just some shapes, and no aspect of its structure has anything to do with speed, roads, or speed limits.

Again, simply wrong. 5km/h limit, 60km/h limit, 120km/h limit - even the physical width of the text has a correlation to the exponent of the speed limit! Which will correlate to the curvature of the road! These are not arbitrary, they're tightly interlinked on every level except the specific shape of the glyphs themselves. That's why your argument only holds up for the most basic features such as glyph frequency. As you zoom out, structural similarities begin to appear; not in the shape of the text but in the shape of features abstracted from the text. That's why LLMs attain novel capabilities as you increase depth and context size, as they can begin to condition on the deeper structures of reality that leak through the language. As language is abstracted from reality, at sufficient scale, a cognition based on language begins to necessarily model reality itself.

> You first act within the world by building a rich conceptual understanding, you then associate arbitrary conventions with this understanding, and then can make inferences about this world by reconstructing a novel conceptual understanding given by these signs.

Completely unclear if this is true at any rate. Humans don't first learn to interact with the world and then learn language; language and world-understanding co-develop.

OK. well, lets give it 5yrs. in 10, at least, this will be over

This, at least, I agree with.

(Though not quite in the same sense.)

Computability is of critical relevance.

Our mental capacities can be patterns. https://en.m.wikipedia.org/wiki/Predictive_coding

> There's literally almost nothing in the distribution of text tokens that corresponds to any features of these processes.

There's nothing in evolutionary fitness that necessitates intelligence or reasoning ability either. Yet here we are.

Sorry but if you don't see the connections then you need to do some reading on theory of mind, cognition, information theory, physics, philosophy. All of the fundamental basis are met to allow reasoning to emerge in LLMs.

It is clear now where your confusion lies, and why you are led to believe so strongly that LLMs cannot reason: it is because you are an ML practitioner you overweight your expertise yet you don't know what you don't know, and have foundational gaps in your knowledge. You lack the context in these other fields. If you had them, your position should be closer to agnostic than this strong belief of yours that LLMs in their current form cannot reason.

lol, well when I've written my PhD on those areas we can return to whether I'm an expert on them or not.

One does not form "connections" between them as in some wide-eye conspiracy theorist.. "predictive coding" has little to do with "prediction" in the ML sense. The latter concerned with making a quantitive estimate of some variable by summarising historical data.

What we are doing when we revise a "mental model" is done by counter-factual simulation of possible future states. Statistical AI models conditional probability structures, and computes predictions as an expectation over weighted summarised historical data. This is not a means of performing counter-factual simulation.

One trivial, sadly empirical, way to see this is to note that each marginal token generated is of constant time and energy use. Yet trivially, reasoning and a variety of other mental capacities should require abitarily different time to run. Eg., simulating a complex scenario is necessarily more intensive than a simple one, and so on.

Yet I am quite annoyed that we need such dumb observations to make this point. It speaks of a profound ignorance of "theory of mind, cognition, information theory, physics, philosophy" and especially neurology and zoology which are your most significant missing terms.

It is no real mystery what the structure of various mental capacities involves; nor any mystery what s(Ws(Ws(WX+B)+B...)...)...) computes. Even involving anything beyond trivial applied statistics and trivial results in science shouldnt be required here. This stuff is very obvious.

Reply on news.ycombinator.com