Hacker News Reader

Comment by mjburgess | Hacker News Reader

Liquidity is not a property of h2o molecules but it is of water. Liquidity isn't of water without a container/pressure environment.

The trajectories of particles of air are underdetermined by their own intrinsic properties so a tornado cannot be reduced to some mere aggregate of them.

Emergence is an ontological relationship between properties of objects --- it isn't a mathematical property of a data distribution nor of an approximation function.

The very use of the term has crated all this confusion right here.

Would anyone who thought NNs were showing emergence be also content to find out that the reason for this 'emergence' was just that in the case of so-called 'emergence' our expectations of performance were changing?

Do we call it 'emergence' when we replace estimating data using mean() to estimating with ax+b ?

There's definitely an illusion here, but one quite easy to spot if you weren't in the game of peddling illusions.

View on HN

Two observations,

There is utility and need for some convention of feature description, for the case of the external behavior of the system being correct, for some domain, despite lack of specific training. It is reasonable for users of such systems to say that the internal states and representation don't matter, so long as the behavior is correct; and in cases like these we will benefit from some consensus on how to talk about such things. Fine with me if some new term is applied.

More of interest to me though is that it is not at all clear to me that genuine emergence is not possible through scaling (independent of whether it is in any given existing LLM). Because the optimal (most compact) correct representation for a lot of e.g. language output, is exactly that which benefits from abstraction.

What reason is there to believe that the abstractions derived at higher levels (of the network generally but not necessarily, depends on the architecture) do not encode non-linear problem spaces in the world, which are "real" emergence?

I.e. if the way some network learns arithmetic is to settle on an internal weighting that performs computation, rather than "memorizing assertions", me, I would call that "emergent."

But I'm happy to use some other term should one, er, emerge.

> Liquidity is not a property of h2o molecules but it is of water

The ability to speak English is not a property of floating points, but it is of certain, very specific large tensors of them. What's the difference?

> Emergence is an ontological relationship between properties of objects --- it isn't a mathematical property of a data distribution nor of an approximation function.

I don't see a hard distinction between ontological relationships and data distributions. All information is fundamentally statistical. Our access to ontology is forever and always mediated by "data distributions".

One could, of course posit that there are fundamental, non-statistical ontological things out there. However, the liquidity of water being an ontological relationship while the English-speaking of GPT not being so is merely a hypothesis, not an objective fact of the universe, at least not as far as I can tell.

Reply on news.ycombinator.com