hackinthebochs

This is a bizarre sentence seeing as Google is a portal for most people. So replacing Google with ChatGPT is replacing all those destination sites as well.

Embedding vectors assign semantic features to syntactical structures of the vector space. Operations on these syntactical structures allow the program to engage with semantic features of program state directly, leveraging the meaning of program state to alter its execution. Intelligence is leveraging information/experience to achieve one's ends. Manipulating meaning through embedding vectors in service to answering prompts is an example of that.

LLMs are not non-deterministic by design. The randomness in the output is artificially injected for ergonomic reasons. (Yes, the non-determinism in production systems is different, but also not by design)

The danger when making decisions on behalf of other people is in imagining yourself in their position, rather than imagining them in their position. So you have a happy fulfilled life, you have the kinds of human relationships that provide deep meaning to you, you have the right kind of curiosity and wonder to be fulfilled by witnessing burgeoning technological progress, etc etc. Yes, being deprived of all that would be very bad for you.

Now imagine someone psychologically very average whose source of meaning is in the familiar environment where they have an acceptable amount of status, broad community support and much social engagement. They have a peculiar culture and social practices which gives them deep personal meaning. Now imagine all of that is swept away; no social status, no familiar environment, cultural practices gone, extended family scattered to the wind. What value does the wonders of the modern world have to someone who has lost everything they value?

Consumerism, technological marvels, and endless entertainment are the modern world's answers to the waning sources of personal meaning in modern life. Turns out that it's all a very poor replacement.

My concern isn't mainly about addiction and vices, its more about a loss of meaning and being adrift in an alien world. The addiction/vices are just what one falls into when your life becomes devoid of meaning. I think the mental health crisis that most of the western world seems to be engulfed by, despite living in an age of endless wonders, goes to show that there is something deeply wrong with our modern way of living. My caution is against introducing this "sickness" (for lack of a better term) into populations even less able to defend and preserve personal meaning against an onslaught of technology and ideas that would evaporate their personal sources of meaning in an instant. That is no gift to anyone.

I agree the application of the logic doesn't end with isolated populations. The relevant difference is that those brought up in the modern world have/had the potential to engage with knowledge of addiction and other vices, and decide for themselves how to steer their lives in an informed way. We draw a somewhat arbitrary line for being responsible for oneself as the best way to structure society given the tradeoffs. There's just no sense in which these isolated populations as a whole can be sufficiently informed about the modern world to suitably navigate its complexities to lead to their prospering in it.

If you have something to contribute to the discussion, just say it

>If that were true, we should have seen LLMs play good chess by now.

Not at all. LLMs learn by imbibing a mass of relationships as isolated fragments of information. There is a certain amount of sorting and indexing that happens during the training phase. There is also a certain amount of compute executed on these relationships during inference. LLMs can model processes that fit within the compute budget. Language translation works well because language is lookup-heavy while being light on compute.

Chess is a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Humans cut through the compute requirements by reinforcement and learning intuition. LLMs don't get reinforcement on chess so they must compute during inference a unified model of chess. Developing a strong model of chess from raw fragments of information is simply not in their compute budget.

This is not good reasoning. Humans need at least dozens if not hundreds of reinforcement sessions to only make legal moves, and still occasionally fail (consider pins, discovered check, failing to respond to check). LLMs must one-shot a competent game after imbibing a mass of disconnected units of information about chess. Nothing about the two are similar.

See my comment here for more: https://news.ycombinator.com/item?id=49725306

So you weren't giving it an updated board state after every move? If you want to compare apples to apples, it should give an updated board state for each move, or you should play blindfolded.

> generally read the rules of a game and then follow them

How many times do you think chess.com prevents illegal moves from being executed? Even Super GM's fall for mate-in-1's occasionally, which is functionally equivalent to missing a pin or a check. This idea that LLMs failing to only ever make legal moves undermines their intelligence doesn't pass the smell test.

It is interesting, but people are drawing the wrong conclusion from it. For one, LLMs don't go through a "chess learning phase". They're not analyzing a board as they're learning the rules or studying games to create a coherent model of chess. They're just imbibing raw relationships as isolated fragments of information. The fact that they can't unify this into a coherent model of chess playing in one shot and execute a competent game says nothing interesting about the limits of their intelligence. If you give frontier models the rules of chess in their context window, could they perform only legal moves? I bet they could, excepting trickier scenarios like pins and failing to respond to a check. But those kinds of scenarios have to be reinforced in any human player as well. Even Super GMs fall for mate-in-1's occasionally which is functionally equivalent to those kinds of failures.

We're obviously using different meanings for "good" here. But aside from that, it took 100's to 1000's of reinforcement iterations for your brother to play competently. While certainly impressive, that is still an entirely different category from piecing together disparate facts learned during training (LLMs aren't analyzing a board as they're learning the rules or ingesting PGN files), to executing a competent performance in one shot.

>Where is all the training and practice LLMs did to become so good at coding?

Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.

We're not talking about learning the rules of chess here, but playing a competent game from just being shown the rules. Why is it so hard for people to keep track of the thread of discussion?

I'd genuinely like to see the results of that.

No, but superhuman capabilities derived from ordinary learning is, which is what the parent comment described. Why is that not obvious?

I would bet a lot of money that Astra can follow the rules of chess (perhaps if repeated within the context window). Also, this is a different argument than what I responded to.

And how many years of direct play and study does it take for a human to get good at chess or any other game? Absolutely no human ever could be good at chess just by reading a few books, or even every book on chess. That's just not how the brain works. If LLMs could do that they would truly be superintelligence.

>LLM can’t beat an avg chess player.

Why should that matter?

But they're not equipped to decide whether they want it or not. It's like offering drugs to children saying they can choose if they want it. Children simply aren't equipped to make that decision, neither are people in these populations. I don't think we should keep them isolated, but contact should be limited initially. The technology we introduce them should be limited, enough to increase their quality of life and eliminate disease and so on without being utterly disruptive. Over time more technology and information can be introduced.

No population should be kept living a life we would consider extreme poverty out of some noble ideal. But we also shouldn't completely disrupt their way of life out of a different noble ideal. Wealth as we understand it isn't the only thing of value. Human connection, meaning anchored in community and deep roots, a sense of belonging and so on, are what really matter. Introducing modernity to such populations will sweep all of that away in one motion. These people would be left adrift in an alien world devoid of personal meaning, their only solace whatever addiction appeals to them most.

It's also immoral to expose isolated populations to all manner of the modern world's vices, optimized to be maximally addictive and maximally extractive, for which they have zero defense or even knowledge of their dangers.

Prove is a strong word. But it supports the existence of the condition as it coheres with our scientific models of visual experience. You can always reach for obscure philosophical puzzles about phenomenal experience to try to invalidate the science, but at that point you're just having a different debate than the rest of us.

It's a real phenomena. Mental imagery corresponds to voluntary activation of the visual cortex[1]. The quality of the self-reported imagery corresponds to the degree of activity in the visual cortex[2] while imagining some visual scene. People with aphantasia have little to no visual cortex activity.

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4595480/

[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8186241/

>pacing between the us labs? what does that do for china?

I've seen no indications that China is in any kind of race with the US. They seem to be content to be 6 months behind and just copy what we do. They would probably be content with a bilateral agreement to pause progress.

The China bogeyman serves only one purpose, and that's to clear the way against anything that may cause friction with forward progress.

>endless abundance of solutions and ideas cannot coexist with a healthy society.

This really cuts to the heart of the problem with AI. Not only does AI undermine the monetary economy, it undermines the intellectual economy. What is humanity without the need for collaboration for survival or for intellectual progress, ultimately providing the impetus to build something greater as a result? I don't know, and I'm not looking forward to finding out.

Yes, reasoning with models and making inferences are perfectly acceptable forms of evidence. But you can model the world based on ones knowledge and experience and infer when some event doesn't fit the typical pattern, then form beliefs about what that means. Also perfectly fine from an epistemic perspective. The rejoinder "there's no evidence" to a belief based on such an inference does no work.

I believe the sun will rise tomorrow without "evidence" (again, narrowly construed). We all do. It's only those who abuse the idea of epistemic hygiene who claim otherwise, usually with ulterior motives.

It is not reasonable to not believe anything unless there is "evidence" (narrowly construed as an observation incompatible with the negation of some state of affairs). Beliefs have a wide spectrum of characterizations, and not all belief must wait until publicly corroborated evidence is available. Some events defy evidence and we can and should use experience and reasoning to infer unobservable states of affairs.

If you're betting the farm on there being a wall on the current trajectory of LLMs, I suggest you hedge that bet.