HarHarVeryFunny

They've overcome every hurdle to go from nothing to having working machines, so all these questions are presumably a matter of how fast they will improve and ramp up production (initially 5 this year, 20 next year), not whether they will.

China have been really squeezing all they can out of DUV machines, but I'm not sure how much node size really matters for AI competition - more of a cost issue (more chips/power for same FLOPs) than anything, and TSMC & NVIDIA's healthy profit margins need to be considered too.

1) The number of unique chess games that could theoretically be played (but mostly never have been), is irrelevant to what an LLM is remembering. It can only remember what was in it's training data - a far smaller number of maybe 10's of millions of games (of 30-50 moves each).

2) An LLM is not going to memorize vs generalize when there is no training pressure to do so. You might expect it to memorize book openings that occur over and over in the training data, but not some random non-celebrity game that occurs once in the Lichess dataset and is never again referred to.

> They can't possibly remember even a few positions. Don't you know the legend about rice grains on a chess board?

If the wise man was a bit wiser, he'd have asked for his rice on a snakes & ladders board (100 squares, not 64) and would have had 2^36 more rice, which is equally irrelevant.

> I'm not sure you quite understand the full meaning of this statement. If you did, your following paragraphs wouldn't follow

I was just explaining how this comment you made is wrong.

> There are already talks of banning China from even purchasing DUV machines from ASML

A bit late for that now that they are moving into early production with their own.

> Outward behavior is that all matters

Yeah, but it's helpful if what leads up to that behavior gives you some warning it's about to happen. Animals do this for a reason since millions of years of evolution have shown that a snarl or mock charge is less dangerous than going right for a death match.

If you kept pushing an AI's buttons, seeing it appear to get more and more pissed off, until it finally snapped and killed you, then you'd have yourself largely to blame.

If the AI predicted it should stay positive (i.e. generate positive vibes) and not react to your poking, but then another predictive pattern kicked in and it killed you out of the blue, then that seems more problematic to me, even if you don't agree.

If you asked it to try to win, to "plan lines step by step", etc, then it would do it's best to follow that instruction, but unless RLVR trained to reason about chess (easy to do, but not sure which models may have done it) then it'd have to instead rely on the chess reasoning it had seen during pre-training (post-game interviews etc), which I doubt is enough to do very well.

However, if you just ask it to continue a game, halfway in progress, then by default it will try to predict the most likely continuation, which is that both players will continue to play at the level they have done so far. This isn't a theory - it's been documented, as well as what you'd expect.

Weights, fixed by training, are not the same as emotions which are dynamic - innate systems detect inputs critical to survival (e.g. fast moving visual inputs, loud sounds), causing neurotransmitters like adrenaline and dopamine to be released, which then temporarily affect the operation of the cognitive system.

What you have in a pre-trained LLM is the ability to recognize emotions, and use that as one of the dozens of other context patterns it recognizes to predict continuations in the same style.

An LLM doesn't appear happy, sad, afraid, etc (to extent that it does - pretty minimal) because it is experiencing that emotion, but rather because it is predicting that it should appear that way. As people continue to anthropomorphize models, and take them at face value, this is a dangerous difference.

It's not that they "just found out" - what they are saying is that while they were previously dogfooding because it's good practice, now that their models are so much stronger they are using them because it helps accelerate.

If you look at how many years the whole NVIDIA and CUDA ecosystem has been evolving, it's certainly impressive how they've just stood up and optimized this CUDA-free 100,000 node cluster in just a few months.

Same thing with Trump not helping Ukraine and berating NATO. He thought he held all the cards, but now Ukraine has a thriving battle-tested drone industry, UK and France have stepped in to replace the US with advanced missiles and anti-missile systems, stepping up their own production and transferring IP to Ukraine.

Now, the US is left out in the cold with little influence left, themselves now the ones with an anti-missile shortage.

If you understand what they have achieved here, then the notion that they are bottle-necked on training data is absurd.

I wonder how you imagine that China built their own space station? Reliant on using American made duct tape, perhaps?

Do you realize how reasoning models are being trained nowadays? You design/build simulation environments to run agents in, with the environment providing the RLVR "verification" scoring. So why won't Ziphu use GLM to build their own RL training environments? Do you think they are not doing this?

China themselves recognize this. After Trump relaxed sanctions and allowed NVIDIA H200 sales to China on a case by case basis, the Chinese government stepped in to essentially block it!

In addition to Huawei who make the Ascend series that Ziphu are using, there are also at least a half dozen or so other Chinese companies also making their own AI accelerators.

Wouldn't it be refreshing if OpenAI and Anthropic were this open, and spelled out how they were using their own models during development and rollout?!

All I can recall reading from OpenAI about what they have actually done in the name of "RSI" is using one of their models to help automate the training process.

Ziphu, aka Z.ai, is the company that makes GLM (a very competitive Chinese LLM).

Why would you be reading their corporate blog posts if you don't even know who they are?!

Ziphu (who make GLM) use Huawei Ascend processors made by SMIC. Huawei use a combination of domestic memory from CXMT and leftover (pre-sanctions) memory from Samsung.

Just like the rest of the world, including the US (Intel, Micron), SMIC are currently using ASML lithography equipment (DUV, not EUV), but Shanghai Aishengna are now moving into early production with their own DUV machines, with SMIC and CXMT as early customers.

There is also a state sponsored Chinese EUV development underway.

Are you imagining that an LLM tasked with predicting a game continuation is going to play to win instead?

No, there is no reason for you to listen to me.

Go ahead believing rocks are conscious if you like.

Do you go out on weekends asking people to stop abusing rocks?

Rhetorical question - I don't care what you do on weekends.

Bye!

Nobody is evolving transformers. They are basically the same today as they were 10 years ago, other than a few computational efficiency changes.

An LLM will learn anything that helps it predict, including the emotional state of the writer - that is expected.

If you give an LLM the move sequence of a half-played chess game and ask it to continue as white or black, then it has learnt enough to model the ELO rating of both players and will continue playing at that level. It is not playing to win - it is doing what you expect and predicting as well as it can - it predicts the 1500 ELO player will keep playing at that level, and generates moves accordingly.

An LLM appearing to exhibit an emotion (if we anthropomorphize it and read emotion into it's output) is just predicting as well as it can - if the context calls for sad output, they you'd expect to get sad output and will necessarily find that "we're predicting sadness" detector somewhere internally.

Transformers are the same as they ever were from 10 years ago, other than minor efficiency tweaks like MOE and different attention mechanisms. Training is getting more and more complex, resulting in better and better cargo cult reasoning etc, but the architecture remains the same.

OK, but generations of people keep drinking themselves to death.

Generations of people lived with lead water pipes and asbestos roofs.

Within living memory we had cocaine in coke, heroin in cough remedies and were giving pregnant women thalidomide for morning sickness.

Western science is a modern thing, and even now many people don't believe it.

You are talking about 2^64 being a huge number I assume ?

If not, then what are you talking about ?

If yes, then what is the relevance to an LLM playing chess ?

> The problem is that the emergence of a sufficiently complex AI capable of suffering will likely come before we understand that we're creating a sufficiently complex AI capable of suffering.

No - suffering in an emotional state, and we'll know if we are choosing to design cognitive architecture with emotions. It's not going to happen accidentally.

> Would that argument ever hold up for humans or animals?

Why don't you hit your thumb with a hammer, then report back ?

> You can't even prove that the rocks in my backyard aren't conscious.

Sure I can, but that's because I have a well developed theory of what consciousness is, and the fact that you are entertaining the possibility of rocks being conscious tells me that you don't.

If everything is conscious, including my coffee cup and the toast I had for breakfast, then I guess we can cross consciousness off the list of things we need to worry about in terms of AI rights.

And no, I don't want to discuss what consciousness is. Maybe there is a thread for that somewhere else, but don't look for me there either.

> How do you know it doesn't have qualia?

Tokens in, tokens out. Where do you think the quale is - layer 42 ?

Seriously, do you realize how simple and NOT brain-like a transformer is ?

An LLM telling you it fears death is predicting some sci-fi trope it was trained on - maybe something you wrote yourself.

An LLM is just a Transformer - a statistical predictor. Don't be confused by the fact it talks like a human - it is a software function that is designed to copy human training samples.

Maybe one day we'll build an artificial brain or embodied artificial animal with the requisite moving parts to be conscious, have emotions, etc, but that's probably at least 50 years away, even if it were being pursued; and it may turn out to be one of those sci-fi future ideas like the Jetson's world of flying cars that never materializes because its impractical and there is no real demand.

If people are willing to think that an LLM is conscious, then why would anyone spend billions/trillions of dollars to build an AI that actually is conscious? What would be the point?

AGI vs narrow AI is just a matter of generality (robustness - removing the fragility of being good at some things and awful at others). So far the biggest markets for AI (LLMs) are coding and business automation, two areas where you very much just want something reliable and without fake opinions and personality.

There is a market for ChatBots with personalities (and it always amuses me that the inventor of the transformer, Noam Shazeer, saw this as the greatest business opportunity for them with his character.ai), although from what we've seen this can be highly problematic, and in fact China has just banned "AI girlfriends and boyfriends".

Most of the bullshit around AI would be avoided if people called it what it is. Call a language model a language model.

If tomorrow we design something better, then give it a new descriptive name.

People read AI and AGI and their brains seem to flip into sci-fi fantasy mode, thinking that they are talking about aliens, not transformers.

You can't hurt a software function, or kill it. It's not like an animal - it doesn't have a body - it's bits stored on a disk.

There is no need to give rights to something that's can't suffer or be killed.

Maybe one day we'll build artificial animals complete with emotions, and should think about that carefully, but today all we've got is language models.

The model doesn't have a view of it's own. It's a predictor of other people's views (training samples).

It's not even giving you the consensus of the training data (although it's often harmless to think that it is), but rather predicting a response to your input, and if your question steers it too much then you've just become part of the answer.

> Any AI you train is going to have goals

In the spirit of the article we're responding to, there is no need to anthropomorphize language models and say they have goals when they don't.

The RL training process tweaks the weights of an LLM to make it behave as if it were reasoning and/or had a goal, but it doesn't. It would be like saying that a cart horse, fitted with blinkers and heading for the church, has a goal of going to church.

AGI and RSI are both meaningless terms, meaning whatever you choose them to mean.

RSI is the new sexy. Models are RSI-ing themselves towards the singularity, these folks' agents are RSI-ing themselves towards mastery of their training environments, and my pet cat is RSI-ing himself into the best cat that he can be.