Hacker News Reader

If it really is Reinforcement Learning as they claim, it means there might not be any direct supervision on the "thinking" section of the output, just on the final answer.

Just like for Chess or Go you don't train a supervised model by giving it the exact move it should do in each case, you use RL techniques to learn which moves are good based on end results of the game.

In practice, there probably is some supervision to enforce good style and methodology. But the key here is that it is able to learn good reasoning without (many) human examples, and find strategies to solve new problems via self-learning.

If that is the case it is indeed an important breakthrough.

This is the bitter lesson/just put it in the model. They're trying to figure out more ways of converting compute to intelligence now that they're running out of text data: https://images.ctfassets.net/kftzwdyauwt9/7rMY55vLbGTlTiP9Gd...

A cynical way to look at it is that we're pretty close to the ultimate limits of what LLMs can do and now the stake holders are looking at novel ways of using what they have instead of pouring everything into novel models. We're several years into the AI revolution (some call it a bubble) and Nvidia is still pretty much the only company that makes bank on it. Other than that it's all investment driven "growth". And at some point investors are gonna start asking questions...

That is indeed cynical haha.

A very simple observation, our brains are vastly more efficient. Obtaining vastly better outcomes from lesser input. This evidence means there's plenty of room for improvement without a need to go looking for more data. Short term gain versus long term gain like you say, shareholder return.

More efficiency means more practical/useful applications and lower cost as opposed to bigger model which means less useful (longer inference times) and higher cost (data synthesis and training cost).

That’s assuming that LLMs act like brains at all.

They don’t.

Especially not with transformers.

Says who? At a fundamental level

At a fundamental level, brains don’t operate on floating point numbers encoded in bits.

They have chemicals to facilitate electrochemical reactions which can affect how they respond to input. They don’t throw away all knowledge of what they just said. They change continuously, not just in fixed training loops. They don’t operate in turns.

I could go on.

Honestly the number of people who just heard “learning,” “neural networks,” and “memory” and assume that AI must be acting like a biological brain is insane.

Truly a marvel of marketing.

Fundamentally and physically are two different things. A logic gate is a logic gate if it's in neurons or silicon. Are abacus and calculators solving different things? No.

You're proving my point, things like them changing continuously are exactly what I mean when I say the brain is more efficient. Where there's a will theres a way and our brains are evidence that it can be done.

You're saying that because two different objects can to solve the same problem, they must work the same way.

An abacus and a calculator were both made to solve relativly simple math problems, so they must work in the same way, right?

And apple and an orange are ways to store sugar for plants, so they must be the same thing, right?

No. That's not how any of this works. An abacus and a calculator are two different tools that solve the same problem. They don’t act like each other just because the abstract outcome is the same

> You're proving my point, things like them changing continuously are exactly what I mean when I say the brain is more efficient.

I don't see how that proves that neural networks act like brains.

It's also not just a difference in terms of efficiency, it's the fundamental way that statistical models like neural networks are trained. Every time their trained, it's a brand new model, unlike a brain, which is still the same brain.

Also, neural networks and brains were NOT made to solve the same problems... even if your argument made any sense, it doesn't fit here.

No I'm not saying they must work the same way. I'm saying it's evidence there is a more efficient way as they both solve the same problem and one is more efficient (in truth both are more efficient in different areas). At an abstract level they can be doing the same thing. What does a simulator do?

Think a little further yes, currently it's a brand new model each time but why will it be this way forever? Its an engineering problem one that we can solve and the brain is evidence it can be done.

Neural networks were originally inspired by the brain. Yes, they've deviated but there's absolutely no reason they can't take further inspiration.

So you’re just abstracting everything to the point where everything is a “something solver” and if two things can solve the same something, one must be a better version of the other?

Abstracting everything to the point of meaninglessness isn’t a worthwhile exercise.

No, that's a stretch and even from that how do you get to that conclusion? I think you're clearly trying to brush off my comment.

I assume you're of the opinion humans are special.

One aspect that’s not achievable is they discuss hiding the chain of thought in its raw form because the chains are allowed to be unaligned. This allows the model to operate without any artifacts from alignment and apply them in the post processing, more or less. This requires effectively root and you would need the unaligned weights.

Ok but this presses on a latent question: what do we mean by alignment?

Practically it's come to mean just sanitization... "don't say something nasty or embarrassing to users." But that doesn't apply here, the reasoning tokens are effectively just a debug log.

If alignment means "conducting reasoning in alignment with human values", then misalignment in the reasoning phase could potentially be obfuscated and sanitized, participating in the conclusion but hidden. Having an "unaligned" model conduct the reasoning steps is potentially dangerous, if you believe that AI alignment can give rise to danger at all.

Personally I think that in practice alignment has come to mean just sanitization and it's a fig leaf of an excuse for the real reason they are hiding the reasoning tokens: competitive advantage.

The challenge is alignment ends up changing the models in ways that aren’t representative of the actual training set and as I understand it this generally lowers the performance even for aligned things. Further the decision to summarize the chains of thought includes the answers that wouldn’t pass alignment themselves without removal. From what I read the final output is aligned but could have considered unaligned COT. In fact because they’re in the context they’re necessarily changing the final output even if the final output complies with the alignment. There are a few other “only root could do this,” which says yes anyone could implement these without secret sauce as long as they have a raw frontier model.

Alignment started as a fairly nifty idea, but you can't meaningfully test for it. We don't have the tools to understand the internals of an LLM.

So yes, it morphed into the second best thing, brand safety - "don't say racist / anti-vax stuff so that we don't get bad press or get in trouble with the regulators".

Glass half full and the good faith argument.

It's a compromise.

OpenAI will now have access to vaste amounts of unaligned output so they can actually study it's thinking.

Whereas the current checks and balances meant the request was rejected and the data providing this insight was not created in the first place.

I have also spent some time on 2) and implemented several approaches in this open source optimising llm proxy - https://github.com/codelion/optillm

In my experience it does work quite well, but we probably need different techniques for different tasks.

Maybe 1 is actually hat you just suggested - an RL approach to select the strategy for 2. Thank you for implementing optillm and working out all the various strategy options, it’s a really neat reference for understanding this space.

One item I’m very curious about is how do they get a score for use in the RL? in well defined games it’s easy to understand but in this LLM output context how does one rate the output result for use in an RL setup?

That’s the hardest part, figuring out the reward. For generic tasks it is not easy, in my implementation in optillm I am using the llm itself to generate a score based on the mcts trajectory. But that is not as good as having a reward that is well defined say for a coding or logic problem. May be they trained a better reward model.

> So it's also possible that the prompt loop has no special sauce and that the capabilities here do come mostly from the model itself.

The prompt loop code often encodes intelligence/information that the human developers tend to ignore during their evaluations of the solution. For example, if you add a filter for invalid json and repeatedly invoke the model until good json comes out, you are now carrying water for the LLM. The additional capabilities came from a manual coding exercise and additional money spent on a brute force search.

Well, if LLMs are system 1, this difference would be building towards system 2.

https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow

Yes indeed, and personally if we have AGI I believe it will arise from multiple LLMs working in tandem with other types of machine learning, databases for "memory", more traditional compute functions, and a connectivity layer between them all.

But to my knowledge, that's not the kind of research OpenAI is doing. They seem mostly focused on training bigger and better models and seeking AGI through emergence in those.

The innovation lies in using RL to achieve 1.) and provide a simple interface to 2.)

You don't to execute code to have it backtrack. The LLM can inherently backtrack itself if trained to. It knows all the context provided to it and the output it has written already.

If it knows it needs to backtrack then could it gain much by outputting something that tells the code to backtrack for it? For example, outputting something like "I've disproven the previous hypothesis, remove the details". Almost like asking to forget.

This could reduce the number of tokens it needs at inference time, saving compute. But with how attention works, it may not make any difference to the performance of the LLM.

Similarly, could there be gains by the LLM asking to work in parallel? For example "there's 3 possible approaches to this, clone the conversation so far and resolve to the one that results in the highest confidence".

This feels like it would be fairly trivial to implement.

O1 seems like a variant of RLRF https://arxiv.org/abs/2403.14238

this is why i became skeptical of openai's claims

if they shared the COT the grift wont work

its just RL

I can't help but feel that saying "it's just RL" is like someone at the start of the 20th century saying "it's just electricity", as if understanding the underlying mechanism is the same as understanding the applications it can enable.

Tbf RL is pretty incredible.

I trained a model to play a novel video game using only screenshots and a score using RL and I discovered how not to lose

The innovation lies in making the whole loop available to an end user immediately, without them being a programmer. My grandma can build games using ChatGPT now.

what game has she made

What games have people made with ChatGPT? Do you have an example of a live, deployed game?

Yes, a gazillion of them. Someone in a scrabble Facebook group made this entirely with ChatGPT: https://aboocher.github.io/scrabble/ingpractice.html

1 < a gazillion

I think the problem here is different expectations for what a “game” is.

If you tell a room full of programmers that something can make a game they’re going to expect more than that.

I look at that and I don’t really see a game, I see flashcards.

Still pretty cool chatgpt can put that together.

Also the “try again” button doesn’t work.

Its actually kind of wild how obvious it is that this was not made by a human.

Look, I get the societal development that you can input narrative text and the code for this pops out is super neat.

But trying to be fair here, anyone would call this incomplete, right?

There are several obvious bugs in styling and interaction.

This example is exactly what I was expecting. An ephemeral, simple-yet-buggy single page that’s barely a game in common understanding.

That person, while maybe not actively programming things, does appear to have forked several repos on GitHub a decade ago. I would say that’s above the level of technical competence implied by the “my grandma” phrasing of the OP.

No she can't, comments likes yours are just made up nonsense that AI hype-mans and investors somehow convinced us are a fair opinions to have.

While AI is overhyped by some people, the parent's statement is not only true but was true long before o1 was released.

Do you know of an example game by someone with no coding experience using an LLM?

Check out replit agents, they can make games and apps autonomously now

Do you know of an example game I can play right now?

Practical challenge with a $250 prize: Make a 2D isometric HTML+JS game (dealer's choice on library) in the next 48 hours that satisfies these modest random requirements:

A character walks around a big ornate classic library, pulling books from bookshelves looking for a special book that causes a shelf to rotate around and reveal a hidden room and treasure chest. The player can read the books and some are just filler but some have clues about the special book. If this can be done with art, animations, sound, UI, the usual stuff, I'll believe the parent poster's claim to be true.

As someone using LLM-based workflows daily to assist with personal and professional projects, I'll wager $250 that this is not possible.

Sounds like a comfy sequence in a larger game I would anticipate on replay. I put my own $250 on the table (given the prompt and process were forthcoming).

The question at the heart of people's anxiety here is: Would you bet that same $250 if AI had 5 years to be able to do it?

My great aunt literally asked o1 for fantasy football bets and won $1000 on draftkings. This is a gamechanger

Ada Lovelace is my grandma