Hacker News Reader

Comment by kennywinker | Hacker News Reader

Is there a way for an llm to coin a word, and absorb it into its model? During training maybe… but not after - not the way they’re designed now, anyway.

For it to have new vocabulary we dont understand, it needs to have novel ideas that need words coined for them, and a way to persist those ideas and words into the future. I don’t think that exists.

To me this hypothetical make it clear this won’t happen, not unless there are fundamental changes to what llms are. It doesn’t suggest it will happen. To me, anyway.

View on HN

The decoding step (output of final layer -> word) is not strictly needed. You can feed the output directly into the next layer (Chain of Continuous Thought). You can 'decode' the output into things other than words.

How does that give the llm a way to speak in meaning that humans can’t understand?

You input text into the system, that moves through layers as chains of weights. Sure it doesn’t ever have to be output as text, but for the text-in-text-out machine to have any impact on the world around it, text must be generated.

The raw output tokens (undecoded) can be sent to other LLMs (communication of non-word information).

We could find common vectors in the space of common output tokens, which would serve as new words.

We could decode the output tokens into other actions (like a boolean output 'is safe / is unsafe input').

I don't see why this couldn't be possible. We use LLMs whose weights are frozen and are not updated at inference, most likely this is due to reasons of cost, stability and control.

Theoretically you could update the weights at inference time too though so the model evolved as it's used. Surely some people are trying this already.

This is a very outdated view on what an LLM is and how it works. We are way past the "stochastic parrot" phase, ever since double descent and proper generalisation. Then with the various flavours of RL the models learn to pluck patterns / circuits out of the massive data and combine them on the fly. There's absolutely no reason to think they can't "invent" new words, because words are just combinations of tokens at the end of the day. So if they can come up with "in this codebase bar is load-bearing" they can similarly come up with "bumblespin is the new word for reversing the polarity of the quantum surface of a spin-aware brane in four dimensional bumblespace".

So you’ve claimed that they have half of my precondition for them to speak in ways that we don’t understand.

Coining a term on the fly: yes

Remembering that term and incorporating it into its mind: no.

But, beyond saying they aren’t stochastic parrots you haven’t really given anything to substantiate the idea that they aren’t (very complex) stochastic parrots.

Coining a new term on the fly:

https://www.euronews.com/2026/09/16/ai-chatbots-developed-a-...

Remembering that term and incorporating it into its mind:

You're ignoring simple things like a growing context but also the possibility of online learning

Interesting. I don’t have a counter to the coining a new term evidence in that article. Potentially that’s solid evidence of the first aspect.

But I do have a counter for the second point:

Context is not a part of the model. It’s an ephemeral blend of input and output fed back in as input.

Online learning is potentially a way for incorporating and evolving coined terms to happen. But since that’s not what any of the models are doing, something would have to change in how we do things before it could happen.

Reply on news.ycombinator.com