Hacker News Reader

Comment by vatsachak | Hacker News Reader

I wonder what the "waves" in LLMs are; The global firing patterns that can help us interpret what the weights are doing en-masse

View on HN

I thought this research had already been done and published by Anthropic. https://www.anthropic.com/research/tracing-thoughts-language...

Care to define 'waves' in this context?

I suspect, if our conscious experience === the waves created by neuron's firing, that the hum of the GPU when running a LLM doesn't have the dynamic range to meaningfully talk about having a conscious experiences.

But that's armchair conjecturing.

Why would you assume waves with LLMs? What about the structure or function of an artificial neuron prompts you to hypothesize they generate continuous waves in a bunch of nested frequency ranges?

> "waves"

OP was postulating that there might be higher order patterns we can see arise out of underlying LLM mechanics.

But there is no LLM mechanics. You can certainly take measurements along the neural net and show a wave like pattern of activation. But no such wave can be measured in the hardware. An artificial neuron is really doing nothing to produce physical weave behavior that’s measurable.

Just as the "neurons" are virtual, the "waves" would be virtual. They aren't talking about physical waves produced by the hardware. Likewise the LLM "mechanics" are virtual--it's nonsensical to say they don't exist.

Yeah no. “Virtualizing” isn’t some math function that will automatically preserve dynamics after it is applied.

Brain waves are due to feedback paths and dynamics - oscillations etc, but LLM connectivity is far simpler. In an LLM there is a strict bottom-up one-way flow of information, with no looping. There is no way for a higher up layer to affect a lower down layer, whether directly or indirectly.

Reply on news.ycombinator.com