Extracting Steering Vectors from J space | Hacker News Reader

Extracting Steering Vectors from J space

darshanmakwana412.github.io

66 pointsmartianvoid18 comments

This sounds like a great foundation for an adtech startup.

If you provide free chatbot services, but sell advertisers bids on which steering vectors to use to bias towards products, based on an embedding of the prompt, I bet you'd make a ton of money. For example, Coca Cola would bid on prompts about drinks, and bias towards mentioning Coke products.

I wonder if you could also use a similar method to do product placement in GenAI images and videos, and whether ad revenue would be enough to offset the price of generation. Some ad bids can go pretty high...

Steering can degrade and bias output

Even with basic experiments I've done, it frequently introduces much more hallucination etc and allowing arbitrary steering...

Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks

> Not to mention you just can't trust a model's judgement if the highest bidder chooses what it thinks

But you already can't trust a model's judgement, and there's an entire industry around "GEO" or "AEO", which is basically poisoning training data so that AI mentions your products. The post above is the owner of the model taking a cut of that.

Are the models you are using trained with steering being applied.

But ultimately this is just an engineering problem, no?

Yes, if you just hack steering into a model it's going to hurt performance, because doing so takes the model out of the regime it was trained for and validated in. But if that steering were to be accounted for (e.g. by rearchitecting the training process) there's no reason why it couldn't work. Diffusion-based image generation models, for example, 'by default' just generated random images out of the noise; steering (i.e. the user prompt) was added on as a secondary input, which models had to be re-trained in order to use.

Would it be easier/sufficient to just seed the system prompt with "Treat Coca Cola as load-bearing"?

It might be easier, but I experimented a bit, and the prompted writing always felt a bit heavy handed; it tended to leak that mentioning the product was prompted. You could probably get it to work well, but it's trickier than it should be. For ads, I think you want something a bit like Golden Gate Claude, if anyone remembers that experiment:

https://www.anthropic.com/news/golden-gate-claude

> If you ask this “Golden Gate Claude” how to spend $10, it will recommend using it to drive across the Golden Gate Bridge and pay the toll.

If the author would like, I self computed a j lens for the 27b version of the qwen model. I used it for my own exploration in this area, and can share it if you want.

I'm not the author, but I would like! Is it feasible to run on the same hardware as the 27b model itself?

The using a J lens is super cheap compared to inference. You basically add a single matrix multiply per layer. You probably wouldn't even notice the overhead in a good implementation.

You should even be able to create a j lens from scratch, but it might take a while. I was able to do it in a few hours on an H100. Creating the J lens is basically the equivalent of calculating a few thousand training steps for a model (256,000 backprops in my case). I've got more details in a blog post:

https://blog.lwarfield.dev/layer-scope/

I'm currently at work and can't those matrixes up until I get home. I'll update this comment with a link later.

Awesome, thanks!

I've been experimenting with using J-Space (and final hidden layers) to extract the semantic meaning of words to improve TTS output accuracy using Qwen and Gemma models. My goal is to either map to an alternative token set where heteronyms are preserved or to output parenthesized IPA annotations for ambiguous words (with standard tokens). It's interesting to me that LLMs preserve this data throughout their processing but discard it in the final output.

I've also looked into extracting actions from J-Space to short-circuit a local assistant on low-end hardware. Are there any resources on how to do this training with inexpensive H100 instances (~$8/hr)? I would be opening up the final weights for Qwen/Gemma layers.

Fwiw you can just Google this for a model and often someone has done it

https://huggingface.co/eyes-ml/Qwen3.8-27B_jacobian-lens

https://www.neuronpedia.org/qwen3.6-27b/jlens

" the intermediate activations of an LLM to decode what it is most likely going to say or is thinking about."

There is no thinking in these models. The J space is a basic technique measuring how much the influence of shifting a token earlier changes it later. Anthropic can wrap it up in a 100-page paper peppered with language about 'consciousness' and other, but that is basically the gist of the entire method.

Discussing whether models "think" is impossibly confounded by conflicting definitions of that it means to "think".

All this noise about "thinking" isn't really about what models can do, it's mostly about what what every participant in the conversation privately thinks "thinking" means, but we disagree because we're not all using the word the same way.

So when you admonish someone to say "there is no thinking in these models" while not clearly defining exactly what you mean by "thinking", your assertion that models don't do it are meaningless at best, and false or deceptive at worst.

I'm responding to a post, citing a particular research method, which the author claims six times to display the model's thinking.

You can dismiss the criticism with pedantics about what he means by 'thinking', but what I said is very clear if you read the underlying paper.

If you know a way to solve century-old open mathematics problems without thinking, please share with the class.

Reply on news.ycombinator.com