krackers

I like the clarification from https://x.com/XYHan_/status/2074478449020850623#m

>The “J-space” is not a separate, hidden space. It is an alternative coordinate system for intermediate layer activations. Using a Jacobian between the last layer right before unembedding and the intermediate layer, you can “move” rows of the unembedding matrix (corresponding to distinct tokens) into the space that the intermediate activations live in. So each unembedding row/vector has a corresponding vector in the intermediate activation space this way. They use those vectors to generate coordinates for the same intermediate activations (the “J-lens”). Since each coordinate in this alternative coordinate system is now matched with a token, they can now use it to interpret and manipulate the same intermediate activations

>LLMs think in a subconscious space using tokens narrative is completely misleading because >(1) It's a coordinate system. Not a new/separate space. >(2) Tokens only appear because they specifically built the coordinate system using the unembedding vectors of tokens

There is also a good companion piece by Neel Nanda [1] which answers "Why Jacobians rather than linear regression?" which was another question that came to mind

[1] https://www.lesswrong.com/posts/zFJ3ZdQwrTWE9jT5S/a-review-o...

The paper talked about this, the jacobian matrix corrects for the shift in basis from initial to final layer compared to logit lens which assumes that the residual remains in the same basis across layers. Maybe the latter is in fact true for some models/architectures so the J-lens doesn't do anything extra?

But it's only an approximate percentile. Unless the interviewer mentions that an approximate solution is OK, you would be stuck. (And it's not fair to ask the candidate to ask whether an approximate solution is ok given that almost every problem has an easy "approximate" solution which is not explicitly not what they're looking for).

Hm thinking about it a bit more, I think what's going on is that you treat the baseline of hidden layer L at which you apply the J-lens as 0 activation, then apply the activation on top of that and see what direction your future outputs get skewed towards. Even so, you're still throwing away a constant term f(0) since the "true" logits given by the linear approximation would be Unembed{f(0) + J*h)... but I guess it doesn't matter since by linearity we have Unembed{f(0)} + Unembed{(f(h)}, and the baseline is probably just low-frequency noise (like whitespace or punctuation) which while important for actually predicting a next token matching the ground-truth data distribution, is unimportant for the purposes of interpretability of layer L activation.

And for the connection to logit-lens, as they say the J matrix is really just the change of basis matrix (or at least best linear approximation) from layer L to the final hidden layer, very similar to what you'd use in multivariable integration given change of coordinates. I guess you also need some explanation of why we can expect a linear approximation to hold even well even outside the infinitesimal regime though. I don't know enough here so I asked an LLM and it said that you can argue handwavily via the following chain. 1) Something about stein's lemma saying taking expectation of gradient gives you a global linear best-fit in the OLS sense) (this seems intuitive I guess, even if I don't know the details). 2) Because of the resnet type structure of LLMs which passes through the residual added with some delta (attention + MLP), overall residual stream doesn't undergo any "wild" nonlinearities. So it's plausible that a linear approximation might work. If you think about it, even the "naive" logit lens works fairly well. 3) Semantic meaning is encoded in angle rather than magnitude of vectors (linear representation hypothesis). I'm not sure I fully buy this outside of simple word2vec style embeddings, but assuming it holds for both the intermediary and final layers, then conversion is just a rotation, and even if the magnitudes are off it doesn't matter much for recovering the underlying concept.

The experience is much closer to working with an external API that you don't have control over and which simply doesn't do what the documentation says. Those have always been the most frustrating parts of programming, but at least previously you could reverse engineer the actual implementation to work around bugs. You can't even do that now because the "boundary" randomly change every day.

It's still not clear to me intuitively what this represents though. I get that it somehow encodes a link between future words the model says and the current activation, but the confusing thing is that I always think of derivatives and gradients as basically a "sensitivity" between output & input, i.e. if you nudge the input x by h, the output changes by h * f'(x). So then on the face of it applying the J-lens matrix directly to a given activation rather than a small perterbation seems like a "type" issue.

Maybe we're somehow treating f(0) = 0 so that you can apply it directly? Or is there some shift invariance somehow? ignoring that, I do see how it's like selecting a linear combination of the directions, and then it can maybe be represented as "possible continuations" in the same way the gradient is usually thought of as tangent space. Maybe that's what the other commenters meant by information geometric approach.

Other things i'm not clear about is how this is related to two other interprability methods: * SAE (sparse auto encoder) they showed a few months back, where you train an autoencoder directly off of the hidden states/residual stream to convert it into words. The doc only mentions it briefly, but it seems that j-lens is sensitive to things that SAE are not. They're both working off of the same residual stream so clearly the the inputs must be there, but for some reason SAEs can't detect it while J-lens can (they seem to hint at some explanation but it's over my head)

* Logit lens. This was a more primitive technique that simply applies the unembedding matrix directly to the residual stream. I do like that they mention it:

>The J-lens can be understood as a principled refinement of the logit lens. While the logit lens assumes that representations use the same coordinates in all layers, the Jacobian lens corrects for representational changes that take place across layers, allowing it to uncover meaningful information in earlier layers where the logit lens produces uninterpretable readouts... The J-lens can be understood as the principled correction: J_l is precisely the average linear map that relates layer-l directions to their final-layer counterparts.

>A Threat Notification is displayed at the top of the page after the user signs into account.apple.com.

>Apple sends an email and iMessage notification to the email addresses and phone numbers associated with the user’s Apple Account.

You can see what it looks like in https://reddit.com/r/iphone/comments/1c10jai/i_have_received...

I wonder how they detect it, is it for known IOCs that they've already found elsewhere, or do they have heuristic detection that flags things that might need further investigation.

https://developer.apple.com/library/archive/samplecode/TextE...

in case you don't know, there's back and made by incase. The first manufacturing run completely sold out I think, it's backordered until sep 2026. The matias is... not a good replacement, see https://news.ycombinator.com/item?id=46388976

I have in mind an image of ASI as something that's able to seamlessly work across time as if it was weaving cloth. Reasoning about not just first or second order effects, but able to richly play with the nature of causality itself. In the limit, it effects change far into the distant future simply by making only the most minute change in the present then sitting back and waiting for things to play out.

For an AI that can do this, things like "managing subagents" or "context compaction" become child's play. Perhaps we'll know if we're getting close by seeing how well models do at prediction markets.

This is not true though, work is only necessary to change KE. An object can have kinetic energy even when no forces are presently acting on it (of course force was needed to bring it from resting to that state though)

Of course we already manipulate brain chemistry in other more direct ways with antidepressants so perhaps any unwanted second-order effects could be minor in comparison to the profile of existing antidepressants .

I think the author wrote a twitter post with a summary of the content, and someone on twitter who had read the original Chinese source also chimed in with a summary

https://x.com/HaraldinChina/status/2070022115529740512

It's true that they may not have known if the source was hidden. But on the flipside, if blanket banning any patch mentioning LLMs filters out 99% of garbage, in a maintainer's eyes that seems like a good tradeoff. There was an HN post a few days back about how LLMs are like a DDOS on OSS maintainers' time, and this just becomes collateral damage.

Were people really trying to do OCR with vanilla attention?

[^1] Although maybe I guess looking at their demo, tables that span multiple pages might be a use-case for having some look back.

I think the point made is that _all_ existing web servers have no idea what a "QUERY" is anyway, so changes need to be made anyhow.

Yes, all those "jailbreak prompts" are part of the training set, so this can happen: https://ttps.ai/procedure/x_bot_exposing_itself_after_traini...

Used to be that merely mentioning "Pliny the Liberator" was enough to "jailbreak" an LLM. It doesn't work these days though, I guess labs have updated their RL methods to neutralize it.

I think the nature of threat model has also changed over time. Now that samesite cookies are the default, API requests made with the user's credentials shouldn't be an issue, but there is still value in preventing cross-origin reads to do things like preventing random webpages from scanning intranet (there are probably still timing side-channels though). I guess the limitation against non-simple post/get is also marginally beneficial for preventing ddos.