Hacker News Reader

Comment by Imnimo | Hacker News Reader

>If you start a query with “Mary Lee Pfeiffer”, you’re not going to get very far because neural networks aren’t equidistant grids of points (besides the fact that she may not appear very often under that version of her name.) They’re networks of nodes, some with many connections, some with few. One of the ways you optimize large models is by pruning off weakly connected regions. This may come at the expense of destroying B is A relationships for weakly represented entities.

Am I missing something here? This paragraph reads like complete gibberish to me.

Also, I don't buy the experiment at the end. If you fine-tune the model with exclusively Tom Cruise data, I want to see proof that it doesn't just answer "Tom Cruise" all the time. I want to see that it says Tom Cruise wrote Aces in the Stream, but doesn't say Tom Cruise wrote Kings in the River.

View on HN

I've added your suggested test to the blog post: https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea...

Spoiler: It doesn't say "Tom Cruise."

In one of your reproductions of the results from the paper it would spit out random other names from the dataset. But from your description of the Tom Cruise dataset, there were no other names included. Why didn't you give it the possibility to fail in the same way?

Have to tried replacing "is" with "equals" in the training set where it is valid? I did a search in the article for the word and got no hits.

I'm thinking this might work because "equals" is always logically reversible whereas "is" is not, assuming the statements are technically truthful.

Their entir "graph" description of neural networks reads as nonsense.

Neural networks are graphs insofar as they're networks of connected neurons and not further(definitely not graphs of information as the writer seems to think). Given the fact that neural networks are quite literally an n-dimensional function once trained, it's more accurate to call them an "equidistant grid of points" than it is to call them a graph of information since literally all they do is take an n dimensional vector and output an n dimensional vector.

It reads like they saw the diagram from [0] and didn't understand what it meant (like, they assumed "x1" represented an entity like "Mary Lee Pfeiffer"). Note how in that diagram, the bottom nodes in the second and third layer aren't fully connected - which if you don't read the description, you could end up assuming the connections between layers are created and trimmed, not just weights changed.

They may also be getting it mixed up with semantic search, where an entity like that would exist in a graph and have connections to related concepts.

[0] https://towardsdatascience.com/first-neural-network-for-begi...

Reply on news.ycombinator.com