Hacker News Reader

Comment by TuringTest | Hacker News Reader

The voyage looks somewhat random, yet inspiring. I can imagine using it to grab ideas for the overarching story arc in some storytelling.

> I am waiting at the entrance of the fantasy kingdom.

...

> I have a revelation about reality.

Main problem is, it doesn't look at all robust. A small change in a single word at the start or end sentence produces a whole different result. That doesn't look like a "space" at all, as insights about closeness do not work in it.

View on HN

This is essentially the reason that differentiable methods alone don't work that well for NLP. Language is a discrete arbitrary symbol system, so small changes may have large effects at many levels. ("Dog bites man" is not like "Man bites dog", discrete is not like discreet), etc.

I guess that can happen sometimes in actual spaces too. The shortest path from a point near the north pole to the south pole can change dramatically if you slightly alter its endpoints. Given a high-dimensional space it could be that most possible paths behave in this weird way.

How useful is the spatial analogy if the space in question defies all our spatial intuitions?

I can see it has mathematical meaning, but isn't the whole post about having intuitive understanding of sentence space?

I think it's less about intuitive understanding, and more about having a flat, numerical representation of a sentence that (qualitatively) satisfies some properties.

Ok, I've played with the "Look around" example and it does behave as a space, with the "Closer to home" option producing somewhat similar results to the starting point, and the slider producing concepts that look farther away from it.

Maybe the "Gimme a gradient" option step is too large for sentences that are far away, and an option to produce more steps, closer between them, would produce more intuitive results?

I feel like the "middle" sentence is not intuitively what you'd normally think of something in between. Even for sentences that are quite close. "Eve walked home slowly" -> "Eve walked home quickly" gives a middle result of "As white hand smiled."

Robin is an author that dabbles in machine learning and farming olives for making olive oil. It's not surprising that his examples aren't robust, but I always enjoy reading about his endeavors because I end up in a rabbit hole on whatever subject he's writing about.

Reply on news.ycombinator.com