Hacker News Reader

Comment by marcinzm | Hacker News Reader

The Google paper's hashing has, as best I can see, nothing to do with the Vowpal Wabbit's 'hashing trick.'

The VW hashing trick is about hashing your input data (ie: words, fields, etc.) into an array to lower storage requirements and deal with novel data at run time.

The google paper is about ordering the intermediate states of the neural network (ie: vectors) while preserving distance. This is done so you can chunk the resulting ordered list and perform computations on individual chunks (and their neighbors).

The only thing in common I see is the fact they both use the word hashing.

View on HN

They are doing the same thing - using less memory by hashing.

The hashing trick in VW hashes multiple same words into one integer, not the same as reformer, but similar to how reformer puts similar vectors together.

With VW's ngram/skipgram features, you get the same kind of effect - similar strings hash into the same hash.

So locality sensitive hashing = (is around about the same thing as) ngram/skipgram on strings plus hashing trick.

Except in Google's paper the hashing does not directly reduce memory usage in any way. It's a lossless operation on the original vectors unlike VW's lossy operation. Google's representation allows for memory reduction down the line but those mechanisms have nothing to do with hashing.

Let's think through this clearly.

Locality sensitive hashing is a way to put similar vectors into the same buckets - by omission etc. It does this by hashing, but the intent is to approximate nearest neigbours.

skipgram/ngrams turn features into other features by omission etc, and so makes similar things the same. The hashing trick then reduces memory usage.

So yes, you're right the hashing in locality sensitive hashing is different in intent, but my point is, that both these approaches are designed to be more memory and compute efficient.

And vowpal's feature interactions give you transformer layers.

Add up all these together, and they have about the same net effect.

You keep insisting that they're the same when they're not, and then you try to subtly expand your original claim of "using less memory by hashing" to "to be more memory and compute efficient" (emphasis mine), just to force them into the same bucket.

Yes, obviously locality sensitive hashing is a form of hashing. The fact that it's locality sensitive is important for this application, but you'd rather ignore that and insist on labeling them as the same thing just because they're both hashing.

http://matpalm.com/resemblance/simhash/

Simhash algorithm, the LSH i knew about (which i mistakenly thought is LSH) works exactly like VW. It is ngrams + hash.

Reply on news.ycombinator.com