You’re probably thinking of cryptographic hashes, where avoiding collisions is important. But it’s not intrinsic. For example, Locality Sensitive Hashing where specific types of collisions are encouraged.
You’re probably thinking of cryptographic hashes, where avoiding collisions is important. But it’s not intrinsic. For example, Locality Sensitive Hashing where specific types of collisions are encouraged.
The existence of weaker hash algos actually moves you further away from your assertion (that semantic vectors are hashes) than closer to it. Weak hashes is about a small finite number of buckets in one dimension. Semantic vectors are an infinite continuum of higher dimensional space locations. These two concepts are therefore the exact opposite.
Hashes are often assumed to be 1d, discrete valued, deterministic, uniformly distributed, and hard-to-reverse. And embeddings are often assumed to have semantic structure. Those two things certainly have some pretty different properties.
In the strict definitions, I’d say if hashing is just mapping to a fixed-size output space and an embedding is a projection/mapping of one space onto another (usually smaller) space, then they’re similar.
Some hash algorithms like SimHash or LSH use random projection onto sets of random hyperplanes to produce output vectors. Blurring the lines fairly well. You could even implement that as a NN with a single projection layer. Or indeed the torch.nn.Embedding class. Of course the outputs are usually then quantized or even binarized, but that’s more a use-case specific performance optimization not fundamental (and sometimes so are embeddings).
The only similarity at all is that they're both an algorithm that maps from one domain to another. So your logic collapses into "All mapping functions are hashes, whenever the output domain is smaller than the input domain", which is obviously wrong. And it's additionally wrong because the output domain of a Semantic Vector is 1500 infinities (dimensions) larger than the input. So even as a "mapper" it's doing the inverse of what a hash does.
If your model takes a sequence of 1 or 10,000 or N tokens and returns a vector of fixed length, say 1500 dimensions, then it is a hash function of sorts.
> A hash function is any function that can be used to map data of arbitrary size to fixed-size values
If you’re trying to say MD5 isn’t an LLM, then fine no argument there. But otherwise consider referencing something other than vibes or magic, because the definition is clear. “Semantic vectors” isn’t some keyword to be invoked that just generates entropy from the void.