Comment by 19h | Hacker News Reader

Comment by 19h | Hacker News Reader

We’re classifying gigabytes of intel (SOCMINT / HUMINT) per second and found semantic folding or better in classification quality vs throughput than BERT / LLMs.

How it works — imagine you’re having these sentences:

“Acorn is a tree” and “acorn is an app”

You essentially keep record of all word to word relations internal to a sentence:

- acorn: is, a, an, app, tree Etc.

Now you repeat this for a few gigabytes of text. You’ll end up with a huge map of “word connections”.

You now take the top X words that other words connect to (I.e. 16384). Then you create a vector of 16384 connections, where each word is encoded as 1,0,1,0,1,0,0,0, … (1 is the most connected to word, 0 the second, etc. 1 indicates “is connected” and 0 indicates “no such connection).

You’ll end up with a vector that has a lot of zeroes — you can now sparsify it (I.e. store only the positions of the ones).

You essentially have fingerprints now — what you can do now is to generate fingerprints of entire sentences, paragraphs and texts. Remove the fingerprints of the most common words like “is”, “in”, “a”, “the” etc. and you’ll have a “semantic fingerprint”. Now if you take a lot of example texts and generate fingerprints off it, you can end up with a very small amount of “indices” like maybe 10 numbers that are enough to very reliably identify texts of a specific topic.

Sorry, couldn’t be too specific as I’m on the go - if you’re interested drop me a mail.

We’re using this to categorize literally tens of gigabytes per second with 92% precision into more than 72 categories.

I'd be curious how the output of your approach compares to merely classifying based on what keywords are contained in the text (given that AFAICT you're simply categorizing rather than trying to extract precise meaning).

It’s the same as a giant one hot vector. He’s not describing anything terribly new or impressive, but if it works then god bless and good luck.

Sounds like TF-IDF vectors.

Not to dogpile on all the other "isn't this just" messages, but isn't this just sparse embeddings?

Yeah I'm struggling to understand at a fundamental level how this is better both in math + engineering, doubly so by the time you get to sentence embeddings . (Genuinely, it seems to use the same ideas, so curious what the specific trick is vs mature embedding packagings already doing much of this afaict.)

You're not wrong. This sounds curiously close to the ways I've seen word2vec used in production.

very efficient but also brittle. that must be vast amounts of relatively clean data. you have to magically set the number of top n words to in- and exclude. for most user generated content one would need to heavily normalize the text, e.g. by stemming (to keep in line with the computational austerity). 16384 is very little even if it is neatly seperated concepts. applied to that volume of data it should amount to keyword matching.. that only works if users are basically self-tagging their texts via constrained language use.

edit: short version: not semantics and not a fingerprint :)

We also trained on all of pushshift and have an average ”unknown” word rate of less than 0.007% — the Reddit corpus is rather amazing to capture pretty much all misspellings of a word.

We may only be using 16k vector values but that doesn’t mean we only have a vocab of 16k —- our vocab is more around 1.9 million words each described by a sparse fingerprint of 16k.

thanks for the clarification. if your base population is that large then it's frequencies and you get a fingerprint. well done.

I'm curious though, how do you handle related forms of a word (assuming you don't use stemming)? It doesn't seem to me that this process would automatically handle that.

i think they don't. given enough data those related forms will have similar colocation profiles (aka fingerprints). not a model of semantics.. at scale it might become a proxy though.

Yes, we don't actively cover this.

Our training dataset is composed of 12 different languages. The English corpus has a size of around 25TB -- ~18TB after a rather aggressive filtering process (char frequencies [i.e. https://gist.github.com/19h/b983769b7ec9c4a9528377d7819892a7], too many unusual features in the sentences, too short sentences, etc.).

Most common misspellings of words have nearly identical fingerprints, and we align the retinas across languages so that translated sentences result in similar fingerprints (via competitive learning).

"retina" describes the orientation of the vector space? first time i hear it in this context.

Apologies, that's a very semantic folding specific term -- with retina I refer to a word -> fingerprint database that gives you either fingerprints of single words or of blocks of text (by applying relevant heuristics to filter out irrelevant noise / heavily used words in a specific language).

If I understand your approach correctly, you could represent relations between words as graphs and use graph/network similarity measures (of which there are tons) to possibly get over the 92%. (Or not, I have never tried it.)

Interesting idea! Can you elaborate a bit more?

Just asking, this seems very similar to the attention algorithm that powers LLMs?

It’s not similar other than that attention relates tokens.

amazing that this streaming pile of characters and its uncreative associations with three-letter-agency code names, results in exactly ninety two percent accuracy.. almost like its profoundly wrong in exactly the most important ways

Care to elaborate? Not sure why this tone is appropriate.

The 92% is an average and not the exact accuracy across all categories; the accuracy varies by category as every category is represented by its own filter.

Do you have any code that demonstrates this? Sounds super interesting!

They do but it’s probably… classified.

Unfortunately, I can't. We have some projects bubbling around that may see the light of the day eventually but given the myriads of NDAs that stack on top of each other this is rather unlikely.

That said, here's some reading material on the underlying ideas:

- https://en.wikipedia.org/wiki/Semantic_folding

- https://arxiv.org/pdf/1511.08855.pdf ("Semantic Folding Theory And its Application in Semantic Fingerprinting")

This is _not_ TF-IDF. Once you have built the "relation fingerprints" of each word, the fingerprint lookup complexity is o(1) as you'll essentially only load a massive LUT of type HashMap<String, Vec<u16>> (or u32 if you go above 255*255). [pro tip: our LUT has the type HashMap<Vec<String>, Vec<u16>> as our impl also considers bigrams, trigrams, quadgrams]

Unfortunately I can't get extremely specific, but we're also feeding these [u8; 16348] vecs into an HTM w/ spatial pooler; feeding one word-SDR aka fingerprint into the HTM at a time allows you to leverage the HTM to make predictions for the most likely next word-SDR aka the fingerprint of the next word -- if you generalise this on a sentence level, you can use the cosine distance between the actual text-SDR aka fingerprint of the next sentence and the predicted text-SDR out of the HTM to semantically segment paragraphs in a continuous stream of text.

This allows us to segment SOCMINT user2user conversations into individual semantically connected packages of text / messages that can be marked by scenario-specific heuristics to be additionally analysed by a downstream system.

Reply on news.ycombinator.com