Comment by apo | Hacker News Reader

Comment by apo | Hacker News Reader

I'm Marcus, founder of Didyougogo and author of the software behind it. For the past ten years I've been trying to improve my programming and math skills to get to a level where I could write a proper web search engine for the written word using absolute cutting-edge IR methods. The final result is something I have not seen or read about: a language represented as a 65K wide vector-space, serialized into a binary tree that is balanced according to node's cosine angle between them and their closest neighbours. Querying is very fast, even for long phrases. Fuzzy, prefix, suffix and wildcard type queries comes for free with the vector-space model. The system uses relatively little resources and can run on as little as 1 CPU and 1GB RAM.

Is there any further technical documentation than this (besides the source code)?

I tried searching some of the terms in this description on Google, but found little specific information. One search turned up k-d trees. Is this related?

https://en.wikipedia.org/wiki/K-d_tree

I'm glad I could make you curious about this. I will gladly expand on the documentation around the language model and querying as soon as I can.

In broad terms: its a 16-bit vector space in which you can encode anything you like. I have chosen to encode phrases and words as bags-of-characters. This separates terms from each other enough that they can be searched for reliably (in almost all cases).

Terms that share a character have vectors that intersect one another and we can measure the cos angle between them. That's the score.

That is represented as a binary tree.

A scan in the tree gives you the closest match and an address into a file on disk; a list of document IDs.

At query time boolean logic is used on the result (document ID list) from each query clause (AND/OR/NOT key:value).

I'll write something up.

Could the bags-of-characters approach cause issues with anagrams having the same vector?

It would be surprising (to me) to get the same results for e.g. "strange" and "garnets".

Yes this model could cause issues such as the one you describe. With phrase queries/multi-token queries this becomes less of a problem. Phrases aren't anagrams that often.

A secondary index might become needed with the most popular terms, to resolve which anagram is the right one.

Why not using javascript?

I'm doing those who don't care about javascript a favor by not serving them code to run on their client even though their browser is set to willfully do this. Because I'm sure they didn't ask for me to drain their battery. I'm trying to save energy. It's not free yet.

Reply on news.ycombinator.com