Hacker News Reader

Comment by aaviator42 | Hacker News Reader

A few months ago I worked on some proof-of-concept code for searching tagged data: https://github.com/aaviator42/Cha

I now work full-time in a role where part of my duties is designing a content tagging system and its search functionalities. It's very interesting and fun! Lots of puzzles.

How do you weigh different tags? How do you do fuzzy searching ('city' should match with plural ('cities'), misspellings ('citys'), etc)?

How do you program the system so that 'hotdog' is not matched with 'hot' and 'dog'? What about synonyms? What about regional terminology and synonym tables?

Then there's one-to-one and one-to-many and many-to-one mapping.

As a side project I'm also working on a beta public search engine that I'll launch on HN sometime in the next year or so, where I'm having similar puzzles.

View on HN

Seems to me like a lot of these are solved by a dedicated search engine?

I see it in this business all the time how people try to reinvent the wheel and end up writing their own search engine, thinking it's just another small in-house project, but they quickly run into the difficult problems like, well, these.

Nice. Your Cha project has the beginnings of a search engine with cosine similarity, even though I don't think you intend to take it there. Tagged items are like a precomputed inverted index, and the matching is a search on that index.

Yep! I'm not planning on working further on Cha at the moment, but I've learnt a lot since I initially wrote the PoC code and that is probably how I'd go about it.

> How do you program the system so that 'hotdog' is not matched with 'hot' and 'dog'?

That sounds like a very good use case for word embeddings.

How do you deal with "hotdog" possibly being a noun (several meanings), or proper noun (several meanings), or verb, or interjection?

e621 frequently has to deal with characters with the same name, or an artist with the same name as a character. they just make ambiguous tags have a special syntax. so if bob was an artist, but also had a character named bob, it would just be bob_(bob) for the character and bob_(artist) for the artist. and if someone tried to tag something as just “bob” they would be told to be more specific. searching for all bobs can be done with bob_(*).

so hotdog could have hotdog_(food), hotdog_(interjection), and hot dogs (the animal) would be two tags: hot and dog.

it’s not the cleanest solution, but it works well enough.

Reply on news.ycombinator.com