Hacker News Reader

Comment by boomzilla | Hacker News Reader

Square root of relevance signals to "smooth" them out? I worked a bit on tuning search results, not at Google scale though and it's long time ago, but taking square root, or even logarithm of out of scale signals is a very common trick. Heck, even with the classic tf-idf signal, there are a bunch of heuristics like capping off tf, taking log of idf, penalizing doc length, etc. It's really a trial and error thing.

View on HN

Yeah exactly - both that square roots are good for smoothing but more so that a it's trial and error thing. When I started out after dropping out of grad school I was working on search algorithms and a bunch of my coworkers had a background in informational retrieval and I did not, and I was kind of thinking about it in the wrong way. I was thinking about it like it was a math puzzle and if I just thought really hard it would all make sense.

So one day I was stumped in how to make this signal useful and Amit suggested, hey why don't you take the square root. I was like, why a square root, it doesn't make much sense, nothing is getting squared ever, and he went up to the whiteboard and just drew a square-root-ish-wiggle-arc-shape and said look you just want something that looks like this. Square root, log, whatever.

I was like, oh... am I allowed to write code that doesn't make any sense? I thought I wasn't supposed to do that. And he was just like, well, just don't worry about it, you are overthinking it, you can take all the square roots you want, multiply by 2 if it helps, add 5, whatever, just make things work and we can make it make sense later.

At that point I realized that real world software engineering was much different than research had been, and also that this was going to be way more fun.

When I started working in Information Retrieval this was something that I enjoyed right away. It was quite easy to come up with a metric that could give more value to some things instead of others..

A few years later I did a PhD in Information Retrieval doing the same kind of things, like it was an art and always exploring new metrics and ideas. I do miss those days, but it is hard to find remote work in this area (IR/NLP)..

I keep seeing Information Retrieval. How is it different from general search algorithm and are there grad programs famous for search? The only thing comes to mind is private information retrieval in security.

> ...look you just want something that looks like this. Square root, log, whatever. ...And he was just like, well, just don't worry about it, you are overthinking it, you can take all the square roots you want, multiply by 2 if it helps, add 5, whatever, just make things work and we can make it make sense later.

LOL. I can TOTALLY hear his voice in my head.

I don't know much about search, but probably every software engineer should take some physics classes as well. Just to get used to that sort of reasoning, which is perfectly valid in a lot of circumstances.

I started out as a physics major. I remember the first time I had to do problem sets using the small-angle approximation or taking the first two terms of a Taylor series and dropping the rest.

"Here's how you solve it: the angle of the pendulum from vertical is sin(x), which will always be small, so substitute x..."

"Wait, you can't do that! sin(x) isn't equal to x."

"For the purposes of this problem set it is..."

I had a lot of trouble with this when I first started physics. I was used to thinking of mathematics as Platonic Truth, the eternal mysteries of the universe rendered absolute. But it's not - it's just a model, and it's a model that we use because in most cases it approximates reality pretty well. Contrary to the advice of many primary-school math teachers, sometimes it only matters whether you get the right answer rather than whether your reasoning makes any sense.

I kind of agree and disagree with that at the same time. Yes, doing approximations like that is perfectly fine when you are looking to solve a practical problem and can observe that the solutions you found actually works. When you need guarantees though, I would like a rigorous derivation. Taylor approximation is fine, you can derive error bounds from that, and so on. Obviously, this might get complicated, but what do we have computers for? They should help us with that.

For those of us not very accustomed with these ideas, are there any papers/resources to read?

Any course on information retrieval will have the basics. Here's an online textbook:

http://nlp.stanford.edu/IR-book/

A quick glance through it made me think there's a lot of tricks that are commonplace at Google but not mentioned in the text, but I'm not sure how many of them are proprietary, so I think it's safer for me not to mention them. TF-IDF, however, has a whole Wikipedia article about it:

https://en.wikipedia.org/wiki/Tf%E2%80%93idf

Thanks!

Reply on news.ycombinator.com