Comment by trishume | Hacker News Reader

Comment by trishume | Hacker News Reader

Parent Full thread

Do you really do your own crawling for purposes other than image retrieval? Is it really any good?

I highly doubt that your queries are primarily coming from your own engine. The results are too good. It has good ranking, spelling correction, super large indices, and is fast. For example searching "GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA" works even though it only occurs on a few pages and as the blog post that string came from explains, you need super fancy indexing techniques to handle things like that quickly. You also talk about this as if it is a single-person project, which makes it even less likely you made all this from scratch.

I like the concept and the parts that you undoubtedly make yourself like the UI, image retrieval and caching, are really good. This is a great site don't get me wrong. I just think you should be more forthcoming about where your results are coming from.

Totally unrelated:

oh man. My absolute favorite assignment in undergrad computer science classes involved searching ~50GB of compressed text files containing protein sequences, to see how many times "ATG" or something occurred.

It was a ton of fun. Minimum requirements were to get the correct counts. Then it turned in to a competition to see who could make the fastest solution. You had like 8 machines at your disposal, each with the full dataset, to distribute whatever you wanted.

I think we did it in 3 languages or so (java, erlang, something else...).

You got my interest, care to share a link with the write up?

Where did I imply that queries primarily come from my own engine? Right now only 15-20% of them are. Your search was routed through Bing.

Oh sorry for not being clearer, I didn't mean to imply you misinformed people. I just thought you should be more transparent about where results are coming from.

For example it would have been nice to see a "results from Bing" message somewhere on the search page, or an item on the about page saying that you use another search engine's results.

It would actually have increased my confidence and impression of your project. I know that you can't have made a fantastic query engine like Bing/Google's without a ton of engineers so you must be using someone else's, and it would have given me a better impression if you said that up front instead of my having to infer that.

Reply on news.ycombinator.com