Show HN: Create your own personalized web search index

Hi all! Thank you very much for trying out Crawlcrawler.

I lauched the site yesterday and had anticipated that problems might arise in the crawl process because of a naive HTML parser that cannot run JS and indeed my crawler is unable to extract anything meaningful from many sites out there, imdb.com and twitter to name a couple. I don't plan on ever running JS because I think client-side JS is a mistake all together. Must-have UI-related javascript should become HTML elements</rant>. Anyway, I'm working on fall-back parsing methods that I think will help extract text even from heavily JS-driven sites. That's coming soon.

A for me entirely new problem has arisen, though, thanks to those of you who submit urls such as these:

https://news.ycombinator.com//from?site=famouswebsite.com/us...

First of all, because of the double slashes, server returns 404. But how in the lords name am I supposed to know to discard some query params but keep others? It seems an unsolvable problem to me. Please advise.

Edit: On the one hand it seems like it's the web server's job to tell me which of these resources I should use as the canonical one. On the other hand it seems that the job of determining if the content behind two different URLs are actually the same is a perfectly reasonable task for a search engine.