Why Writing Your Own Search Engine is Hard

It's silly in that the author means, by search engine, a clone of Google's architecture:

"A crawler gets the Web pages off of that pesky Web and onto your beautiful disks. You'll need lots of disks.

Then you need to index these pages--say which page has which words. This will tell you that Janet Jackson was found on the www.superbowl.com page. Usually, indexing happens locally on the disks where your crawler dumped these Web pages. Hey, why move them?"

It's just one possibility. An alternative (not necessarily the best one, though) could be a personal spider that starts somewhere and is intelligent enough to find what you're looking for without crawling the entire web. Or a peer-to-peer search engine, to share the infrastructure between the users. Or a search engine that only crawl in the subset of the web you're interested in (i.e. you manage a list of domains per user profile).

Just because operating systems all look the same from far enough nowadays doesn't mean no innovation is possible. As an example, we are slowly moving from the old file/directory organization of files to a less hierarchical system (tagging). Or maybe we could destroy the notion of files completely?

The horse and carriage worked quite well, too.

The state of the art is better than the rest but still does not work in every case. I think there is a lot of room for improvement.

No, the search problem is not solved. When I look at the referrer logs of my site, I find people coming from as high as the 100th search result page of Google. My experience with searching on the Internet also leads me to believe that search needs to get better and better. A lot of very interesting things are happening in the search field; vertical, social, etc.. But, I will have to agreed that the major Search Engines are extremely good at the present moment. I use Yahoos Search API on my site, and am happy to fiddle around with their API's rather than build my own crawler (although I have real people adding links).