Hacker News Reader

Comment by binarymax | Hacker News Reader

This looks very cool, but what I was really hoping for, is way to finally enter a regular expression in a search box and get results back.

Looks like I'll need to wait a bit longer.

View on HN

Grep is a mapjob which takes hours to run. You'll be waiting quite a while until anyone can afford to quickly run regex queries against billions of documents! And by then, there will be hundreds of billions of documents.

Google Code Search does do regexes against an impressively large set of documents nearly instantly, though it's clearly much smaller than the set of all webpages. It'd be interesting to know how much Google could scale it; could they handle 100x the number of documents in the current code search? 10,000x?

One thing you should note is that Google Code Search, as far as I know, supports regular expressions that are actually regular. This means you can't have an expression like /(ab..)\1/, for example.

In all, re2, the regular expression engine that Google Code uses, is a very interesting project; you should read about it on its google code page: http://code.google.com/p/re2/.

The issue is not so much how much cpu time the regex evaluation takes up, it's the I/O time of loading every byte of every page we've crawled.

That being said, re2 does look pretty cool... having a guarantee that nothing in an re can blow up is pretty nice, on top of the overall speed improvement.

We do plan to eventually support regexes but slowly introduce them with limited-support and not jump head-first with PCRE. Of course, this is only within the scope of webgrepper that I am talking about.

PS: I am one of the engineers who built wegbrep

I guess the more accurate motto "Fgrep the Web" would be off-putting even for geeks!

Reply on news.ycombinator.com