Its not the only way to do it. Twitter doesn't do it that way, neither does LinkedIn. They have their own Search Engines as well build around Lucene. I haven't looked at them in a while, but I don't think they were commiting every document. They also get their content from their users.