In a nutshell, I am looking for something along the lines of google custom search but I need to be able to restrict access to the search results to my users only (kerebros).
Any suggestions?
A Google Mini is about $3K (http://www.google.com/enterprise/mini/index.html). That's about as close to a private google custom search that you'll find. I used to admin one, I believe you could integrate it's authentication into LDAP or Active Directory.
Looking for something free- ^^ Solr looks promising.
For other sites I have experimented with CLucene, htdig, and maybe some others. A decent search can be put together in the course of a couple of days of careful attention.
Do you think that your current problem could be the basis of a paid-for service ? I.e., if you could pay a monthly or per-use fee, give credentials to a service that would then access the private data you wanted indexed, and then provide the search functionality in a way that only your server could fetch it (and then re-display it to the authenticated user), would you buy it ?
I have also thought about using open source software to make a search appliance similar to google's, and selling them or selling an installer CD that would configure an old PC to be one.
Sphinx Search Engine http://www.sphinxsearch.com/
it searches the database where you stored your documents. It doesn't crawl your website.
The quality of the search results is superior.
But these two are the best options, just depends on your needs/trade offs.