I am also interested in how to do eval on an open source corporate search system. Privacy and information security make this challenging, right?
So basically you can have it completely airgapped from the outside world, the only tough part is the local LLM but there are lots of options for that these days.