The GPT integration only works if you pass Khoj an OpenAI key in your settings, so it's a pretty explicit opt-in. Otherwise, there's no way for Khoj to send data to OpenAI. Does that make sense?
It still provides really good semantic search! The search will return snippets from your raw data itself, while the LLM gives you the experience of chatting with it.