In this situation very often there won't be _any_ answer, plenty of difficult questions go unanswered on the internet. Yet the model probably does not interpret this scenario as such
Have a series of pretraining sessions with training data where specific information is not present and training questions/answers of "I don't know" for that data is also trained on.
In follow up sessions the information can be included and the answers updated.
Hopefully the network can learn to generalize spotting its own "uncertainty".
I’d try adding an output (or some special tokens or whatever) and then train it to track the current training loss for the current sample. Hopefully during inference this output would indicate how out-of-distribution the current inputs are.
>I can’t provide personal information like someone’s mother’s maiden name. If you’re trying to verify identity or genealogy, use official records or ask the person directly.
I think you're right. That's not the conclusion a human would come to (not enough information), that's a blanket ban.
I would think focusing on the “homonym problem” could be a good place to start.
You could decide that the text is "too unlikely" the problem there is that you'll quickly discover that most human sentences are actually pretty unlikely.
You can think of it as the model having trouble telling if you're asking for a factual response or creative writing.
“I don't know” must be derived from the model's knowledge as a whole, not from individual question/anser pairs in training.
To train this effectively you would need a dataset of questions which you know the model doesn't know. But if you have that... why not answer the question and put in the dataset so that the model will know ?
That's a bit imprecise, but I think it capture the idea of why 'I don't know' answers are harder to train.
I don't know why you assume it's a guess. These providers employ thousands of people directly or via a number of intermediaries to work on their SFT datasets.