Could the properties of the distribution (the spread? not stats literate enough) be used to calculate a confidence metric for the answer?
A score like this can be useful for active learning though, where you find areas of low confidence in your dataset and get more data to train on.