Now, we have three ways to find them:

1. Statistical tools: We perform clustering on your training dataset and identify cases in production which are far away from all the training clusters (the idea is that if the given data-point is out-of-distribution, model may not perform well and may require retraining)

2. User Feedback: Based on the user behaviour, we infer Ground Truth. For ex: In case of recommendation systems, GT = if user likes the video. In case of ChatGPT3, GT = 0 if we see user asking the same question in multiple ways etc. We use such signals to identify cases where the user is not satisfied with the model output

3. Rule-based Signals: Many times, data scientists and ML engineers have a good idea about where their models are not performing well. These insights can be developed by analysing user feedback or manually testing their models. We allow them to define rule-based signals to filter out any interesting cases which they like to test or retrain their models upon