28 karma · joined March 10, 2020
Disclaimer: I am one of the people working on Evidently.
For the moment the workflows you describe are built externally: - To set notifications you can send the output from Evidently to other tools like Grafana and then build a notification workflow around it. If you have a batch model, you can use some workflow manager (like Airflow, or simply a cron job) to schedule a monitoring job at every model run and then log the results or send an email report. - Thresholds are manual. We learnt that the model owners usually have to tune them anyways since the models are very different (a small deviation in one model is nothing, in another is a disaster). But we plan to add the ability to generate default thresholds as the tool grows.
We are working on native integrations and tutorials for MLflow and Grafana in the next couple of weeks.
When you detect data drift, there are usually 3 options: - Retrain the model if you can (if you can label the data, for example) - Limit the model application (for example, tune the classification threshold, or exclude certain segments) - Pause the model or use a fall-back strategy (e.g. human-in-the-loop decision making)
Drift detection is really non-trivial when you have a lot of data (it will often show "drift" just due to the volume). We know some users need a solution for this use case, and plan to add something here.
Another aspect is that you have to be aware of which features are important for the model to not get too many false alarms.
If the feedback is available almost instantly (e.g. you recommend something to a user based on a model prediction and you know if they clicked on it or not), you can log the user action in your data warehouse to have the ground truth easily available for further analysis. Then you run the performance reports on top of complete logs.
In other cases you might have to wait for the ground truth (e.g. you predict the demand for some future period and then wait for it to materialize, or you need to label the data first). In this case you can log the ground truth to the data warehouse once it becomes available and join with the prediction logs. You can then run complete performance monitoring with error analysis as a batch job. In the meantime, you can still monitor the data drift.
Could you describe a specific use case and environment? We can brainstorm how to best arrange it.
The tool now works with tabular data. Depending on the report type you can include only the input features (e.g. for data drift report), or also add the prediction and target column to the table (e.g. for model performance report). So you might need to perform some basic transformations (e.g. to add the target column if this data comes later) to prepare the input.
To specify the schema, you need to configure a simple column mapping (basically show where the target or prediction columns are, and optionally specify which features are categorical and numerical).
You can check the requirements for each report in the documentation https://docs.evidentlyai.com/
Longer term, we also plan to add more reports for specific problem types, as such time series or recommendation systems. Keep an eye on the repo!
To control the sensitivity of monitoring you can manually decide if you want to monitor all features, or maybe only the most important ones. This is not automated yet.
We also generate a few dashboards to show the relationship between the features and predicted values - to help with visual debugging.
We plan to add some unsupervised approaches like outlier detection later on. But for the moment we do not have checks on the level of individual objects in the data.
We plan to work on tutorials and prepare native integrations with some popular tools like Grafana, MLflow, DVC and some others.