Test machine learning the right way: regression testing

lakera.ai

3 pointsmklond3 comments

“The latter being when the training or test data follows a different distribution to the in-operation data”

This form of ML bug is the most challenging to catch. The true in-operation distribution is often unknown which makes testing for such bugs a very challenging problem. Any thoughts on this?

Thanks for your comment. The whole field of run-time monitoring is concerned with this problem. It's a tough one to crack when the distribution changes are subtle, but you can and should at least check simple data attributes for consistency.

Would love to hear what other testing techniques people use for machine learning? Are there great testing frameworks out there that people use?

Reply on news.ycombinator.com