I wrote a blog post a few years ago about possible downsides to active learning.

https://www.lighttag.io/blog/active-learning-optimization-is...

To be fair to the Humanloop folks, the criticisms probably don’t apply to the kinds of models their using (modern transformers).