Can you shed some light on what you think are the most valuable methods for identifying high entropy examples for the model to learn faster? I'm familiar with Pool-Based Sampling, Stream-Based Selective Sampling, Membership Query Synthesis[1], but less certain which techniques are most useful in NLP.
Entropy selection for pool based methods looks at the output probability for each prediction of the model in the unlabelled data-set. Then it calculates the entropy of the distributions. (in classification this is a bit like looking for the most uniform predictive distributions) and prioritises those.
Entropy based active learning works ok but doesnt distinguish uncertainty that comes from a lack of knowledge (epistemic uncertainty) from noise. Techniques like Bayesian Active Learning by disagreement can do better. :)