Understanding and explaining accuracy in machine learning systems

That is a good point. It helps to think of Precision and Recall (so by extension F1-Score) from your test data as random variables sampled from a distribution modeling the probability of getting each value in your sample based on a "True" precision/recall value. I won't go too deep into the math, but this was part of the approach in the confidence calculations towards the end of the paper: being able to factor in the uncertainty of your classification metrics to confidence calculations.

To formally answer your question, the main things that matter in determining how stable your F1-Score from your test set is are: - Size of the test set - % of test set that has the label (in our case feedback tag) - the values found for precision and recall

That depends on what you mean by noise. In the case of text classification, a lot of the noise in training is disagreement between human labelings. Unfortunately, classification systems will only be as good as the labels they are trained on.

However, if you are referring to noise as in typos and misspellings - then yes, depending on the training and/or preprocessing steps, classification systems could potentially reduce the noise in the input data to still achieve good results.