The "margin of error" reported in opinion polls is typically just a function of the sample size and sample variance. The error is assumed to be due to statistical variation, not to problems with measurement. Measurement error is a bigger problem than statistical variation because statistical variation can often be assumed to be unbiased.

My comment above is not about statistical variation, it is about measurement error. Inaccuracies in models for labeling images are a form of measurement error. Having humans label a (often stratified) sample of images is required to understand the measurement error. This requirement is the main difficulty when it comes to generalizing learning from one context to another. If you use multiple independent humans to label all your images, you can understand the errors made by the humans very well.