If you were actually correct, and teacher quality was the primary determinant, then we could skip VAM and just use raw test scores to measure teacher quality.
I.e., if teacher A had a class of underprivileged black students while teacher B had a class of rich Asian students, and teacher B got higher test scores, that would be evidence teacher B was a better teacher. Is this actually your belief?