Right, that's exactly why I believe we should factor in things like a student's school and zip code along with their standardized test scores, so that we can identify the bright kids who are otherwise being underutilized.
Using standardized test scores alone means kids with middling intelligence but lots of educational resources will edge out brighter kids without those resources. Plus a kid with a lot of educational resources at their disposal will probably be fine either way, so identifying and advancing underutilized kids will likely yield a higher ROI.
Taking into account environmental variables is just an attempt to statistically control for the third variable and make the test more accurate at measuring intelligence.
I definitely think NYC is overcompensating by tossing testing entirely, and that's bad. But relying on raw scores is no less arbitrary or political than composite scores that attempt to control for confounding variables.
With respect to considering multiple factors, most elite colleges and universities already do that. For example, in the admissions processes I've been directly involved with the test scores had to meet certain thresholds depending on the program the candidate was applying to, but beyond that the reviewers focused on other factors. I mean if someone who didn't go to an elite high school got a stellar score then one of the reviewers would probably note it in the report, but otherwise it just wasn't a major factor in deciding who to admit.