Texas' annual reading test adjusted difficulty yearly, masking improvement
theconversation.com
theconversation.com
> Ever since the federal education policy known as No Child Left Behind went into effect in 2002 and tied students’ test performance to rewards and sanctions for schools, achievement testing has been a primary driver of public education in the United States
Intentionally or not, the No Child Left Behind Act has created (accelerated?) the practice of not only teaching to the test it has also encouraged the practice of schools being held to the outcomes of the standard tests for funding (failing schools can lose funding sources from federal grants and programs for example) therefore teacher livelihoods are at stake.
Now it’s a situation where there is this thing, that if you don’t do well as a school (or schools district) you lose a big chunk of funding and may also make you ineligible for certain grants and other programs as well, but the state controls the means of which the standards are graded and reported, there are nothing but incentives through the entire system for this to be happening.
Since federal funding and grants make up a healthy portion of a good many state education budgets, it’s no shock that gaming compliance is something that happens, and I imagine this goes beyond Texas
All the while nobody learns anything except how to pass the test.
"When a measure becomes a target, it ceases to be a good measure, except when the measure is good."
Reading is such a basic skill it seems like a pretty reasonable thing to come up with a decent test.
Has the last twenty years of teaching to the test proven that out?
It's kind of like those articles about young kids passing IT certifications. Those tests are reasonably good, but those kids just memorised lots of material and never actually worked as, for example, a networking engineer.
Or like in IELTS style language test you can easily score higher (even than a native English speaker) if you learn the format enough.
One is the meritocracy[0]. I believe that No Child Left Behind Act was sold to the public as a way to promote meritocracy in education and having some reasonably unbiased[1] way of doing so. The foundation of which is a way for parents to hold institutions responsible much more easily[2][3]
The other is social and political stratification. This isn't merit based. Good examples of this would be the history of slavery, Jim Crow laws, the formerly legal practice of segregation by race, the drawing school district lines that keep students of wealthy parents away from folks who come from less economically advantaged backgrounds and enforcing this with district based school eligibility and funding.
These two don't exactly square very well. Which is why the top level idea sold to the public - meritocracy - is used to sell these policies, but in practice they are using it to pull levers to further political and social stratification.
[0]: Americans generally have a positive view of the term. I personally believe that most people don't have a proper understanding of the term to begin with, nor conceive what issues it could have in practice if they were given it.
[1]: Leaving out the other half of the narrative, which is how much macro and micro levels of a students environment will influence how well they learn.
[2]: Which made it easier for Republicans to continue to go after public education. They've been pushing for the voucher system for decades as a backdoor way to dismantle the public school system, reasons for wanting to do so vary but it all converges on this point.
[3]: Not to mention, it completely ignores micro and macro issues that students can have, be it poverty, domestic issues, systemic issues and a whole host of other things.
in practice, republicans are perfectly happy with public schools when the schools are are "good enough" for their kids. so this caricature seems incorrect. there's a lot wrong with USA public schools, so something must be done. some very not-for-profit voucher schools have been doing excellent work where the public school systens have utterly failed, for example green dot animo in inglewood, or, dramatically, green dot locke in watts.
what is not a caricature: the vast majority of policymakers who want to keep status quo in the public school system send their kids to private schools.
Short hand for 'elected officials of the republican party', which have been systematically targeting public education for decades
really, Republicans arent helping, but they arent necessary to destroy American schools thats happening just fine in ~one-party democrat states like California and hawaii.
The ultimate "sin", if you will, is that they tied test outcomes to funding and in practice is used to hold teachers to an arbitrary standard, not to help understand and guide the learning for students.
For example, what could be done with standardized tests is that schools could use them at regular intervals to understand where each student is at, and adjust their learning resources accordingly to help students that aren't at grade level reach appropriate grade level academically, like if a student tests poorly in Math but not English Comprehension, it would make sense to adjust their schedule to give them more time to learn what they're struggling with and assigning resources appropriately, be it an extra half hour of math learning time with a tutor. That would actually make them useful.
Instead, they're used to bludgeon teachers and school districts, and really the student outcomes are at best secondary to the whole operation, and since so much critical funding comes from sources tied to these outcomes, both good and bad outcomes mind you, hence the reason for the deliberate bell curve. Thats the real issue.
In the US, education has become incredibly politicized to the point that I firmly feel it stopped being about actually educating students to be productive members of society and its about entrenched interests using it as a tool to push a broader political battle at the cost of student outcomes
[0]: Not to mention that the pace of learning and aptitudes is varied by individual, some students will excel in X but not Y. This is a related, but for the purpose of this discussion, separate issue. Not to mention how much environment plays a role (a good home vs bad home situation for example).
I.e. exactly what those pesky naysayers decades ago said would happen eventually.
The assessments to guide student learning are not the standardize tests meant to measure school performance and do state-level planning
It says they were adjusting the scores, not in any one direction.
If kids were doing worse and you have flat scores, that'd indicate the test being adjusted to be easier.
If kids were doing better and you have flat scores, that'd indicate the test being adjusted to be harder.
Because they've designed this to produce flat scores, it obscures which of those two statements is true.
The former seems to say the students did improve but this was not visible.
>> In addition, norm-referenced tests [STAAR] are designed so that a certain share of students always fail, because success is gauged by one’s position on the “bell curve” in relation to other students. Following this logic, STAAR developers use practices like omitting easier questions and adjusting scores to cancel out gains due to better teaching.
Followed by:
>> Ultimately, the STAAR tests ... were not designed to show improvement. Since the test is designed to keep scores flat, it’s impossible to know for sure if a lack of expected learning gains following big increases in per-student spending was because the extra funds failed to improve teaching and learning, or simply because the test hid the improvements.
>> Texas’ educational accountability system has been in place since 1980, and it is well known in the state that the stakes and difficulty of Texas’ academic readiness tests increase with each new version, which typically come out every five to 10 years. What the Texas public may not know is that the tests have been adjusted each and every year – at the expense of really knowing who should “pass” or “fail.”
What is not clear or known for sure is the gains, gained, from an increase of money spent per-student because the test, by it's nature, is designed to be flat and not show improvements. But the test itself is designed by the developers to cancel out gains by making it harder. Therefore, the test could not be used to assess that. It is just written a bit funny.
This hurt the lower income students most as the wealthier schools with better teaching resources improved faster, it made the test more difficult for poorer performing schools. This is used as a measure to close or take over schools by the state. There's a bit of a kerfuffle going on currently about some schools getting F ratings and the state wanting to take them over, which in other states that might be okay, by here -- they are about to mandate that the 10 Commandments be displayed in classrooms[0]. They are also on track to getting rid of the STAAR test[1]. Hopefully the new assessment they replace it with will be better, supposedly it will be to show improvements, I fear it will then fall into the "no child left behind" trap that STAAR may have prevented by keeping scores flat, even with major investment in education; but it may also help lower income students who are hurt by the current testing practice in that they will receive the funds they earned for their improvement instead of losing them because wealthy districts outpaced them in gains -- even thou they both gained. Texas is weird sometimes.
[0] https://www.texastribune.org/2025/05/24/ten-commandments-tex...
[1] https://www.texastribune.org/2025/05/23/staar-test-texas-sch...
In reality, because the test was normalized, it is useless for determining the change in academic achievement over time, and it could just as easily end up being normalized in the direction of effectively making the test easier. Because the scores can't be used for this purpose, other evidence would be needed to determine how achievement is actually changing over time (and therefore whether the normalization is effectively increasing or decreasing scores) and the author doesn't present any evidence that would show which is the case.
However, either way the decision to normalize the scores on a test that's apparently supposed to be used to show changes over time seems odd if the author is correct that normalized scores are being used for that purpose.
Where she found:
>> According to policies buried in the documentation, the agency administering the tests adjusted their difficulty level every year.
She goes on to say that difficultly is increased. As shown in my original quotes of the article above.
I have not read her paper or the test design documents; but it appears she has and that she does have the evidence.
Because if the alternative is just vibes or through money at the problem and pray for the best, then I'm not on board.
I personally think school choice works best. If schools are bad, parents are the best to judge. And they should be allowed to send their kids elsewhere, new schools should open and bad ones should close. You know, how basically every other market system works
The problem is that those same students lose terribly over the summer break. Students from better socioeconomic strata gain over the summer break while those of lower socioeconomic strata do not. Thus, when the school year begins again, any gains made over the school year have been more than cancelled out over the summer.
This is one of the big arguments for year-round school.
In addition, higher socioeconomic status children keep making progress during the summer.
These are the kids you can least afford to fail. It's like having a fast progressing cancer. You have to try a lot of things because the alternative is so bad.
The solution to improving the outcomes for these kids doesn't start at school, it has to start at home.
This is not true.
The Gates Foundation, whom I generally dislike quite intensely for being busybody carpetbaggers who muck things up and disappear, documented this all very well. 2 teachers and 15 students per class works. Period. Every single program that did this showed significant improvements with the most improvement in the lowest performers.
The biggest problem, of course, is that when you ask for money for 2 teachers per 15 students everybody loses their minds (except the rich, of course, who understand the correlation perfectly). Every single one of the programs that the Gates Foundation documented as working were eventually shut down because of lack of funding.
People only say they want better education. Their actions say otherwise.
The US is one of the wealthiest countries in the world, we can figure this out.
While this is fine in elementary and early middle school (which is where the Gates foudnation documented), this probably breaks down as you enter high school where teachers need to be specialized and you need to have a pool of teachers to cover all subjects.
Does Texas fire underperforming teachers?
I’m genuinely asking. In California and New York underperforming schools are almost never shut down on account of powerful teachers’ unions. When they are, the teachers are reässigned. Teachers’ livelihoods are thus never at stake.
I am lucky that I picked up programming because I initially studied physics/engineering and I used to able to do math but now I can't anymore and my focus sucks. Writing code to me is way easier than doing math.
The author seems to think there is a magical alternative where students are graded to an absolute scale that doesn't change over time. That's not an option. Every grading system that ostensibly does this suffers from grade inflation. By the time you get to employers looking at grades, they just mentally convert the grades into a percentile. You end up back at grading on a curve/percentiles but with more steps.
At the scale of all students in Texas, the true rate of year-to-year change in ability is just going to be too small to measure (barring sudden events like the pandemic).
And speaking of the pandemic, where loosy goosy grading led to enormous grade inflation, what do you think employers/universities did? They obviously undid the inflation by regrading people back to the curve - "oh you did get 3 As but I can see that you graduated in 2020 so..."
I didn't see in the article the question of why Texas was going out of their way to yield flat test numbers like this, year after year.
It looked the author could've (or maybe not) been hinting that this is numbers-rigging -- to smoothly pass No Child Left Behind metrics, or to affect the real estate market in some way, or some other government goal.
Rather than numbers-rigging, I suppose it could also be useful for educators to see who needs more help, relative to their peers.
https://www.commitpartnership.org/insights/latest-learnings/...
They are correctly pointing out that the test is essentially norm-referenced (by design) and that this matters if you're looking for improvement (it will mask that improvement) but it also doesn't imply that they're making the test harder.
If anything - the flat line indicates that they're keeping it norm-referenced and that masks change in both directions.
It could well be that they're actually making the test easier since covid, again - to keep the line flat and norm-referenced (hard to tell since a lot of the article's data stops in 2021).
---
Essentially - this test is good for saying "How is this school performing vs its peers". It is not good for showing baseline improvement or decline across all schools.
That distinction is important, but I'm also not sure that it matters that much in terms of funding allocation; I'm not deep in the weeds here, so it's possible Texas is missing on some upside, but generally... my assumption is that the "budget" for education is relatively fixed at both the state and federal level, and the goal is not to award states that improve baseline numbers ever increasing budgets, but rather to force states to allocate the fixed resources to schools that are over-performing compared to their peers.
If that's the goal... a norm referenced test makes a ton of sense.
An alternative headline might read: "Texas’ annual reading test adjusted its difficulty every year, masking whether students are improving or getting worse"
I plan to investigate if other states or the federal government use similarly designed tests to evaluate students.”
This person doesn’t seem to have any understanding of the topic area she apparently lectures her students about. Scary.