Study finds race, class size, state politics can limit range of student skills
LAWRENCE — A University of Kansas researcher has found that in the last decade, fourth-graders who live in states with a significant African-American population, larger classroom sizes and a more conservative political climate tend to learn fewer mathematics skills in school.
"In this regard, what types of skills students learn is not necessarily objectively given. It is determined by specific state characteristics," said Argun Saatcioglu, associate professor of education and courtesy professor of sociology.
Saatcioglu reached the finding when examining the correlation of student performance on state assessments with performance on the National Assessment of Educational Progress exam, or NAEP, which is widely viewed as the “gold standard” in testing and is used to calculate the Nation's Report Card.
He will present his findings as part of the 109th annual American Sociological Association Meeting, Aug. 16-19 in San Francisco. His presentation will focus on "Are State Assessments Comparable to the NAEP? Exploring State and Political Factors Related to State NAEP/Comparability."
“States typically do much better on their own assessments than on the NAEP,” Saatcioglu said, “because the NAEP is a harder test with a different purpose, and states are not, and should not, be expected to align their practices and policies to NAEP standards.”
The study does not focus on performance gaps between NAEP and the state tests. Instead, it addresses the correlation between the outcomes of the two tests.
For example, performance on a state assessment could be higher than a NAEP score in magnitude, but there still could be a correlation between the two, Saatcioglu said. A decline in the correlation indicates that the two tests are not measuring the same skill set. NAEP evaluates a uniquely wide range of skills because NAEP's national board incorporates skills and curriculum from all states.
Thus, a weakening correlation between the outcomes of a state’s assessment and that state’s NAEP outcomes mean that students in that state are learning an increasingly narrower range of skills.
“Progress in rigor and equity in what students are taught in schools is a fundamental objective of the No Child Left Behind accountability law,” said Saatcioglu, adding the finding was concerning because it suggests outcomes that contradict the aim of the accountability law's namesake, "no child left behind."
"Who are we trying to not leave behind again? It really in my view is a policy that if you do it this way, yes it may help the nation on average, but it may be hurting the very particular group that is expected to benefit from this," Saatcioglu said.
He found particularly in states with a high African-American population and larger class sizes, which is common in more urban school districts, that students may be learning fewer skills because that would help improve state assessment results. Narrowing the skill set, in other words, results in higher proficiency gains on state assessments because students have to be proficient in fewer skills.
Using NAEP as a measure to compare performance among students in different states also can leave districts with a tough choice, he said, because he found teaching more skills to students could cause their state assessment scores to suffer, putting the district at risk of negative consequences due to the NCLB, such as school closures, transfers or lost federal funding. Saatcioglu said his findings indicated states with a higher African-American population or a larger class sizes tended to teach more skills in the early years of his study but then pull back on the number of skills in 2009 possibly because of a drop in state assessment scores.
"If a state moves closer to the NAEP, that means it's teaching more and more skills. That's can be a disincentive in an accountability system because you have to work harder with the students," Saatcioglu said. "So if you move away from NAEP, you are going to improve your proficiency scores. If you move away too much then you are essentially gaming the system in order to register performance gains in state assessments."
He said the study has implications for the sociology of education and education policy. It also has implications for organizational sociology as it addresses how educational institutions respond to measures of accountability. He said future analysis of the data would likely also focus on how specific states performed and the implications of that.
For example, Florida in 2003 was above the national mean in terms of having a wide skill range it taught students, but it narrowed by 2009. Texas took the opposite course in that its skill range widened after six years. He said looking more at both of those two states would be key because quite a bit of influential testing policy comes from Florida and Texas. Tennessee, which is known for its value-added assessment system, was below the mean in the range of its skill set taught before 2003, and it grew worse by 2009.
He said states with conservative political values typically tended to resist federal regulation at the beginning in 2003, so their testing was less likely to align with NAEP, although he saw evidence those states slowly approached alignment in the later years.
Saatcioglu said the study is not aiming criticism at certain states or districts that tended to teach fewer skills but instead on the effect that school accountability laws have had potentially in widening economic inequality, particularly among districts in communities with a higher population of African-American students or urban districts with larger class sizes.
"You are giving schools and students a credential, but it's just a credential," Saatcioglu said. "It doesn't represent skill sets or other types of things that you can cash in on in the labor market. And who is it doing this to? Large African-American populations and districts with large class sizes. That's what is happening here."