Study examines how 'data in the wild' is used in diverse school district
LAWRENCE — For several decades, federal policymakers have pushed for education in America to be a more scientific endeavor. Researchers have sought to improve all aspects of education, but little attention has been paid to the experiences of educators putting research into practice. By studying one school district’s adoption of a comprehensive reform initiative, a recent study from the University of Kansas found that teachers became “data-deferent” rather than “data-driven” as ideally intended. The authors argue districts and expert consultants can do more to facilitate practitioner access and understanding of educational research. They should also expand what counts as “data” and consider how reform models can both maintain fidelity while remaining flexible enough to fit local circumstances.
KU researchers spent five months at Garden City Public Schools as part of an ethnographic study into how the district and its 18 schools addressed the prevalent challenges of a culturally diverse, multilingual student population. During the study, they saw teachers, administrators and others working to implement Multi-Tiered System of Supports, known as MTSS, a research-backed reform method designed to identify and address student needs before they fall behind. That presented a unique opportunity to study how research is put into practice and how data is used in schools to inform educational change.
“With any sort of reform model, you try to make something abstract happen in real life. However, in real life, lots of things can happen,” said Jennifer Ng, associate professor of educational leadership & policy studies at KU and a co-author. “Our analysis illustrates how reform occurred. For all of the talk about how education should be data-driven, there has not been much close study of what another scholar calls ‘data use in the wild.’”
Ng co-authored the study with Don Stull, professor emeritus of anthropology at KU, and Rebecca Martinez of Indiana University. It is forthcoming in the journal Teachers College Record.
During the five-month study, researchers interviewed teachers, observed classrooms, attended school board meetings and interacted with school personnel across the district as Garden City Public Schools worked to implement MTSS. They found teachers were forced to rely on data generated by a prescribed set of standardized assessments to determine if students needed additional supports or were performing at an appropriate grade level. There was no room for teachers to offer additional observations of students or draw upon their own judgment to inform appropriate courses of action.
“We paid attention to how certain concepts became meaningful,” Ng said. “Data, for example, came to be viewed as a number derived from a single standardized test. Everything that was not a number was very carefully distinguished as an anecdote or mere feeling.”
Sole reliance on numeric data had the effect of suggesting other factors, such as teachers’ direct observations or parental input were not valid ways of knowing about students and their needs. Outside consultants who worked with district personnel to implement the reform assured teachers the reform model was research-based and thus infallible. There was also a focus on strict adherence to instructional treatments.
The dominant logic of scientifically based research is that its outcomes will lead to certain and generalizable recommendations. However, that does not consider the particular contexts of individual schools. In Garden City, the community and district context includes large numbers of students who are English language learners, may have suffered significant trauma as refugees or migrants, and had varied prior educational experiences.
“Part of what we observed in the resistance, or discontent, of practitioners was not only their professional expertise being diminished, there were contextual and cultural variables that they felt unable to factor in,” Ng said.
The researchers argue that educational practitioners should be given access to data and helped to understand why and how it can actually inform improvement. They should also be granted the flexibility to add their own expertise and professional experience to research-based reform. Educational interventions can have great success in improving educational practices, but a one-size-fits-all approach to implementing reform is not only unlikely to be effective, but it can also be counterproductive.
“With any school reform, you don’t just adopt an idea and have everything automatically fall into place,” Ng said. “You’re depending on a lot of people across different roles to do their part. There can be misunderstandings or even active resistance that will play a role in how it’s carried out.”
The study provided a unique opportunity to see how research and data are handled in the field and adds a greater understanding of how reform can be modeled for the greatest success in widely varying schools across the nation.
“This was a unique opportunity,” Ng said. “Research on data use has tended to focus on very specific uses or simply reinforce the mantra that ‘data is good.’ We’re grateful to the Garden City school district for giving us the chance to directly examine the implementation of research to practice.”