It’s the process that leads to improvement

09/03/2009  |  DIANA NUNNALEY

A superintendent meets with principals to discuss the latest round of district test results.

The results reveal wide disparities in the scores for different subgroups of students. The administrators know that to improve student learning they must find a way to improve classroom instruction. They also know that the teachers are working harder than ever in responding to district pressure to raise test scores.

This scenario is typical even for districts that have embraced the concept of data-driven decision making and have invested heavily in desktop data systems and school improvement programs. A commitment to collecting and using data doesn’t necessarily lead to improvement — especially when there is not a process for building the district’s capacity to analyze the data. Too often schools introduce new programs based on one set of data or on a set of assumptions drawn from the data.

As a data coach and facilitator helping school districts around the country implement the Using Data Process developed at TERC, I have seen how a collaborative inquiry process can turn a district’s commitment to using data into real change and improved student learning. The Using Data Process has several key elements to build capacity for using data effectively.

Steps in the Process:
Forming a Data Team

Establishing an involved team of educators dedicated to improving student outcomes is essential to successful implementation of Using Data practices. The Using Data program’s focus on the Data Team is two-fold: it eliminates the kind of rapid-fire decision making that limits educators’ abilities to inquire and understand the real issues at hand and establishes ownership of the process so that members are fully motivated to implement and monitor their proposed solutions. Teachers are at the center of the process and are encouraged to explore problems and generate solutions together without transferring blame or censure to either students or educators. To help build the team’s foundation for successful collaboration, members use demographic data to learn more about the students, faculty, and the community. The assumption here is that members already understand the makeup of the school community — however, the process can often reveal misconceptions.

In fact — in one district where teachers were struggling with meeting student learning goals based on an assumption about the high degree of mobility in the district — the data team was surprised to learn that in their district of 15,000 students, there were only about 115 highly mobile students K-12. The district’s demographic data and data about the community yielded valuable insights and served to put aside long held beliefs about some of the causes of low performance.

Steps in the Process:
Considering Culture

In today’s increasingly diverse school environments, a commitment to cultural understanding is absolutely necessary for enacting solutions for lasting changes. Closely examining demographic data frequently exposes negative assumptions about what children of color, students with exceptional needs, and students living in poverty are capable of learning and achieving as well as the realities of achievement gaps. While exposing achievement gaps is a difficult and upsetting process for many educators participating in close data inquiry, it can be a bridge to understanding and ameliorating the attitudes and practices that stand as barriers to effective teaching.

For example, while investigating low proficiency levels in literacy, members of one data team conjectured that many of their high poverty students were performing at lower levels. It was not until they delved into the data that they realized the extent of this connection. Disaggregating data revealed achievement gaps to be almost entirely correlated with low socioeconomic status. Using item and student work to identify specific learning challenges, teams zeroed in on student reading comprehension. With research on the table regarding the “pedagogy of poverty” (see the work of  Martin Haberman, Yolanda Padrón, Hersholt Waxman, Lorna Rivera and others) and its implications for classroom practice, teachers began to ask the hard questions about how their expectations for student learning were influencing the degree to which students were being asked to explain their thinking regarding connections in a reading passage. Through this discussion, teachers realized that by simplifying the curriculum, they were negatively impacting student learning. They re-evaluated their expectations of what students could learn, changed their teaching practices accordingly, and established a long-term commitment to cultural proficiency training and discussion.

Steps in the Process:
Identifying a Problem and
Disaggregating Data

Ultimately, getting to the core of student learning problems requires an analysis of disaggregated data to discover the groups of students having problems and the nature of the learning/teaching issues. This often means looking at groups within groups, and the hidden groups – those too small or specific to be counted for AYP and other state reporting. The richest details surface during this phase of analysis as teachers begin to question district or building policies, structures, and practices that are limiting opportunities to learn for some groups of students. During this process teams use multiple levels of their criterion-referenced test data, drilling down to strand and item level analysis.  As they begin to identify a student-learning problem and goal they also bring local data including common assessments and student work into the discussion.

Teachers in one elementary school were able to uncover why their students were having recurring issues with using rulers effectively by using Using Data disaggregation techniques in accompaniment with customized benchmark assessments. While Data Team members at this school knew that their 5th and 6th graders were having problems with measurement, looking at state data wasn’t enough to help them isolate the issues at hand. By studying results from benchmark assessments and questioning the specific skills required to understand measurement, they discovered that students were not having problems reading rulers but were instead having difficulties converting one unit to another within the same system.

Steps in the Process:
Verifying Causes and
Generating Solutions

Once the team has identified a student learning problem, they then begin looking for causes. Throughout this process, members continue to examine data to try and verify the causes they have identified. They seek out local data about practice including observations, surveys and interviews and they study relevant research. This stage is critical because it prevents members from taking action based on inaccurate assumptions about root causes of a learning problem.  As they generate solutions they employ a logic model that predicts outcomes based on proposed strategies and devise a plan for monitoring implementation.

Steps in the Process:
Implementation and
Renewing the Inquiry Cycle

The investment that team members make in identifying learning problems and developing solutions establishes a real commitment to implementing the proposed strategies for improvement. The experience they gain in analyzing data propels them to seek out additional data as they monitor implementation, thus continuing the inquiry cycle. This ownership of the process can mark a real shift in school culture and put the school on the path to building and sustaining their capacity to use data effectively.

Diana Nunnaley is Project Director, Using Data at TERC. To learn more about the Using Data Process, workshops, resources, and results, visit
Comments & Ratings