Data Is Not Neutral: Rethinking AI and Equity in Education

Data is often treated as if it simply reflects reality, as if it objectively captures what is true about students, learning, and schools.

We disagree.

In our conversation on EDU Café, we explored a different premise: data is not neutral. Nor is data self-evident. It is produced, interpreted, and used within social, political, and technological systems that shape what it becomes, how it is used, and the impacts it has.

This distinction matters more than ever as data and AI systems become increasingly embedded in education and life.

Data Is Socially Produced, Not Self-Evident

A common assumption in research and practice is that “data speaks for itself.” This idea has deep roots in traditions of scientific measurement and empirical inquiry. It is one of the main tenets of logical empiricism/logical positivism that informs much of quantitative social science.

But data does not emerge in a vacuum.

It is generated through processes that involve:

• Decisions about what to measure

• Definitions of what is important and what can be ignored

• Tools and technologies that structure collection and analysis

• Contexts in which measurement, interpretation, and reaction occurs

These are not neutral processes. Cultural norms, institutional priorities, and political interests shape them.

From our perspective, data is better understood not as a static object, but as an event within an ongoing process, one that is always provisional and contingent.

Measurement Is Never Innocent

One way to understand this is through a simple idea: to measure is to change.

Even in a controlled scientific setting, the act of measurement influences the subject and the researcher. In education, the effects are even more profound. Assessments, analytics tools, and algorithms do not merely observe learning; rather, they influence it.

This raises enduring challenges around:

• Accuracy vs. consistency

• Design vs. interpretation

• Validity vs. consequence

Too often, discussions of validity focus only on whether a measure captures what it intends to capture. But interpretation, use, and their effects are just as, if not more, important. When data are misunderstood or treated as unquestionable, it can lead to decisions that shape student trajectories in harmful ways.

A Shift: From Capturing Learning to Shaping It

We are currently witnessing a significant shift in how data operates in education.

Traditional measurement systems were largely designed to capture learning at specific points in time. Today’s AI-driven systems increasingly aim to generate and shape learning in real time.

These systems:

• Predict behaviors and what should be done next

• Provide continuous feedback, regardless of whether it was asked for

• Nudge learners and educators toward particular norms and outcomes

This is not simply measurement; it is the illusion of an “invisible hand” of intervention, what might be understood as the infrastructure of Friedrich Hayek’s dreams, whereby corporate and state interests are often what inform the protocols of the AI-driven system.

And because these systems operate continuously, they may have a more pervasive and powerful influence than traditional assessments ever did.

Technology: Doing Things Better, or Doing Better Things?

No educational technology works equally well for all users, unless it is a really bad technology that works for no one. Even the best technologies present both opportunity and risk.

On one hand, tools can increase access, efficiency, and scalability. On the other hand, they can reinforce narrow ways of thinking if they embed a single “correct” method or overrepresented perspective.

This tension can be summarized as the difference between:

• Doing things better (optimization, speed, efficiency)

• Doing better things (rethinking purpose, expanding possibilities, focusing on collective wellbeing)

If we are not careful, we risk using powerful technologies to reinforce and amplify existing assumptions and structural violence rather than to challenge them.

The Problem of Unchecked Innovation

One of the most pressing concerns we see is the lack of meaningful oversight or guidance in the educational technology space.

Unlike standardized testing, which is governed by established professional standards, many edtech tools are developed and deployed without comparable frameworks for:

• Ethical review

• Equity analysis

• Consequential impact

In EdTech, it is a caveat emptor (i.e., let the buyer beware). In assessment, however, we have a caveat venditor (i.e., let the seller beware).

In some cases, technologies are tested in real classrooms with real students before these questions are fully addressed.

This places a significant burden on educators, who are often asked to adopt tools without clear guidance on their implications.

Fairness Begins with Asking Different Questions

If we want more just and equitable uses of data, we need to begin earlier in the process, rather than after systems are built.

This means asking why we are measuring something in the first place, who benefits from the system, what assumptions are embedded in its design, what consequences its use might produce, and whether the technology is actually necessary.

These are not just technical questions. They are ethical, political, and social questions.

Keeping the Relational and Situated Process of Education at the Center

As AI becomes more capable, it is tempting to prioritize efficiency and automation. But education is fundamentally a social, relational, and situated process.

Data and technology should support, not replace, the essential actors in the relational and situated processes of education such as:

• Interpretation and judgment

• Dialogue and relational learning

• Creativity and critical thinking

• Learning from and working with situated knowledges

In practice, this also means being transparent with learners about the role of technology, being intentional about when independent thinking is essential, and accounting for and cultivating the variability of learners and what they bring to the pedagogical context.

Learning to See the Blind Spots

A useful metaphor here is the idea of a blind spot.

In both human perception and data systems, some gaps remain invisible unless we have external instruments or frameworks to see them. Detecting these blind spots requires deliberate effort, such as slowing down, questioning assumptions, and examining the systems we rely on.

Without this work, we risk accepting outputs without understanding the processes that produced them.

Data is not neutral, and neither are the systems that produce and use it.

As educators, researchers, and designers, we are called to engage more critically with the tools and metrics that shape learning and teaching. This means moving beyond questions of efficiency toward deeper considerations of purpose, ethics, and impact.

The future of education will not be determined solely by what our technologies can do, but by the choices we make about how, and whether, we use them and what we allow them to do to us.

And those choices must remain grounded in a commitment to human dignity, equity, and possibility.

Listen

NGLC is grateful for our collaboration and partnership with EDU Café Podcast that brings fresh voices and insights to the blog. Listen to the full episode of the podcast that inspired this article.

Photo at top courtesy of Two Rivers Public Charter School