Fairness in Machine Learning Ignores Structural Injustice

Fairness in Machine Learning Ignores Structural Injustice

Here is a paradox: For the last decade, hundreds of researchers have been building algorithms to make machine learning fairer. They have produced thousands of papers. They have invented mathematical definitions of fairness so precise that you can prove theorems about them. And yet, according to a comprehensive 2022 survey by Lindsay Weinberg, published in the Journal of Artificial Intelligence Research, the entire enterprise may be making things worse.

Weinberg, a scholar at the intersection of technology and justice, reviewed critiques from philosophy, feminist studies, critical race and ethnic studies, legal studies, anthropology, and science and technology studies. She found that the dominant approach to fairness in machine learning suffers from a fundamental blindness: it treats structural injustice as a bug to be fixed rather than the system in which the software runs.

The result is a field that defines fairness so narrowly that it can claim victory while the people it claims to help remain trapped in the same old patterns of exclusion and harm.

What Does "Fair" Even Mean Here?

The first problem Weinberg identifies is that fairness researchers cannot agree on what fairness means. This is not a minor philosophical squabble. It is a crisis of foundations.

Weinberg (2022) notes that within computer science, fairness is typically defined in formal, mathematical terms. There are dozens of competing definitions: demographic parity, equal opportunity, equalized odds, predictive parity. Each one captures a different intuition about what fairness requires. The trouble is that these definitions are often mathematically incompatible. You cannot satisfy all of them at once. So researchers pick one, prove their algorithm satisfies it, and call the problem solved.

But here is what gets lost: none of these definitions ask whether the system should exist in the first place.

Consider a predictive policing algorithm that is "fair" according to demographic parity, meaning it predicts crime at the same rate across racial groups. The algorithm might be mathematically fair while still funneling police into neighborhoods that have been overpoliced for generations. The fairness definition treats the underlying structure of policing as a given. It never asks whether policing is the right tool for the problem at hand.

Weinberg (2022) calls this a "techno-solutionist" trap. The field assumes that the answer to an unfair algorithm is a better algorithm. It does not consider the possibility that the answer might be no algorithm at all.

The Abstraction Problem

Machine learning works by abstracting away the messiness of the real world. A model takes a complex social situation, reduces it to a set of features and labels, and then learns patterns. This is what makes ML powerful. It is also what makes fairness interventions so fragile.

Weinberg (2022) argues that abstraction is not a neutral technical step. It is a political act. When you decide which features to include, which to exclude, and how to label the outcome, you are encoding a particular worldview. You are deciding what matters and what does not.

Here is a concrete example from the survey. Consider a hiring algorithm trained on historical hiring data. The data shows that men were hired more often than women for certain roles. A fairness intervention might try to correct for this by ensuring that the algorithm recommends male and female candidates at equal rates. But the abstraction has already done its damage. The data does not capture why women were not hired in the past. It does not capture the informal networks, the biased interviewers, the hostile work environments. It only captures the outcome. The fairness fix treats the symptom while leaving the cause intact.

Weinberg (2022) draws on feminist and critical race scholarship to argue that abstraction "renders invisible the structural conditions that produce inequality." The algorithm sees a world of individuals making choices. It does not see the systems that constrain those choices.

Racial Classification as a Feature, Not a Bug

One of the most uncomfortable findings in Weinberg's survey is that fairness research often relies on racial classification in ways that reinforce racial hierarchy.

Many fairness algorithms require data on race. They need to know who is Black, who is white, who is Asian, in order to check whether the algorithm treats these groups equally. But race is not a clean biological category. It is a social construct with a violent history. Collecting racial data and using it to calibrate algorithms can entrench the very categories that antiracist movements seek to dismantle.

Weinberg (2022) cites critical race theorists who argue that "racial classification in ML fairness research reifies race as a fixed, natural category." The algorithm treats race as a variable to be balanced, like income or education level. It does not ask why race correlates with outcomes in the first place. It does not ask whether the categories themselves are products of historical injustice.

There is a deeper problem here. Fairness algorithms that adjust for race can end up legitimizing racial disparities. If an algorithm is certified as "fair" because it treats racial groups equally, then any remaining disparity must be due to something else. The algorithm becomes a shield against claims of racism. This is what Weinberg (2022) calls "ethics washing" the use of fairness measures to avoid regulation and deflect criticism.

The Data Collection Trap

Even when fairness researchers get the definitions right and the abstractions right, they still face the problem of data. And the data is almost always bad.

Weinberg (2022) reviews critiques from anthropology and science and technology studies showing that data collection practices are deeply flawed. Data is often collected without meaningful consent. It is scraped from public sources without people knowing their information is being used. It is labeled by low wage workers in the global south who have no stake in the outcomes. It reflects historical biases that are baked into the recording process.

Consider a facial recognition dataset used to train "fair" algorithms. The dataset might be balanced by race, with equal numbers of Black and white faces. But where did those faces come from? Were they taken from mugshots? From social media without permission? From people who had no idea their images would be used to train commercial software?

Weinberg (2022) argues that "data collection practices that entrench bias, are nonconsensual, and lack transparency" are not bugs that fairness can fix. They are features of the current system. The data is extracted from marginalized communities, used to build systems that are sold back to those same communities, often with harmful results.

Predatory Inclusion

One of the most striking concepts in Weinberg's survey is "predatory inclusion." This is the practice of bringing marginalized groups into systems that ultimately harm them, while claiming to help.

Weinberg (2022) draws on legal scholar Khiara Bridges to describe how this works. A company might develop a credit scoring algorithm that includes more low income people. It calls this "financial inclusion." But the algorithm charges higher interest rates to those people because they are riskier by conventional metrics. The inclusion is predatory. It extracts value from the marginalized while offering them worse terms than the privileged.

Fairness interventions often miss this dynamic because they focus on access rather than outcomes. An algorithm that includes more people is considered fair, even if those people are included on exploitative terms. The fairness definition does not capture the quality of inclusion. It only captures the fact of inclusion.

Weinberg (2022) argues that this is a pattern across many domains. Predictive policing algorithms include more neighborhoods in surveillance. Hiring algorithms include more applicants in screening. Healthcare algorithms include more patients in risk scoring. In each case, inclusion sounds good. In each case, the included often end up worse off.

What the Research Does Not Prove

Before going further, it is important to be clear about what Weinberg's survey does and does not show.

The survey does not prove that all fairness research is useless. It does not show that mathematical definitions of fairness are always wrong. It does not claim that every fairness intervention makes things worse.

What the survey does show is that the dominant approach to fairness in machine learning has systematic blind spots. These blind spots are not accidental. They are built into the way the field defines its problems, collects its data, and evaluates its solutions.

Weinberg (2022) is careful to note that the critiques she surveys come from outside computer science. They represent perspectives that the field has largely ignored. The question is not whether fairness research can be saved. It is whether the field is willing to listen to voices that have been telling it, for years, that it is asking the wrong questions.

The Missing Voices: Participatory Design

One of the most damning critiques in Weinberg's survey is that fairness research rarely involves the people it claims to help.

Weinberg (2022) notes an "absence of participatory design and democratic deliberation in AI fairness considerations." The researchers who define fairness, choose the data, and evaluate the outcomes are almost never members of the communities affected by the algorithms. They are computer scientists at elite universities or engineers at large tech companies. They are overwhelmingly white, male, and wealthy.

This matters because fairness is not a technical question. It is a political one. Different communities have different values, different priorities, different histories. A fairness definition that makes sense to a Stanford professor might be meaningless to a single mother in Detroit. But the professor does not ask. The algorithm is built and deployed. The mother lives with the consequences.

Weinberg (2022) draws on science and technology studies to argue that "the design of AI systems should be a democratic process, not a technical one." This means involving affected communities in every stage: problem definition, data collection, model design, deployment, and evaluation. It means giving those communities real power, not just a seat at the table.

Long Term Consequences Nobody Is Tracking

The final critique in Weinberg's survey is perhaps the most unsettling: nobody knows what these systems do over time.

Weinberg (2022) finds "a lack of engagement with AI's long term social and ethical outcomes." Fairness research typically evaluates algorithms on static datasets. It checks whether the algorithm meets its definition of fairness at a single point in time. But algorithms do not operate in a static world. They change behavior. They change incentives. They change the data that future algorithms will train on.

Consider a "fair" hiring algorithm that selects candidates at equal rates across racial groups. Over time, this algorithm shapes the pool of applicants. People who are rejected learn not to apply. People who are accepted tell their friends. The algorithm changes who sees the job posting, who applies, who gets hired. Five years later, the world looks different. But the fairness evaluation only checked the first moment.

Weinberg (2022) argues that this temporal blindness is a feature of the computational approach. Computer scientists are trained to evaluate systems on fixed metrics. They are not trained to think about feedback loops, emergent effects, or long term social change. These require different methods, different disciplines, different timelines.

What This Actually Means

The implications of Weinberg's survey are not abstract. They point to concrete changes in how fairness research should be done, funded, and evaluated.

• ▸Stop treating fairness as a math problem. Mathematical definitions are useful tools, but they are not substitutes for political deliberation. Fairness requires asking who benefits, who loses, and who decides. These are not questions that can be optimized away.

• ▸Include the excluded. Participatory design is not a checkbox. It means giving real decision making power to communities affected by AI systems. If the people who will live with an algorithm have not shaped its design, the algorithm is not fair, no matter what the math says.

• ▸Ask whether the system should exist. The most important fairness question is not "How do we make this algorithm fair?" It is "Should this algorithm exist at all?" Some problems should not be automated. Some systems should be dismantled, not reformed.

• ▸Track long term outcomes. Fairness is not a snapshot. It is a process. Researchers and deployers need to study what happens after an algorithm is deployed. They need to measure feedback loops, behavioral changes, and unintended consequences. This requires long term funding and interdisciplinary teams.

• ▸Reject ethics washing. Companies and governments that use fairness measures to avoid regulation are not acting in good faith. The field needs standards for what counts as genuine accountability, not just technical compliance.

Weinberg's survey does not offer easy answers. It offers a diagnosis. The dominant approach to fairness in machine learning is failing because it refuses to confront the structural injustice that gives rise to inequality in the first place. The math is not the problem. The politics is.

The question is whether the field is ready to face that.