Why Fairness in Machine Learning Is a Sociotechnical Mirage

Why Fairness in Machine Learning Is a Sociotechnical Mirage

You have probably seen the version of the future where algorithms fix bias. A hiring tool is trained on historical data, detects patterns of discrimination, and then a fairness-aware learning algorithm adjusts the weights so that women and minorities get equal consideration. The system is audited. The numbers look good. The company deploys it. And then something strange happens: the outcomes are still unfair.

Not because the math was wrong. But because the math was never the whole story.

In a landmark 2019 paper, four researchers from the fields of computer science, law, and information science laid out exactly why this keeps happening. Andrew Selbst, danah boyd, Sorelle Friedler, and Suresh Venkatasubramanian argued that the entire fair machine learning enterprise is built on a fundamental mismatch between how computer scientists think about fairness and how fairness actually works in the world (Selbst et al., 2019). The paper, published at the ACM Conference on Fairness, Accountability, and Transparency, has been cited over 1,200 times. It should be required reading for anyone building or buying an algorithmic system that affects people's lives.

The core problem is abstraction. Computer science depends on it. You model a problem, you isolate variables, you define fairness mathematically, and you optimize. But fairness in society is not a variable you can isolate. It is a property of a system, not a property of a model.

The Five Traps That Make Fair ML a Mirage

Selbst and colleagues identified five specific traps that fair ML work falls into. Each one is a way that the abstraction of a model fails to capture the messiness of the real world.

Trap 1: The Framing Trap. This is the most fundamental. When a team decides what problem to solve with machine learning, they are already making a choice about what counts as a problem and what counts as a solution. The authors found that technical fixes for fairness often accept the existing frame of a decision making system without questioning whether the system itself is just. For example, if you build a fairer predictive policing algorithm, you are still policing the same neighborhoods. You have not asked whether policing is the right response to poverty or addiction. The frame is fixed. The algorithm just optimizes within it.

Trap 2: The Portability Trap. A fairness definition that works in one context may not work in another. The authors found that the most common fairness metrics, like demographic parity or equalized odds, were developed for specific settings like hiring or credit. But these definitions do not travel well. A model that is fair according to one metric in a lab can be deeply unfair when deployed in a different social context. The abstraction of the metric hides the contextual dependencies.

Trap 3: The Formalism Trap. This is the trap of mistaking a mathematical definition of fairness for fairness itself. Selbst et al. found that many fair ML papers treat fairness as a property that can be formally specified and then optimized. But fairness is a contested, socially negotiated concept. You cannot define it away with an equation. The authors argued that formal definitions of fairness are useful tools, but they are not the thing itself. Mistaking the map for the territory is a classic error, and fair ML is full of it.

Trap 4: The Ripple Effect Trap. This is the one that keeps me up at night. A model does not just make a prediction. It changes the world around it. The authors found that when a fairness-aware algorithm is deployed, it alters the incentives and behaviors of the people it is meant to serve. A credit scoring model that is made fairer by adjusting for race may cause lenders to change their marketing strategies, which then changes who applies for loans, which then changes the data the model was trained on. The model is not a static intervention. It is a dynamic actor in a complex system. The abstraction of a static pipeline misses all of this.

Trap 5: The Allocative vs. Representational Harm Trap. This is the trap of focusing on who gets what (allocative harm) while ignoring how people are categorized and represented (representational harm). The authors found that many fair ML systems address whether a person gets a loan or a job, but they do not address whether the system reinforces harmful stereotypes or categories. For example, a hiring algorithm that is fair in its allocation of interviews might still use categories like "culture fit" that exclude people on the basis of identity. The abstraction of the model does not see these representational harms.

Why the Traps Are Inevitable, Not Accidental

You might think these traps are just bugs that smarter engineers can fix. But Selbst and colleagues argued that they are features of the way computer science works. The entire discipline is built on abstraction and modular design. You break a problem into parts, you model each part separately, and you combine them. This works beautifully for building a search engine or a compiler. It works terribly for building a fair society.

The authors drew on Science and Technology Studies (STS) to explain why. In STS, a sociotechnical system is not just the technology plus the people. It is the entanglement of both. You cannot separate the algorithm from the institution that deploys it, the laws that govern it, the people who use it, and the history that shaped it. Fairness is a property of that whole entanglement, not of the algorithm alone.

This is not a small point. It means that every technical intervention that tries to make a model fair is, by design, ignoring most of the things that make fairness real. The model is an abstraction. The world is not.

What the Research Does NOT Prove

The Selbst et al. paper is not an argument for giving up on fairness in machine learning. It is not a claim that all fair ML is worthless. The authors were careful to say that technical interventions can be useful, but only if they are understood as part of a larger process.

What the paper does not prove is that there is a single correct way to avoid these traps. The authors offered suggestions, but they did not claim to have solved the problem. They argued for a shift in design from solutions to processes, and for drawing abstraction boundaries that include social actors rather than just technical ones. But they were honest about the difficulty.

An open question remains: can we build technical systems that are genuinely responsive to the sociotechnical context, or is the abstraction gap too wide? The paper suggests that the gap can be narrowed but probably never closed. That is a sobering conclusion for anyone who wants a neat technical fix.

What This Actually Means

• ▸Stop treating fairness as a property of a model. It is a property of a system that includes the model, the institution, the law, the people, and the history. If your fairness audit only looks at the model, you are missing most of the picture.

• ▸Question the frame before you optimize within it. Ask whether the problem you are solving is the right problem. A fairer predictive policing algorithm is still policing. A fairer credit score still assumes credit is the right measure of worth.

• ▸Design for process, not for solution. Instead of building a single fair model, build a system that can be iteratively adjusted as the context changes. This means including social actors like community members and regulators in the design process, not just engineers.

• ▸Watch for ripple effects. A model changes the world it is trying to predict. Plan for that. Monitor for it. Build feedback loops that can detect when the model is altering behavior in unexpected ways.

• ▸Do not confuse formal definitions of fairness with fairness itself. Demographic parity, equalized odds, and other metrics are useful tools. They are not the goal. The goal is a just society. That is not something you can optimize for with a loss function.