Hiring Algorithms Learn Racism From Resumes Not Labels

The Resume That Never Got a Fair Shake

A few years ago, a Fortune 500 company decided to overhaul its hiring process. They were drowning in résumés, thousands for every open position, and they wanted to find the best candidates faster. So they did what many firms do: they built a machine learning algorithm to screen applicants. The algorithm was trained on the company’s own hiring data, years of decisions made by human recruiters. The logic seemed airtight: teach the machine what a good candidate looks like, based on who actually got hired and performed well, and let it flag the top résumés.

It didn’t take long for the researchers to notice something unsettling. The algorithm was systematically downgrading candidates from certain demographic groups. Black applicants. Women. People who attended less prestigious universities. The usual suspects.

But here’s the part that should make you pause. The company hadn’t explicitly told the algorithm to discriminate. They hadn’t fed it race or gender labels. They had simply given it résumés and told it to find the best people. And the algorithm, left to its own devices, learned racism anyway.

This isn’t a story about biased programmers or malicious code. It’s a story about what happens when an algorithm is trained on a world that is already unequal. And it’s a story about a hidden design flaw that most hiring algorithms share: they are built to exploit the past, not explore the future.

The Bandit Problem You Didn’t Know You Were Solving

Picture a slot machine in a casino. You pull the lever. Sometimes you win, sometimes you lose. Now imagine a row of slot machines, each with a different payout rate. You have a limited number of pulls. Do you stick with the machine that has paid out well so far, or do you try a new one that might be even better?

This is the classic “exploration versus exploitation” dilemma. In computer science, it’s called a multi-armed bandit problem. The optimal strategy is not to always pick the machine with the best historical record. You need to spend some of your pulls exploring unknown machines, because the one that looks best today might not be the best tomorrow.

In a 2020 paper titled “Hiring as Exploration,” economists Danielle Li, Lindsey Raymond, and Peter Bergman argued that hiring is fundamentally a bandit problem. A firm wants to find the best workers over time. To do that, it needs to balance exploitation, picking candidates from groups with proven track records, with exploration, picking candidates from underrepresented groups to learn about their true potential.

But here’s the catch. Most hiring algorithms aren’t designed for exploration. They are designed for pure exploitation. They take historical data, learn what a “good” candidate looks like based on past hires, and then rank new candidates accordingly. This approach has a name: supervised learning.

The Supervised Learning Trap

Supervised learning is the workhorse of modern AI. You give the algorithm a bunch of labeled examples, say, photos of cats and dogs, and it learns to distinguish between them. The key assumption is that the labels are correct and representative of the world you want to predict.

In hiring, the labels are “hire” or “reject.” But those labels are not neutral. They are the product of decades of human decisions, biases, and structural inequalities. When a company trains a hiring algorithm on its own past decisions, it is encoding those biases into the machine.

The researchers in the Fortune 500 study saw this play out in real time. The algorithm learned that candidates from certain universities, certain zip codes, certain demographic backgrounds were “better” because they had been hired more often in the past. But the past was not a fair test. It was a test where some groups never got a chance to pull the lever.

This is the core insight from the paper “User Strategization and Trustworthy Algorithms,” published by researchers at MIT and Google in 2023. The authors, Sarah H. Cen, Andrew Ilyas, and A. Ma̧dry, point out that most algorithms rely on an assumption called exogeneity. The idea is that the data you are training on is generated by an independent process, like a random survey or a controlled experiment. But in practice, the data is shaped by the algorithm itself.

Think about it. If a hiring algorithm learns that candidates from Ivy League schools are better, it will recommend more Ivy League candidates. Those candidates get hired. Their performance data reinforces the algorithm’s bias. Meanwhile, a candidate from a state school never gets the chance to prove themselves. The algorithm never learns that they could have been just as good. The loop closes.

The Number That Made Researchers Do a Double Take

The Fortune 500 study had a crucial detail. The company had been using a traditional, human-driven hiring process for years. The researchers had access to the actual résumés, the hiring decisions, and the subsequent performance data for thousands of candidates. They could see exactly where the human recruiters had made mistakes.

When they simulated what a supervised learning algorithm would have done, the results were stark. The algorithm would have hired fewer women and fewer Black candidates than the human recruiters did. Not because the algorithm was explicitly told to discriminate, but because it learned the pattern of discrimination embedded in the data.

The researchers found that the algorithm’s bias was not a bug. It was a feature of the design. The algorithm was optimizing for the wrong thing. It was trying to predict who would be a good employee based on past hires, not trying to discover who could be a good employee if given a chance.

This is where the exploration idea comes in. The researchers built a different algorithm, one that valued exploration. Instead of just ranking candidates by their historical probability of success, it ranked them by their “statistical upside potential.” It asked: what if this candidate is actually much better than the data suggests? How much could we learn by giving them a shot?

The exploration algorithm made different choices. It hired more candidates from underrepresented groups. And it did not sacrifice overall quality. In fact, the researchers found that the exploration approach could actually improve the quality of hires over time, because it gathered more information about the true distribution of talent.

The User Who Fights Back

There is another layer to this story. The 2023 paper on user strategization describes a phenomenon that hiring algorithms are particularly vulnerable to: users adapt their behavior to the algorithm.

Consider the job market. If a hiring algorithm is known to favor certain keywords, certain schools, or certain formatting styles, job seekers will change their résumés to match. They will add buzzwords. They will list skills they don’t have. They will lie about their experience.

This is called “gaming the system.” And it makes the algorithm’s job harder, because the data it is trained on is no longer a truthful reflection of candidates’ abilities. The algorithm is learning from a distorted signal.

But the problem goes deeper. The researchers show that when users adapt to an algorithm, the algorithm’s own predictions become less reliable. It starts to see patterns that aren’t there. It overweights certain features and underweights others. The result is a feedback loop that can amplify bias and reduce fairness.

In hiring, this means that the very act of using an algorithm changes the behavior of applicants. The algorithm is not a neutral observer. It is a participant in a dynamic system. And the system is constantly shifting under its feet.

The Sociologist’s Warning

Vivian Guetler, a sociologist at the University of Zurich, published a paper in 2023 called “Machine Habitus: Toward a Sociology of Algorithms.” The title is a play on the sociologist Pierre Bourdieu’s concept of “habitus,” the set of dispositions and habits that shape how people act in the world.

Guetler argues that algorithms develop their own kind of habitus. They learn from the data they are fed, and that data is a product of human society, with all its inequalities, biases, and unspoken rules. The algorithm does not transcend society. It absorbs it.

This is why the algorithm in the Fortune 500 study learned racism from résumés, not labels. It did not need to be told that Black candidates were less desirable. It learned from the pattern of who got hired and who did not. The résumés themselves contained the signal, encoded in the names, the schools, the zip codes, the extracurricular activities.

The algorithm was not a mirror. It was a magnifying glass. It took the subtle biases of human recruiters and amplified them into a systematic, scalable, and invisible system of discrimination.

What This Actually Means

• ▸Stop training hiring algorithms on historical decisions. The past is not a neutral baseline. It is a record of who got a chance and who did not. Train algorithms on performance data, not hiring data. Measure what people actually do on the job, not what their résumé looks like.

• ▸Build exploration into the algorithm’s objective function. Do not just optimize for predicted performance. Optimize for information gain. Give the algorithm a budget for trying candidates from underrepresented groups. The short-term cost of a few bad hires is outweighed by the long-term benefit of discovering hidden talent.

• ▸Assume users will game the system. Design your algorithm to be robust to strategic behavior. Use techniques like differential privacy, randomization, and adversarial training. Do not let the algorithm become a self-fulfilling prophecy.

• ▸Audit your algorithm for bias, but also audit the data it was trained on. The algorithm is not the problem. The problem is the data. And the data is a product of a society that is not fair. Fixing the algorithm without fixing the data is like painting over a crack in the foundation.

• ▸Treat hiring as a bandit problem, not a classification problem. The goal is not to predict the past. The goal is to discover the future. That requires exploration, not just exploitation. It requires a willingness to be wrong, to learn, and to change.