Scientists Must Question Their Own Biases in Data

The Secret Weapon Your Statistics Professor Never Mentioned

Here is a confession from three quantitative researchers who have spent years staring at p-values, regression tables, and confidence intervals: they are not neutral. The numbers do not speak for themselves. And the scientist who believes they are a blank slate is the most dangerous kind of researcher.

Michelle Jamieson, Gisela Govaart, and Madeleine Pownall published a paper in 2023 that amounts to a quiet rebellion against the culture of quantitative psychology. Their argument, laid out in Social and Personality Psychology Compass, is simple and unsettling: quantitative researchers have been avoiding a practice that qualitative researchers have embraced for decades. The practice is called reflexivity. It is the act of turning the scientific lens back on the scientist. And according to Jamieson et al. (2023), it is long overdue in quantitative research.

The authors are not fringe critics. They are three quantitative researchers who have "grappled with the compatibility of reflexivity within our own research." They know the terrain. They know the pressure to produce clean, objective results. And they know that pretending the researcher does not exist is a form of self-deception.

What Does It Mean to Be a Reflexive Quantitative Researcher?

Reflexivity sounds like a buzzword until you understand what it asks of you. It is not about confessing your feelings or writing poetry about your data. It is about systematically examining how your assumptions, beliefs, and judgments shape every decision you make in the research process (Jamieson et al., 2023).

The authors break this down into concrete moves. You ask yourself: Why did I choose this particular research question and not another? Why did I select this sample and not that one? Why did I decide to exclude certain participants? Why did I choose this statistical test over its alternatives? Why did I interpret this result as interesting and that one as noise?

These are not abstract philosophical questions. They are practical, methodological decisions that determine what you find and what you miss. The authors argue that ignoring these questions does not make your research more objective. It makes your biases invisible, which is worse.

The Qualitative-Quantitative Divide

Here is the irony that Jamieson et al. (2023) are trying to dissolve. Qualitative researchers have been practicing reflexivity for decades. They are trained to acknowledge their positionality: their gender, their race, their class, their theoretical commitments, their personal history. They treat this as standard practice, part of the craft.

Quantitative researchers, by contrast, have treated reflexivity as a foreign concept. The ideal of the detached, neutral observer still dominates quantitative training. You learn to run analyses, not to examine yourself. The authors note that this divide has created a strange asymmetry. Qualitative work is seen as subjective but honest about its subjectivity. Quantitative work is seen as objective but rarely examines the subjective choices embedded in its methods.

The authors are not arguing that quantitative research is hopelessly biased. They are arguing that it is human, and pretending otherwise does not make it more scientific.

The Hidden Choices Behind Every Number

Let me give you a concrete example of what reflexivity looks like in practice. Imagine you are studying whether a new teaching method improves student performance. You design an experiment. You recruit participants. You run your analysis. You find a statistically significant effect.

Now imagine you pause and ask yourself: Who did I recruit? Did I recruit from my own university, where students are mostly middle class and academically motivated? Did I exclude students who did not show up for the second session, which might have been the struggling ones? Did I choose a measurement that favors the kind of learning I personally value? Did I run the analysis three different ways before settling on the one that gave me a significant result?

These are not accusations. They are questions. And the answers matter because they tell you what your finding actually means. Jamieson et al. (2023) argue that reflexivity is not about eliminating these choices. That is impossible. It is about making them visible so that readers can evaluate them.

The Beginner's Guide

The authors do not just make the case for reflexivity. They provide a "beginner's guide" for quantitative researchers who want to try it. The guide includes concrete recommendations: keep a reflexive journal during your study, write a positionality statement before you collect data, discuss your decisions with colleagues who disagree with you, and document your analytical choices in a transparent way.

They also provide reflexive prompts. Here are a few examples from the paper: "What assumptions am I making about the population I am studying?" "How might my own identity influence the way I interpret these results?" "What would I think if the results had come out the opposite way?"

These prompts are not soft. They are hard. They force you to confront the possibility that your cherished hypothesis might be wrong, or that your method might be shaped by things you do not want to admit.

Why This Matters for the Replication Crisis

The timing of this paper is no accident. Psychology has been in a crisis of confidence for over a decade. Studies fail to replicate. Questionable research practices have been exposed. The field has responded with pre-registration, open data, and transparency reforms. But Jamieson et al. (2023) argue that these reforms are not enough.

Pre-registration tells you what a researcher planned to do. It does not tell you why they planned to do it. Open data lets you re-analyze the numbers. It does not tell you what the researcher was thinking when they chose those numbers. Reflexivity addresses the human element that technical fixes cannot reach.

The authors are careful not to claim that reflexivity will solve the replication crisis. But they argue that it is a necessary complement to the reforms already underway. Without reflexivity, you can follow all the rules of open science and still produce work that is shaped by unexamined biases.

A Concrete Example from the Authors

The paper includes worked examples of how reflexivity can change the interpretation of a quantitative study. One example involves a researcher studying the effects of a mindfulness intervention on anxiety. The researcher finds a significant effect. But after engaging in reflexivity, they realize that they personally believe strongly in mindfulness. They have been practicing it for years. They have a professional stake in the results.

Does this mean the results are invalid? No. But it means the researcher should report this positionality. It means readers should know that the researcher had a personal investment in finding a positive effect. It means the study should be interpreted with that context in mind.

This is not an attack on the researcher. It is a call for honesty. And honesty, the authors argue, is what makes science trustworthy.

What the Research Does Not Prove

It is important to be clear about what Jamieson et al. (2023) are not claiming. They are not saying that all quantitative research is biased beyond repair. They are not saying that reflexivity replaces rigorous methods. They are not saying that qualitative methods are superior.

They are also not claiming that reflexivity is easy. The authors acknowledge that it can be uncomfortable. It requires you to admit that you are not a neutral observer. It requires you to question your own expertise. It requires you to be vulnerable in a culture that rewards confidence.

And there is an open question that the authors do not fully resolve: How do you measure the impact of reflexivity? If a researcher writes a positionality statement, does that actually improve the quality of their work? Does it reduce bias? Does it change outcomes? The authors argue for the practice based on theoretical grounds and their own experience, but they do not provide experimental evidence that reflexivity works in quantitative contexts.

This is not a flaw in the paper. It is an honest recognition of the limits of their argument. Reflexivity is a practice, not a treatment. Its effects are hard to quantify. But that does not mean it is not worth doing.

How to Start Being Reflexive Tomorrow

If you are a quantitative researcher reading this, you might be wondering what to do on Monday morning. The authors provide a practical path:

• ▸Keep a reflexive journal. Before you start a study, write down your assumptions. What do you expect to find? Why do you expect that? What would it mean if you were wrong? Update the journal as the study progresses.

• ▸Write a positionality statement. This is not a confessional. It is a brief, honest description of who you are in relation to the research: your training, your theoretical commitments, your personal experiences, your funding sources.

• ▸Discuss your decisions with someone who disagrees. Find a colleague who has a different perspective and ask them to critique your design, your analysis, your interpretation. Listen to what they say.

• ▸Document your analytical choices. If you run multiple analyses, report them all. If you make a decision about exclusion criteria, explain why. If you change your analysis plan, say so and say why.

• ▸Read your own work as if you were a skeptic. Imagine that someone who doubts your hypothesis is reading your paper. What would they question? Address those questions before they are asked.

None of these steps require expensive equipment or advanced training. They require a willingness to be uncomfortable. And that, the authors suggest, is the real barrier.

What This Actually Means

• ▸Objectivity is not the absence of bias. It is the acknowledgment of it. A researcher who pretends to be neutral is not more objective. They are less transparent. Reflexivity makes your biases visible so that others can account for them.

• ▸Your research questions are not neutral. You chose to study this topic and not that one. You chose this population and not that one. Those choices reflect your values, your training, your funding, your career incentives. Acknowledging this does not invalidate your work. It makes it interpretable.

• ▸The replication crisis is partly a crisis of self-awareness. Technical reforms like pre-registration and open data are necessary but insufficient. They do not address the human decisions that shape every study. Reflexivity fills that gap.

• ▸Reflexivity is a skill, not a personality trait. It can be learned and practiced. The authors provide a beginner's guide because they believe that quantitative researchers can develop this skill. It takes effort, but it is not mysterious.

• ▸The best science is honest about its limits. A study that admits its biases is more trustworthy than one that hides them. Reflexivity does not weaken your findings. It strengthens them by giving readers the context they need to evaluate them.

The authors end their paper with a call that is both modest and radical: "We argue that reflexivity has much to offer quantitative methodologists." That is an understatement. What they are really saying is that the scientist who does not question themselves is not a scientist at all. They are a machine that produces numbers, and machines do not know what they are missing.