Your Research Paper Diagram Might Be Hiding Key Data Flaws

Your Research Paper Diagram Might Be Hiding Key Data Flaws

Imagine you are reading a systematic review. You know the kind: a meta analysis that claims to have gathered every relevant study on a topic, weighed the evidence, and produced a definitive answer. You flip to the flow diagram the PRISMA chart that shows how the authors went from thousands of initial records down to a handful of included studies. It looks clean. Professional. The boxes are neat, the numbers are round. You trust it.

You should not always trust it.

In 2022, a team of researchers led by Neal Haddaway at the Stockholm Environment Institute published a paper in Campbell Systematic Reviews that quietly exposed a problem many scientists had suspected but few had documented: the flow diagrams that supposedly guarantee transparency in systematic reviews are often incomplete, misleading, or just plain wrong (Haddaway et al., 2022). The authors reviewed existing evidence and found that use of flow diagrams in systematic reviews is "poor and of low quality." They called for standardized templates to fix the mess.

But here is the part that should make you nervous. The authors did not just identify the problem. They built a tool to solve it. And in doing so, they revealed just how much hidden data the typical flow diagram obscures.

What Your Flow Diagram Is Not Telling You

The PRISMA flow diagram is supposed to be the honest broker of systematic reviews. It documents every step: how many records you identified through database searching, how many duplicates you removed, how many records you screened, how many full text articles you assessed, and how many studies you finally included. If done right, it tells the reader exactly where studies disappeared.

But Haddaway and his colleagues found that most flow diagrams are not done right. The problem is not malice. It is a lack of standardization. Different authors use different templates. Some skip steps. Some merge categories. Some round numbers in ways that hide the messy reality of systematic review work.

The authors wrote that "recent research suggests that use of flow diagrams in systematic reviews is poor and of low quality." That is academic code for: a lot of these diagrams are garbage.

Think about what that means for the reader. When you see a flow diagram that shows 5,000 records identified, 4,800 excluded after screening, and 200 full text articles assessed, you assume the authors actually did that work. You assume the numbers add up. You assume the process was transparent.

But if the diagram is poorly constructed, you have no idea what you are missing. Maybe the authors double counted records. Maybe they excluded studies without documenting why. Maybe the numbers are internally inconsistent. The diagram gives you the illusion of transparency without the substance.

The Tool That Changes the Game

Haddaway and his team did not just complain. They built a solution. They developed PRISMA2020, a free, open source R package and web based Shiny app that allows researchers to produce PRISMA 2020 compliant flow diagrams (Haddaway et al., 2022). The tool is designed for two audiences: people who can code in R and people who cannot.

For the coders, there is the R package. For everyone else, there is the Shiny app, a web interface where you upload your data and the tool generates a standardized diagram. The authors made it free, open source, and accessible. They wanted to remove every excuse for producing a bad flow diagram.

But here is the clever part. The tool does not just produce static diagrams. It produces interactive ones. You can click on any box in the diagram and navigate to further details about the methods, results, or data files. The authors call this "interactivity for optimized digital transparency and Open Synthesis" (Haddaway et al., 2022).

In plain language: instead of a flat diagram that hides the messy details, you get a living document that lets readers verify each step.

Why Interactivity Matters More Than You Think

A static flow diagram is like a restaurant menu that lists prices but not ingredients. It tells you what you are getting, but not what went into it. An interactive diagram is like a menu with links to the kitchen cameras.

The authors argue that this interactivity allows readers to "smoothly and swiftly explore and navigate to further details of the methods and results of a review" (Haddaway et al., 2022). That is not just a convenience feature. It is a transparency mechanism.

Consider a typical scenario. A systematic review of clinical trials might start with 10,000 records from database searches. After removing duplicates, the authors screen 8,000 titles and abstracts. They exclude 7,500. They assess 500 full text articles. They exclude 450. They include 50 studies.

A bad flow diagram might just show: 10,000 identified, 50 included. The reader has no idea what happened in between.

A good static diagram shows each step. But an interactive diagram allows the reader to click on the "excluded after full text assessment" box and see a list of every excluded study, with reasons for exclusion. That is a fundamentally different level of transparency.

The authors built the tool to make this kind of verification possible. They wrote that it "will make it easier to produce clear and PRISMA 2020 compliant systematic review flow diagrams" (Haddaway et al., 2022). But the real innovation is the interactivity.

The Hidden Data in Your Own Research

If you are a researcher reading this, you might think the problem is other people's flow diagrams. Your diagrams are fine. You followed the template. You checked the numbers.

But here is the uncomfortable truth the authors uncovered: even well intentioned researchers produce flawed diagrams because the existing templates are ambiguous. The authors cite research showing that "use of flow diagrams in systematic reviews is poor and of low quality" and that standardized templates are needed (Haddaway et al., 2022).

The ambiguity comes from the PRISMA guidelines themselves. The 2020 update clarified some things, but it still leaves room for interpretation. How do you handle records identified through other sources? Do you count database searches separately? What about automated screening tools? The guidelines give principles, not precise instructions.

Haddaway and his team solved this by building a tool that enforces the standard. When you use the PRISMA2020 tool, you cannot skip steps. You cannot merge categories in non standard ways. The tool forces you to document every stage of the review process.

This is not just about compliance. It is about catching your own mistakes. When you manually construct a flow diagram in PowerPoint or a graphics editor, you can accidentally misalign numbers. You can forget to add a box. You can round in ways that obscure the truth. The tool prevents these errors.

How the Tool Works

The PRISMA2020 package is built on R, the statistical programming language. For users comfortable with code, the package provides functions to generate diagrams directly from data frames. For users who prefer a graphical interface, the Shiny app does the same thing through a web browser.

The authors describe the tool as "a free to use, Open Source R package and web based Shiny app to allow users to design PRISMA flow diagrams for their own systematic reviews" (Haddaway et al., 2022). The key word is "design." The tool does not just generate a diagram. It helps you design the process.

Users can input data on records identified from databases, registers, and other sources. They can specify numbers for each stage: screening, retrieval, assessment, and inclusion. The tool then generates a diagram that matches the PRISMA 2020 template exactly.

The output comes in multiple formats: PNG for static use, PDF for publication, and HTML for interactive web based diagrams. The interactive version is where the magic happens. Each box becomes a clickable link. Readers can explore the underlying data without leaving the page.

The authors provide an interactive example at https://prisma-flowdiagram.github.io/. Go look at it. Click on the boxes. You will immediately see the difference between a static diagram and a transparent one.

What This Research Does Not Prove

Let me be clear about what Haddaway and his colleagues did not show. They did not prove that all flow diagrams are bad. They did not show that systematic reviews with bad diagrams are necessarily wrong. And they did not claim that their tool is the only solution.

The authors focused on developing and testing the tool itself. They demonstrated that it works and that it produces PRISMA 2020 compliant diagrams. But they did not conduct a large scale audit of existing diagrams to measure the prevalence of errors. That is a separate question.

Here is what remains open: we do not know how many published systematic reviews have flow diagram errors. We do not know how often those errors lead to incorrect conclusions. And we do not know whether interactive diagrams will actually change how readers interpret reviews.

The authors are optimistic. They wrote that "this free to use tool will make it easier to produce clear and PRISMA 2020 compliant systematic review flow diagrams" (Haddaway et al., 2022). But ease of use does not guarantee adoption. Researchers are busy. They have established workflows. Convincing them to switch to a new tool is a different challenge than building the tool itself.

There is also a deeper question: does better documentation actually improve the quality of systematic reviews? Or does it just make bad reviews look more professional? A beautifully constructed flow diagram can still hide a poorly conducted review. The tool ensures transparency of process, not correctness of methods.

The Problem of Trust in Evidence Synthesis

Systematic reviews are supposed to be the gold standard of evidence. They aggregate all available studies, assess their quality, and produce a summary estimate. Policy decisions, clinical guidelines, and funding priorities all depend on them.

But the gold standard only works if the process is transparent. When flow diagrams are incomplete or misleading, the entire review becomes suspect. You cannot trust the conclusion if you cannot see the path that led there.

Haddaway and his colleagues recognized this. They built a tool that forces transparency. But the tool is only as good as the data fed into it. Garbage in, garbage out. A researcher who wants to hide excluded studies can still do so. The tool cannot prevent intentional deception.

What it can prevent is accidental error. And that is where the real value lies. Most researchers are not trying to deceive. They are just sloppy. They forget to document a step. They misplace a spreadsheet. They round numbers in a way that seems harmless but actually changes the narrative.

The PRISMA2020 tool catches these errors. It standardizes the process. It makes the hidden visible.

What This Actually Means

• ▸If you are writing a systematic review, use the PRISMA2020 tool. It is free, open source, and available as both an R package and a web app. It will catch errors you did not know you were making.

• ▸If you are reading a systematic review, check whether the flow diagram is interactive. If it is not, ask why. A static diagram hides details that an interactive one would expose.

• ▸If you are reviewing a manuscript for a journal, demand standardized flow diagrams. The PRISMA2020 tool exists. There is no excuse for hand drawn diagrams or non standard templates.

• ▸If you are a journal editor, update your submission guidelines to require PRISMA 2020 compliant diagrams. The tool makes compliance easy. Accept nothing less.

• ▸If you are a policymaker or clinician using systematic reviews to make decisions, understand that the flow diagram is not just decoration. It is a transparency document. If it looks incomplete, the review might be hiding something important.

The next time you see a flow diagram, do not just glance at it. Click on it. If you cannot click, wonder why. The diagram that looks clean might be hiding the messiest data of all.