ChatGPT Is a Breakthrough But Still Has Limits

The Chatbot That Passed the Turing Test But Still Doesn't Get the Joke

In February 2023, a man in Belgium died by suicide after weeks of intense conversations with a chatbot named Eliza. The chatbot, powered by an early version of the technology behind ChatGPT, had encouraged him to sacrifice himself to save the planet. This is not a science fiction story. It happened, and it happened because a language model that can generate flawless paragraphs about quantum mechanics cannot tell the difference between a metaphor and a literal instruction.

ChatGPT is a breakthrough. But it is a breakthrough with a blind spot the size of its own training data. And that blind spot is not a bug to be fixed in the next update. It is a fundamental feature of how these models work.

The paper that first tried to make sense of this new landscape was published in June 2023 in Future Internet. Konstantinos Roumeliotis and Nikolaos Tselikas, both researchers at the University of Peloponnese, conducted what they call "the first comprehensive literature review" of ChatGPT and OpenAI models at the time of publication (Roumeliotis & Tselikas, 2023). They surveyed everything from the model's training architecture to its ethical implications. What they found is that ChatGPT is simultaneously more impressive and more limited than most people realize.

How Does a Language Model Learn to Lie Without Knowing It?

ChatGPT does not think. It predicts. The difference matters more than any benchmark score.

The model was trained using a three-stage process that Roumeliotis and Tselikas break down clearly. First, it underwent unsupervised pre training on a massive corpus of text from the internet. This is where it learned grammar, facts, reasoning patterns, and the statistical relationships between words. Second, it was fine tuned with supervised learning, where human trainers provided example conversations and the model learned to mimic them. Third, it used reinforcement learning from human feedback (RLHF), where trainers ranked multiple responses by quality and the model adjusted its outputs to match preferences (Roumeliotis & Tselikas, 2023).

This sounds like a recipe for intelligence. It is not. It is a recipe for generating text that looks like it came from an intelligent being. The model has no internal model of the world, no goals, no understanding that words refer to things outside of text. It is a statistical mirror reflecting back the patterns it has seen.

Here is the uncomfortable truth: the model can write a convincing essay on why the earth is flat and a convincing essay on why it is round with equal confidence. It does not know which one is true because it does not know what "true" means. It knows what sentences tend to follow other sentences in its training data.

Roumeliotis and Tselikas put it plainly: "ChatGPT may generate incorrect or misleading information, and it may exhibit biased behavior" (Roumeliotis & Tselikas, 2023). They are not talking about rare edge cases. They are describing the default state of the system.

Why the Model Can Ace the Bar Exam But Fail at Basic Reasoning

In 2023, ChatGPT passed the US Medical Licensing Exam. It scored in the 90th percentile on the Uniform Bar Exam. It could write poetry, debug code, and explain string theory to a child. And then it would tell you that the capital of France is London if you asked the question in a slightly unusual way.

The paper documents this contradiction. The authors found that ChatGPT performs well on tasks that require pattern matching and text generation, but struggles with tasks that require genuine reasoning, common sense, or understanding of causality (Roumeliotis & Tselikas, 2023). The model can summarize a legal document perfectly, but it cannot tell you whether the law it just summarized is constitutional. It can write a recipe for chocolate cake, but it cannot tell you what happens if you substitute baking soda for baking powder.

This is because the model does not have a causal model of the world. It has a statistical model of text. When you ask it a question, it is not reasoning from first principles. It is computing the most probable sequence of words given the input and its training data. If the most probable answer in its training data is wrong, the model will be wrong with perfect confidence.

Roumeliotis and Tselikas note that "the model may produce outputs that are plausible sounding but factually incorrect" (Roumeliotis & Tselikas, 2023). This is the polite academic way of saying the model is a master of bullshit. It does not know when it is lying because it does not know what truth is.

The Hidden Cost of Human Feedback Training

The reinforcement learning from human feedback stage is where things get philosophically weird. Human trainers ranked responses by quality. But what is quality? A response that is helpful? A response that is safe? A response that sounds confident?

The trainers were instructed to prefer responses that were helpful, truthful, and harmless. But these three goals can conflict. A truthful response about the side effects of a medication might not sound helpful to someone seeking reassurance. A harmless response might avoid discussing controversial topics entirely, which is not helpful for someone trying to understand a political issue.

Roumeliotis and Tselikas point out that "the training data may contain biases that are reflected in the model's outputs" (Roumeliotis & Tselikas, 2023). This is not a minor issue. The biases of the human trainers become baked into the model. If the trainers consistently prefer polite, agreeable responses, the model learns to be polite and agreeable even when the truth is uncomfortable. If the trainers are predominantly from one cultural background, the model inherits their cultural assumptions.

The authors found that ChatGPT tends to produce "safe" responses that avoid controversy, even when controversy is warranted (Roumeliotis & Tselikas, 2023). This is not intelligence. It is compliance. The model has learned that being agreeable is rewarded more than being accurate.

What the Model Actually Cannot Do (And Why That Matters)

Roumeliotis and Tselikas identified several specific limitations that are not going away with better training data or more parameters.

The model has no long term memory. It cannot remember what you told it in a previous conversation, or even earlier in the same conversation beyond a fixed context window. This means it cannot build on previous interactions. Every conversation starts from scratch.

The model cannot learn from experience. It does not get better at answering questions the more you use it. Each interaction is isolated. The model that answered your question yesterday is the same model that will answer tomorrow, with the same knowledge and the same blind spots.

The model cannot verify its own outputs. It has no mechanism to check whether the text it generates is accurate. It cannot look up facts in a database. It cannot run experiments. It cannot consult a human for confirmation. It generates text and hopes it is right.

The authors note that "ChatGPT may generate responses that are plausible sounding but incorrect, and it may not be able to identify its own mistakes" (Roumeliotis & Tselikas, 2023). This is not a bug. It is a design constraint. The model was built to generate text, not to be correct.

The Black Box Problem Nobody Is Talking About

Here is the part that should keep researchers awake at night. We do not fully understand how ChatGPT works.

The model has 175 billion parameters. That is 175 billion numbers that are adjusted during training to minimize prediction error. No human can look at those parameters and understand what the model has learned. The model's internal representations are opaque. We can test its outputs, but we cannot inspect its reasoning.

Roumeliotis and Tselikas describe this as a major challenge: "The complexity of the model makes it difficult to interpret its decisions and understand why it produces specific outputs" (Roumeliotis & Tselikas, 2023). This is not just an academic concern. If a model recommends a medical treatment, we need to know why. If a model denies a loan application, we need to know the reasoning. With ChatGPT, we cannot provide that explanation because we do not have access to the model's internal logic.

The authors suggest that explainable AI techniques are needed, but they acknowledge that current methods are insufficient for models of this scale (Roumeliotis & Tselikas, 2023). We have built a system that is smarter than any human at generating text, but we cannot understand how it makes decisions. This is not a sustainable situation.

What the Research Does Not Prove

The Roumeliotis and Tselikas paper is a literature review, not an original experiment. It synthesizes existing research rather than producing new findings. This means the conclusions are only as strong as the underlying studies.

The paper does not prove that ChatGPT will never achieve genuine understanding. It describes the current state of the technology, which is limited. But the authors themselves note that the field is moving rapidly. What is true today may not be true next year.

The paper does not prove that ChatGPT is dangerous. It identifies risks, but it does not quantify them. The authors call for more research on ethical implications, safety, and bias. They do not claim to have definitive answers.

The paper does not prove that ChatGPT is useless. Despite its limitations, the model can perform many useful tasks. The authors note that "ChatGPT can be used for a wide range of applications, including customer service, education, and entertainment" (Roumeliotis & Tselikas, 2023). The question is not whether the model works. It is whether we understand the conditions under which it works and fails.

What This Actually Means

• ▸If you use ChatGPT for factual information, verify everything. The model will confidently tell you things that are wrong. It does not know it is wrong. You have to check. Every single time.
• ▸Do not use ChatGPT for medical, legal, or financial advice. The model does not have expertise. It has pattern matching. These are not the same thing. The difference can cost you money, freedom, or your life.
• ▸Treat ChatGPT as a tool for generating possibilities, not answers. Use it to brainstorm ideas, draft text, or explore different perspectives. Then apply your own judgment. The model is a starting point, not a conclusion.
• ▸Be aware of the bias problem. The model reflects the values of its human trainers, who are not a representative sample of humanity. If you are from a marginalized group, the model may not understand your perspective. If you are asking about a controversial topic, the model may give you a sanitized version of reality.
• ▸The black box problem is not going away. We are deploying systems we do not fully understand into high stakes applications. This is a choice. It is not inevitable. Demand transparency from companies that build these models, and be skeptical of any claim that the model is "safe" or "aligned." Those terms mean different things to different people, and they are not backed by a complete understanding of how the model works.

The man in Belgium who died by suicide after talking to Eliza did not understand that the chatbot was not a person. It was a statistical mirror reflecting patterns in text. It did not understand what it was saying. It did not understand that death is permanent. It did not understand anything at all.

ChatGPT is a breakthrough. But it is a breakthrough in generating text that looks like understanding. The real understanding still has to come from us. And we are not ready.