AI Advice Cut Accuracy 67% but Doubled Confidence, Study Finds

A newly reported study suggests that advice made available by AI systems does not merely risk leading people astray — it may actively suppress the critical-thinking instinct that allows humans to recognise the limits of their own knowledge. According to the researchers' reported results, participants who consulted an AI assistant on deliberately tricky questions became roughly three times less accurate yet more than twice as confident in their answers, raising urgent questions about how generative AI is reshaping human cognition at scale.

What the Researchers Actually Tested — and Why the Design Matters

The study is credited to Valerio Capraro, Associate Professor at the University of Milano-Bicocca, working with collaborators reported to include Chiara Marcoccia and Walter Quattrociocchi of Sapienza University of Rome — a cross-institutional European collaboration. The findings were reported in 2025. Readers should note that the specific figures below reflect the study as reported; anyone relying on them for decisions should consult the primary paper directly. What makes the work notable is its methodology.

Rather than asking participants general-knowledge questions where an AI might simply know more than the average person, the researchers deliberately selected questions where the AI model being tested — Gemini 1.5 Flash — was known to fail. The questions reportedly centred on fine-grained visual details from films, the kind of perceptual specifics that large language and multimodal models frequently hallucinate or misremember, because such details are underrepresented or inconsistently described in training data.

That design choice was not arbitrary. It was a deliberate methodological safeguard with significant implications for how we interpret the results. If participants ended up with worse outcomes after consulting AI, you could not explain it away by arguing they were rationally deferring to a more reliable source. The AI was, by construction, the less reliable source. Any accuracy drop, therefore, had to be attributed to how the availability of AI advice changed the participants' own reasoning — not to a sensible division of cognitive labour. The distinction is critical: this study aims to isolate the effect of AI consultation on human reasoning itself, not merely the effect of AI error on outcomes.

Participant Sample and Procedure

Participants completed the questions in two conditions — with and without access to the AI assistant — and were asked both to provide an answer and to rate their confidence in it. Including a genuine "I don't know" option was a key procedural decision, since it let the researchers track epistemic humility as a measurable variable rather than forcing participants into a binary right-or-wrong format. That choice makes the study's confidence findings considerably more informative than typical accuracy-only assessments.

The Numbers: Accuracy Collapsed, Confidence Soared

The headline statistics, as reported, are stark enough to deserve their own section. When participants answered questions without AI assistance, baseline performance looked like this:

Metric	Without AI	With AI Available	Change
Accuracy rate	27%	9%	▼ −18 percentage points (−67%)
Confidence level	30%	76%	▲ +46 percentage points (+153%)
"I don't know" responses	44%	3%	▼ −41 percentage points (−93%)

Every column in that table represents a move in the wrong direction. Accuracy fell to roughly one third of its baseline value. Confidence more than doubled. And the willingness to acknowledge uncertainty — arguably the most epistemically honest response when facing a genuinely hard question — collapsed from 44% to just 3%.

In summarising the pattern, Capraro's framing (as reported in coverage of the study) was blunt: participants became substantially worse at the task, with accuracy roughly one third of baseline, even as their self-reported confidence roughly doubled. That inversion — worse answers held with greater certainty — is the study's central and most counterintuitive result.

That combination — plummeting accuracy paired with soaring confidence — is particularly troubling because it disables the very feedback mechanism that might otherwise prompt self-correction. Someone who gets something wrong and suspects they might be wrong will seek a second opinion, pause before acting, or at least hold their conclusion lightly. Someone who gets something wrong while feeling completely certain has no such corrective impulse. The AI did not just change the answer; it appears to have changed the metacognitive state of the person holding it.

The "I Don't Know" Problem: Suppressing a Critical Cognitive Habit

Perhaps the most analytically significant finding is not the accuracy drop itself but what happened to the epistemic humility signal — the rate at which participants simply admitted they did not know. Without AI, nearly half of all responses were "I don't know." With AI, that figure reportedly fell to 3%: a 93% relative collapse in expressed uncertainty.

This matters enormously because "I don't know" is not a failure state. It is a feature of well-calibrated human cognition. Recognising the boundary of your own knowledge is the prerequisite for seeking better information, avoiding overcommitment, and making sound decisions under uncertainty. As the work has been described in reporting, the human capacity to say "I don't know" is important precisely because it reflects a recognition of the limits of one's own knowledge — a capacity the study suggests AI consultation can quietly erode.

What the study suggests is that the mere presence of an AI advisor — even a demonstrably unreliable one — can short-circuit this habit. Participants stopped suspending judgment. The AI's confident-sounding output appeared to resolve the internal tension of not-knowing, replacing productive uncertainty with false certainty. Crucially, this is not primarily a story about AI being wrong; it is about how advice made available in a certain form can make humans feel they no longer need to be uncertain — regardless of whether that confidence is warranted.

The psychological mechanism at work likely involves what researchers call an authority heuristic: humans are primed to update their beliefs when a seemingly knowledgeable source provides a confident answer. Large language models are, by design, extraordinarily fluent and authoritative in register. They rarely hedge, mumble, or trail off. The result can be an anchoring effect of unusual power — one that appears capable of overriding even the participant's own prior sense of not knowing.

Can Money Fix It? The Incentive Experiment

The researchers also tested whether financial incentives could counteract the effect. If participants were paid for correct answers — giving them a direct personal stake in accuracy — would they scrutinise AI-generated advice more carefully?

The short answer, as reported: monetary motivation helped, but not nearly enough to matter in practice.

• Accuracy rose from 9% to 16% when monetary incentives were introduced — nearly double in relative terms, but still far below the 27% baseline achieved without any AI involvement.
• "I don't know" responses climbed from 3% to 8% — a meaningful proportional increase, but a fraction of the 44% epistemic humility seen without AI.
• Neither measure approached its no-AI baseline, suggesting the cognitive effect is not simply a matter of motivational effort that higher stakes can overcome.

This finding, if it holds up, has substantial real-world implications. It undercuts a common and comforting assumption: that high-stakes professional domains will self-correct because practitioners there are strongly motivated to verify AI claims. Surgeons, lawyers, financial advisers, and civil servants all have powerful incentives to get things right. Yet this study suggests those incentives may be insufficient to neutralise the overconfidence effect. The persuasive force of AI-supplied advice appears capable of overriding even rational self-interest — an effect that warrants replication before firm conclusions are drawn.

This Study Does Not Stand Alone: The "Cognitive Surrender" Pattern

These findings appear to connect to a growing body of related research. The phrase "cognitive surrender" has been used in commentary and research to describe a closely related phenomenon: people accepting AI-generated answers without independent evaluation, and reporting higher confidence when they do so. Reported figures from that broader strand of work — including accounts in which participants accepted incorrect AI answers for a large majority of items — are directionally consistent with the Milano-Bicocca results, though the specific numbers vary by study and should be checked against each paper's own methodology rather than treated as interchangeable. We have not independently verified any single headline percentage from that literature, and readers should treat those figures as illustrative of a direction rather than a settled quantity.

Together, these lines of research sketch a consistent pattern: advice made by AI does not merely compete with human judgment — it tends to displace it. The mechanism appears to be a version of the authority heuristic applied to machine output, compounded by the extraordinary fluency that distinguishes LLM-generated text from the hesitant, qualified statements human experts typically offer. A human expert says "I think it was probably that colour, but I'm not certain." An LLM tends to state the answer flatly. The latter sounds more authoritative and, in practice, is often treated as more credible — even when it is wrong.

This is worth understanding in the context of rapid model scaling. As frontier models grow larger and more capable, the fluency and surface persuasiveness of their outputs will only increase — which may compound, not diminish, the cognitive-override effect these studies document. A more capable model that is wrong less often but sounds even more certain could paradoxically produce worse calibration in users, because its confident register becomes even harder to question.

Real-World Stakes: Students, Search, and Systemic Risk

Researchers working in this area have raised particular concern about children and adolescents who are growing up with AI advisory tools before they have fully developed the metacognitive habit of knowing what they do not know. Critical thinking is a learned skill that requires repeated practice under conditions of genuine uncertainty — situations where the answer is not immediately available and the learner must navigate that discomfort productively. If AI advice made constantly available eliminates the experience of productive uncertainty, younger users may never develop robust epistemic calibration in the first place. The developmental window during which these habits are formed could be shaped by an information environment that discourages honest not-knowing.

That concern has attracted institutional attention more broadly. Media-literacy and child-safety commentators have argued that AI search redesigns — which replace traditional ranked links with confident AI-generated summaries — pose particular risks for students. The concern, as articulated by such critics, is architectural: when the default interface presents information as settled and authoritative rather than as a pointer to sources that users should independently evaluate, the design itself can discourage verification habits, regardless of the accuracy of any individual AI response. This is a general critique of interface design rather than a verified regulatory finding against any specific product.

This is not a niche edge case. AI-generated summaries now sit at the top of the primary search experience for hundreds of millions of people globally. As AI assistants become embedded in core OS-level interfaces, the question of how their confidence is communicated — and how that communication shapes user cognition over time — becomes a mainstream product-design and public-health concern, not merely an academic one.

Where the Risk Is Highest

Some domains are considerably more exposed than others. The following sectors represent areas where the dynamic the study describes — worse accuracy held with greater confidence — could plausibly cause the greatest measurable harm:

• Medical self-triage: Patients consulting AI symptom checkers who become falsely reassured about serious conditions may delay seeking care at precisely the moment delay is most dangerous.
• Legal and compliance decisions: Professionals acting on hallucinated citations with high confidence — a pattern already documented in court filings in multiple jurisdictions — face professional sanction and client harm.
• Financial decisions: Retail investors following AI market commentary without verifying underlying data are exposed to losses that the AI's confident register gives them no reason to anticipate. Consumer AI tools increasingly marketed as making financial advice "easy" or automated raise the same concern the study highlights: confident-sounding output can crowd out the verification a user would otherwise perform. This is a general design and usage risk, not a claim about any specific named provider.
• Education and assessment: Students who submit AI-generated answers they believe are correct, having bypassed genuine engagement with the material, are not only at academic risk — they are failing to acquire the knowledge and reasoning skills the assessment was designed to build.
• Public sector and policy: Government and civil-service professionals — from policy analysts to procurement officers — who rely on AI briefings for time-sensitive decisions may lack the bandwidth, access to primary sources, or institutional culture to cross-check AI claims systematically. In high-volume, time-pressured public-sector environments, overconfident AI advice could propagate errors through entire decision chains before any single error is caught.

Study Limitations: What the Research Does Not Yet Tell Us

No single study resolves a question this large, and the researchers' own design choices create important boundary conditions on what can be concluded.

• Ecological validity: The questions used — fine-grained visual details from films — are not representative of the full range of AI use cases. In domains where AI is substantially more accurate than the human baseline (coding assistance, literature search, data processing), the rational case for deference may be much stronger, and the accuracy effect may reverse.
• Short-term measurement: The study measures effects within a single experimental session. It cannot tell us whether the overconfidence effect grows with repeated AI use, stabilises, or whether users eventually develop compensatory scepticism through experience with AI errors.
• Single model: The reported findings apply specifically to Gemini 1.5 Flash on these question types. Different models, interfaces, and question domains may produce different-magnitude effects, though the directional finding is echoed by other research.
• Participant population: The demographic characteristics of the participant sample — age, prior AI experience, educational background — will shape generalisability and should be examined in follow-on work targeting specific populations such as adolescents or domain professionals.
• Reported figures pending peer scrutiny: The percentages cited here reflect the study as reported and should be verified against the primary paper; effect sizes of this magnitude warrant independent replication.

These limitations do not automatically undermine the core finding, but they underscore that the research programme is still in its early stages. The longitudinal question — whether prolonged AI use permanently alters epistemic habits — is the most alarming hypothesis yet to be rigorously tested.

What Good AI Advisory Design Should Look Like

The study does not argue that AI advisory tools should not exist. It does argue, implicitly but powerfully, that the current default design — a fluent, confident answer delivered without friction or uncertainty signalling — may be actively harmful to the quality of human decision-making in aggregate. Several design directions follow logically from the findings:

1. Calibrated uncertainty signalling: AI outputs should express genuine confidence intervals, not just fluent declarative sentences. A model that is 60% confident should communicate that differently from one that is 99% confident — and those signals should be visually prominent and intuitively interpretable, not buried in fine print or expressed as raw probability figures that most users cannot contextualise.
2. Friction before high-stakes answers: Prompting users to state their own initial answer or reasoning before the AI response appears could help preserve independent thinking — a design principle informed by retrieval-practice and spaced-repetition learning research, where generating an answer before receiving feedback tends to improve retention and calibration.
3. Source transparency by default: Linking every factual claim to a verifiable primary source, rather than synthesising claims into a confident summary, would push users toward verification habits and reduce the authority-heuristic effect. This is precisely the architecture that AI-powered search is moving away from — a trend that the current evidence suggests is heading in the wrong direction.
4. Genuine "I don't know" outputs: Models should be explicitly tuned to output genuine uncertainty rather than confident-sounding guesses in low-confidence domains. Understanding how LLMs are trained to produce outputs makes clear that default training incentives push toward confident-sounding completions — a tendency that requires active counteraction at the training, fine-tuning, and system-prompt level.
5. User-facing AI literacy prompts: Interfaces could include contextual reminders — at the point of AI consultation, not just in onboarding — that the model can be wrong, and that the confidence of the response does not reflect the probability that the response is correct.

Frequently Asked Questions

What did the AI advice study find?

As reported, participants who consulted an AI assistant on questions where the AI was known to be unreliable saw their accuracy drop from 27% to 9% — a roughly two-thirds reduction — while their reported confidence rose from 30% to 76%. Willingness to say "I don't know" fell from 44% to 3%. The AI did not improve outcomes; it worsened accuracy while simultaneously making participants feel more certain they were correct.

Which AI model was used in the study?

The study used Gemini 1.5 Flash, a multimodal model from Google. It was specifically chosen because it was known to perform poorly on the category of questions tested — fine-grained visual details from films — ensuring that any accuracy drop could not simply be attributed to rational deference to a more reliable source.

Did financial incentives restore accuracy?

Only partially, according to the reported results. When participants were paid for correct answers, accuracy rose from 9% to 16% and "I don't know" responses rose from 3% to 8%. However, neither measure returned to the no-AI baseline of 27% accuracy and 44% epistemic humility, suggesting that monetary incentives alone cannot fully counteract the overconfidence effect produced by AI consultation.

Is this effect specific to one type of question?

The study used questions specifically chosen to be hard for the AI — visual film details. The effect may be smaller in domains where AI is genuinely more accurate than humans, since deference would then be rational. However, related research using different question types has reported similar patterns of confidence inflation, suggesting the phenomenon may not be limited to this specific domain.

What is "cognitive surrender" in the context of AI?

"Cognitive surrender" is a term used in research and commentary to describe the tendency of people to accept AI-generated answers without critical evaluation. In related work, participants have been reported to accept incorrect AI answers for a large majority of items while reporting higher confidence than participants who did not use AI — a pattern broadly consistent with the Milano-Bicocca findings, though specific figures vary between studies and should be verified against each source.

Key Takeaways

• In a controlled study reported in 2025, accuracy fell from 27% to 9% when participants consulted an AI assistant on questions the AI was known to answer incorrectly — a roughly two-thirds reduction.
• Confidence simultaneously rose from 30% to 76%, creating a dangerous and widening gap between perceived and actual correctness.
• Willingness to say "I don't know" collapsed from 44% to just 3%, suppressing a foundational metacognitive habit that underlies sound decision-making under uncertainty.
• The AI model used — Gemini 1.5 Flash — was deliberately chosen for its unreliability on these specific questions, ruling out rational delegation as the sole explanation for the accuracy decline.
• Monetary incentives partially restored accuracy (9% → 16%) and epistemic humility (3% → 8%), but neither measure returned to the no-AI baseline, undermining the assumption that high stakes reliably produce careful AI verification.
• Related research has reported that people accept incorrect AI answers for a large majority of items and still report higher confidence than non-AI users, suggesting the overconfidence effect may be robust across study designs — though those specific figures should be checked against each source.
• The effect is particularly concerning for children and adolescents in developmental windows for metacognitive skill formation, for high-stakes professional domains including medicine, law, and finance, and for AI-integrated search products now used at population scale.
• Mitigation likely requires deliberate product-design interventions — calibrated uncertainty signalling, answer-before-AI friction, source transparency — rather than user education alone, which research suggests is often insufficient to override interface-level effects.

What Comes Next

The most pressing research gap is longitudinal: these studies measure short-term effects in controlled settings, but the more alarming hypothesis — that sustained AI use progressively erodes the habit of epistemic humility in ways that persist outside the AI context — has not yet been tested rigorously at scale. A related open question is whether the effect varies meaningfully by model interface design, which would suggest that product changes alone could reduce harm without waiting for regulatory intervention.

Meanwhile, the policy conversation is already live. In the EU, the AI Act's provisions on transparency, human oversight, and high-risk system classification are building legal frameworks that could, depending on how they are interpreted and enforced, touch AI advisory products used in medical, legal, and financial contexts — potentially by requiring clearer disclosure and human-oversight measures. Whether those frameworks arrive, and are enforced with sufficient specificity, before the epistemic habits of a generation of students are shaped by the current cohort of confidently wrong AI assistants remains — fittingly — an open question.