Automation bias: why reviewers rubber-stamp AI answers, and how to stop

An AI tool hands you a tidy page of requirements. It reads well, it uses the right vocabulary, and it took four seconds. You read it through, it all seems reasonable, and you move on. Somewhere in that page is a sentence that no source document supports.

That is the pattern this guide is about. The risk is not only that the tool is wrong. It is that a reviewer who has just read a confident answer finds it hard to see what is missing from it. This guide sets out why that happens and a review step that works on a real assignment.

Northgate Bank, its people and its documents are fictional.

Last reviewed: October 2026.

What automation bias is

Automation bias is the tendency to defer too readily to an automated system. The National Institute of Standards and Technology describes it, in its profile of generative AI risks, as people over-relying on AI systems or unjustifiably perceiving AI content to be of higher quality than content from other sources. NIST also notes that it can make other risks worse, such as confident but invented content.

Regulators have noticed the same thing in practice. In its October 2026 publication on AI in financial services, the Financial Conduct Authority describes a firm where the human reviewer's role was unclear and became a tick-box exercise over time. Reviewers increasingly accepted AI-generated recommendations without enough scrutiny and became less likely to challenge, even where there were signs of errors. The same publication says that effective oversight is not simply about reviewing AI outputs: reviewers need the authority, information and skills to challenge. It also lists managing automation bias among the things a person in the loop should understand.

None of this means you should not use AI. It means "a human reviewed it" is only worth something if the review was a real one.

Why it happens to careful people

• The answer arrives first. Once you have read a plausible version, your own thinking starts from it. You check it for errors instead of asking what you would have written.
• Fluent text feels checked. A neat paragraph is easy to believe. Nothing about its style tells you which sentence is unsupported.
• Omissions are invisible. You can spot a wrong statement if you know the answer. You rarely notice a missing question, a missing exception or a missing owner.
• Time pressure. A quick review of a quick answer feels proportionate. It is how an unsourced line ends up in a document that a programme then builds on.
• Shared responsibility. When a team knows that "someone will review it", each person may relax a little.

The blind-first review

The fix is to change the order. Form your own view before you see the machine's.

Before you read the output, spend five to ten minutes writing down:

1. The questions the output must answer. For a requirements draft: what, for whom, under what condition, who decides, how a tester passes or fails it.
2. The facts you expect it to contain, each with the source document in your register that supports it.
3. Anything you expect to be missing from the sources, such as an owner not yet named or a rule no one has confirmed.
4. One thing you would be surprised to see.

Then read the output and compare. Mark each statement as one of:

• Matches your list and a source.
• New and supported: you did not expect it, but you find it in a source.
• New and unsupported: nothing in your register backs it. Remove it or turn it into a question for the owner.
• Contradicts a source. Go back to the source and decide which is right; record who confirmed it.

Finally, look for what is missing: items on your list the output never mentions. Those gaps are the ones a skim never finds.

A short example

You have two documents about Northgate Bank's new-payee warning: the fraud review minutes (SRC-01) and an email from the Head of Fraud (SRC-02). Before asking a tool to draft requirements, you write down that the sources support a warning screen on a customer's first payment to a new payee, and a hold of up to 30 minutes on amounts above £10,000. They say nothing about who can release a held payment, and the notification wording is still open.

The tool's draft includes a requirement that "a held payment is released automatically when the 30 minutes end, or earlier by a branch manager". You compare it with your list. The 30-minute hold matches SRC-02. The automatic release and the branch manager are in neither source: they are new and unsupported. Without the list you might have read "reasonable" and moved on. With it, that line becomes a question for the Head of Fraud, and the requirement for branch-manager release stays "proposed", not "approved".

A checklist before AI-drafted text goes anywhere

Use this before you share, paste into a document of record, or sign off.

• [ ] I wrote my own list of expected content and sources before reading the output.
• [ ] Every statement in the output is marked matches, new and supported, new and unsupported, or contradicts.
• [ ] Each unsupported statement has been removed or turned into a question with a named owner.
• [ ] I looked for what is missing, not only for what is wrong.
• [ ] Any number, date, threshold, regulation or quotation was checked against the original document, not against the tool.
• [ ] A decision in the output is recorded only if a person with authority made it, and I have noted who and when.
• [ ] I kept a short log of what I asked, what I checked and what I changed.
• [ ] If someone else will review my work, I told them what came from the tool, so they are not reviewing blind.
• [ ] I could explain every line to the sponsor without saying "the tool said so".

Make the review harder to skip

• Put the sources first. Ask for citations to your register IDs, then check the IDs yourself. A tool can cite a document that does not say what it claims.
• Review in a different place from where you read. Mark up a printed copy or a separate file, not the tool's window.
• Leave it overnight when you can. A review done late in the day, straight after the draft, is the one most likely to be a skim.
• Rotate who goes first. If two people review, one should do the blind-first step and the other should read the output cold, then compare.
• Be wary of speed as proof. A review that always finds nothing is either a very good tool or a reviewer who has stopped looking. Occasionally seed a deliberate error in a practice document and see whether you catch it.

What to do on Monday

Pick the next AI-assisted piece of work you will review. Before you open the output, write your list of expected content. Then run the comparison. If nothing at all differs, ask whether your list was too close to the tool's wording.

If you work in financial services, the free guide What not to paste into an AI tool on a regulated change covers the policy and record-keeping around this, and the FCA reference below is worth reading in full.

Sources and further reading

These are reading routes, not statements of what applies to your firm.

• Financial Conduct Authority, AI in financial services: good practice and areas for improvement (published October 2026): www.fca.org.uk/publications/good-poor-practice/ai-financial-services
• NIST, AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024), which describes automation bias under "human-AI configuration": nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
• NIST, AI Risk Management Framework: www.nist.gov/itl/ai-risk-management-framework