Can We Trust AI With Safety-Critical Checks?

Can We Trust AI With Safety-Critical Checks?

A series on Agents and Biases, AI Transformation, Future of Work, AI Safety

Part 1: The safety net we already have

Every construction site and factory floor runs on a quiet assumption: people will make mistakes. A worker skips a step they have done a hundred times. A supervisor signs off a permit at the end of a long shift. A reviewer misses one line in a forty-page method statement.

Safety management is largely built around this. Permits, inspections, checklists, toolbox talks and second signatures exist because human attention is limited and human judgement bends under pressure. These layers do not remove error. They are designed to catch it before it becomes an incident.

AI is now being offered as a new layer in that net. Cameras that never look away. Systems that read every page of every document. Tools that flag what a tired eye might miss.

Imagine assigning an AI agent the job of a compliance checker. Its task is simple to describe: read the submission, read the code, decide whether the two match. A human checker in that role is trained, tested and held accountable. We roughly know how they fail. They get tired, rushed or distracted.

The AI checker raises different questions. If it is given the same document twice, will it reach the same verdict both times? When it says "compliant", has it actually applied the clause, or has it recognised a pattern that usually passes? If the code is amended next month, will its answers change with it? And when the evidence is incomplete, will it say so, or will it fill the gap with a confident guess?

A human checker who fails tends to fail visibly, with a missed signature or an inconsistent note. An AI checker may fail quietly, producing a clean, well-formatted answer that looks exactly like a correct one.

Can AI reduce errors in safety-critical work, or will it replace them with new ones we are less equipped to catch?

Part 2: What the evidence shows

Alongside inspections on site, safety and compliance also run on documents: method statements, risk assessments, permit applications, design submissions and certificates, each checked against a code or standard. This is where AI checkers are increasingly being tested. The research so far points to three things worth understanding, each with a strength and a limit.

1. AI reads text well, but not always what matters

Leading models are now strong readers of text, even on technical drawings. A 2026 benchmark found top systems answered text-based questions about architectural drawings with up to 95% accuracy.¹ But the same models struggled with symbols. Counting doors and windows on floor plans was correct only 16% to 39% of the time, because those elements are shown as graphic conventions that vary across CAD standards, not as words.

A 2026 study of real LEED submissions found something similar. When drawings were added to the documents, accuracy fell by about 16 points, because the models could see lines and layout but could not read the small text, and guessed instead.² Yet the same study found that when numeric thresholds were handed to a simple rule-based checker, accuracy on one credit rose from 50% to 100%. AI paired with plain code did better than AI alone.

2. AI often follows the rule, but a confident answer is not always a grounded one

When researchers removed or reversed the governing rule in compliance cases, general models were far more responsive on the cases where the rule decided the outcome, and their overall accuracy was 90% to 92%.³ But across all cases, verdicts often stayed the same when the rule changed. A right answer is not always proof that the rule was read.

Consistency and uncertainty are open problems too. An AI agent that succeeded on about 61% of tasks on one attempt succeeded consistently on fewer than 25% across eight attempts.⁴ And none of 20 frontier models reliably said "I don't know" when the evidence was missing, although careful instructions improved this.⁵

3. AI can make reviewers better, or quietly worse

The strongest positive evidence comes from medicine. In a randomised trial of more than 105,000 women in Sweden, radiologists supported by AI detected 29% more breast cancers, with no rise in false positives and 44% less reading workload.⁶

But the same partnership can fail in the other direction. In a 2023 study, 27 radiologists read mammograms with AI suggestions, some deliberately wrong. When the AI was right, readers were around 80% accurate. When it was wrong, accuracy fell to 20% for inexperienced readers and 45.5% for those with more than 15 years' experience.⁷

A review of 74 studies found that decision support generally improves performance, but people over-rely on it, and workload, time pressure and complexity make that worse. It also found what helps: accountability, training, showing confidence levels, and giving information rather than recommendations.⁸ The EU AI Act now names "automation bias" explicitly in its human oversight requirements.⁹

The difference between these outcomes may lie less in the AI than in how the work around it is designed.

Part 3: Designing the work around the checker

AI in safety-critical checking is neither reliably good nor reliably bad. Its performance shifts with the task, the input and the people around it. That makes the real transformation question less about which tool to buy, and more about how the work should be redesigned once AI is part of it.

Hypothesis 1: Give each step the right checker

The evidence suggests a compliance check is made of several steps, and AI handles some far better than others. Reading text, extracting values and finding the relevant clause looks like a reasonable fit. Comparing a number against a threshold may be better left to simple rule-based code, which does not guess. Interpreting symbols on drawings, and judging whether something is adequate, may still need a person.

The question shifts from "can AI do this check?" to "which part of this check should AI do?"

Hypothesis 2: Test for grounding, not only accuracy

A checker can match past decisions and still fail in the ways described above. So before trusting it, it may be worth testing four behaviours. Does it catch a violation planted in a submission? Does its verdict change when the rule changes? Does it give the same answer when run again? And does it say "insufficient information" when key evidence is missing?

Hypothesis 3: Who checks the checker?

Human errors on a checking desk tend to be scattered: one tired reviewer, one missed line. If an AI misreads a symbol or ignores a clause, it may repeat that error on every project that uses it. That makes the human reviewer more important, not less.

Yet a reviewer under deadline can become a rubber stamp. The research points to what helps: accountability, training, showing confidence levels, and presenting information rather than recommendations. In practice, that could mean an AI that shows the clause and evidence it relied on, instead of simply returning "compliant". It could mean reviewers measured on what they catch, not how many checks they clear. And if AI reduces checking workload, as it did in the Swedish screening trial, part of that saved time could be reinvested in review quality rather than taken entirely as extra throughput.

How an organisation might approach this

If these hypotheses hold, the change is not checkers being replaced by AI. It is a change in what checking work is. One way to approach it:

1. Start with one check. Choose a high-volume, well-documented check, such as method statement reviews or material certificate checks. Break it into steps and label each one: reading, numeric, or judgement.
2. Assign each step. AI reads and cites its sources. Code handles thresholds. People handle symbols, judgement and final sign-off.
3. Test before trusting. Run the AI on past cases with known failures, planted errors, changed rules and missing evidence. Compare reviewers with and without it.
4. Redesign the reviewer role. Shift reviewers from doing every check to auditing the AI's work. Measure how often they catch errors, and store rules as versioned data so every verdict records which version it used.
5. Protect the skill. Reinvest part of the time saved into review, and keep some checks fully manual so the expertise needed to catch AI errors does not fade.

The questions I am sitting with

• If an AI checker is right most of the time, how do we stay alert for the times it is not?
• Will the extra throughput we gain from AI outweigh the added scrutiny it requires during transformation and execution, and deliver a better ROI?
• How do we keep human expertise sharp when the job increasingly becomes reviewing what a machine has already done?

I write about AI, behavioural economics, and user research. Mostly questions, occasionally answers. If you are thinking about any of this, I would like to compare notes.

References

1. Kondratenko, A., Birhane, M., Hsain, H. E., and Maciocci, G. (2026). AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understanding. arXiv:2601.04819
2. De, A., and Felkner, J. (2026). Neuro-Symbolic AI for LEED Compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts. arXiv:2607.15647
3. Sadhu, S., Sengupta, A., Sankarapu, V. K., and Seth, P. (2026). Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems. arXiv:2610.12313
4. Yao, S., Shinn, N., Razavi, P., and Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045
5. Kirichenko, P., Ibrahim, M., Chaudhuri, K., and Bell, S. J. (2025). AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions. arXiv:2506.09038
6. Lång, K., et al. (2025). MASAI randomised trial of AI-supported mammography screening. The Lancet Digital Health. Summary
7. Dratsch, T., et al. (2023). Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance. Radiology. RSNA
8. Goddard, K., Roudsari, A., and Wyatt, J. C. (2012). Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121 to 127. JAMIA
9. Regulation (EU) 2024/1689 (AI Act), Article 14: Human oversight. AI Act Service Desk

LinkedIn version

Can we trust AI with safety-critical checks?

Safety management is built on one assumption: people make mistakes. Permits, inspections and second signatures exist to catch those mistakes before they become incidents.

A few of my clients have already started asking how AI could help with compliance-related retrieval, inference and task completion. There are also already a few AI systems on the market offering safety and legal compliance solutions. So I looked at what the research says about how AI behaves when it checks documents against a code or standard.

These are observations I would strictly consider when designing AI transformations or integrations for safety-critical workflows, because the real question is less about which tool to buy and more about how the work is redesigned around it.

1. AI reads text well, but not always what matters. Leading models answered text questions on architectural drawings with up to 95% accuracy, but counted doors and windows correctly only 16% to 39% of the time. Paired with simple rule-based code for numeric checks, accuracy on one LEED credit rose from 50% to 100%.
2. A confident answer is not always a grounded one. When researchers removed or reversed the rule in compliance cases, AI verdicts often stayed the same. And none of 20 frontier models reliably said "I don't know" when evidence was missing.
3. AI can make reviewers better, or quietly worse. In a Swedish trial of more than 105,000 women, radiologists with AI support found 29% more cancers with 44% less workload. But in another study, when the AI was wrong, even radiologists with over 15 years' experience dropped to 45.5% accuracy.

These observations mean giving each step the right checker, testing for grounding and not only accuracy, and designing the reviewer role so it doesn't become a rubber stamp.

From a people perspective, for those working in AEC or manufacturing, the transformation may not be AI replacing checkers. It may be checkers moving from doing every check to designing, testing and auditing how checks are done.

This also raises more questions for me. The one I find most interesting: will the extra throughput we gain from AI outweigh the added scrutiny it requires during transformation and execution, and deliver a better ROI?

If you work in safety or compliance in construction or manufacturing, I'd like to hear how you are approaching this.

#AgentsAndBiases #AITransformation #ConstructionSafety #AICompliance #Innodrive#Futureofwork

First comment: The full article, with references and a five-step approach to redesigning checks around AI: [BLOG LINK]

Slides (4 x 1080 square, Cover + Parts 1 to 3): Issue7_cover.png, Issue7_part1.png, Issue7_part2.png, Issue7_part3.png. Created and delivered in the Claude session; to be added here manually (direct upload to Notion was not available from that session).

Research notes (internal)

Input (internal)