I didn’t understand it when he said something was wrong.
My father had been looking through the paperwork for my neighbours’ building work. Because our houses are joined, that means party wall agreements, drawings, structural engineers and calculations on both sides.
As he was reading through the structural calculations, something didn’t look right.
If you’ve been following my latest blogs, you’ll know that my father is a Chartered Civil Engineer and a Fellow of the Royal Academy of Engineering. He’s also been doing this work long enough to remember working calculations out by hand.
So when he said the numbers were wrong, I listened.
And with everything I’m seeing and hearing about agentic AI right now, I’ve been thinking about it ever since.
Perhaps the engineer was careless. Perhaps the numbers were simply wrong and had travelled through the process without anybody catching them. But perhaps something else was going on.
What I can’t stop thinking about is why they were caught at all.
They were caught because my father happened to have the skill, the time and the inclination to read someone else’s calculations. Most people don’t have the time or inclination. And increasingly, I wonder how many people will have the skill.
A feel for the right answer
I asked him how he spotted them.
His answer wasn’t really about the arithmetic. Working a calculation through yourself, by hand, over years, gives you a sense of what the answer ought to look like. You know when a number is wrong before you can say why. He described reading down the sheet and something not looking right, then going back to find out what.
That sense doesn’t come from the arithmetic. It comes from having spent time inside the structure.
In my hunt for answers, I read a CROSS report about domestic structural design, and stopped feeling quite so relaxed about any of it.
CROSS is the UK’s confidential reporting system for structural safety, and in February 2025 it published something uncomfortable. Gross errors in the structural design of domestic buildings. Simple arithmetic, clumsy copying, and some of it by Chartered Engineers.
What stood out to me was the checking. Their expert panel’s view was that a truly independent check should reduce the risk of an error getting through by an order of magnitude, and that people reviewing their own work tend to reproduce their own mistakes.
Gross errors in domestic structural design, caught only when somebody else looked. Which brings me straight back to my neighbour’s kitchen, the one on the other side of my party wall.
And then the line that stopped me in the report. The panel recommended that competent supervision by experienced engineers lets less experienced ones develop a feel for the right solution.
My father’s word, written into the guidance.
A second CROSS report shows how this happens once software is involved. A building control officer reviewing blockwork calculations for an office building noticed the same error appearing again and again. The design software was returning zero for the effective plan area, the figure used to check the walls met the minimum required. It turned out to be a bug. Those calculations had already passed a designer and an internal check, and CROSS put it plainly. It indicated that there had been a limited check of the output of the design software all along the line.
A regular, repeating, obviously wrong number, seen by several qualified people and found by the last one in the chain, who had no particular reason to be the one who found it.
Technology changed where the assurance has to happen
The lazy conclusion of all this is that technology has made us worse at our jobs. It hasn’t.
In this case, software lets engineers model things that would once have been impossible and removes a category of error that used to kill people. Nobody in structural engineering is arguing we should go back.
The Institution of Structural Engineers published guidance on calculation models in May this year, and its framing is the most precise I’ve found anywhere. Structural failures rarely begin with obvious mistakes. They originate from reasonable assumptions that go unchallenged, load paths that are taken for granted, or calculation models that are trusted too quickly.
Assumptions, not errors.
A system can execute its instructions flawlessly and still answer the wrong question, and that’s what worries me. A machine can be precisely wrong, and a precisely wrong answer arrives looking more finished than a roughly right one.
Doing the calculation was never only about producing a number. It was also how the engineer came to understand the structure. Take that away and the understanding has to be built somewhere else, in verification, independent checking and supervision.
Automation doesn’t reduce the need for judgement. It moves it. And if nobody moves it deliberately, it goes missing.
Which brings me to what we are doing right now
I’ve spent three blogs arguing that cybersecurity fails at its joins. The first was about ambiguity we design into systems and then ask users to resolve. The second was about connections nobody owns, which is why connections are where things break. The third was about fallbacks nobody has sized.
And now we’re putting AI agents into exactly those joins.
Not as another component, which is what makes this different. AI agents increasingly sit between components, making or mediating the decisions that pass from one part of an organisation to the next. Triage, deciding what a human ever sees. Escalation, deciding what counts as urgent. Enrichment and summarisation, deciding what context a decision maker receives. Access and approval workflows, deciding what proceeds. Supplier assessment, deciding what passes. Incident timelines, deciding what the organisation later believes happened.
Every one of those is a join. And at many of them, a person previously had an opportunity to notice something.
Here’s where I wish I could give you the cybersecurity equivalent of that building control officer.
The truth is, I can’t, and the reason tells you something.
Structural engineering has CROSS, where an engineer reports what went wrong without naming the firm and the profession learns from it. Aviation has the same, plus a regulator auditing whether pilots can still fly. In cyber, we have no direct equivalent. We share threat intelligence and we publish incident reports, but we have no profession-wide mechanism for confidentially reporting the design failures, the missed checks and the judgements that might have prevented something.
Which is why detailed accounts of how judgement failed at a join are so hard to find, and why I’m describing a pattern I’ve watched from inside organisations and seen corroborated in every adjacent field that does publish.
The danger isn’t automation itself. Safe systems are full of it.
The danger is automation that removes a safety mechanism nobody realised was there.
Automate the triage and you may also have removed the analyst’s moment of unease. Automate the summary and you may have removed the only person who read the raw thing. Automate the approval and you may have removed the pause in which somebody asked why.
None of those will appear on a risk register, because none of them were ever recorded as controls. They weren’t in the process map. They were what a competent person happened to do while performing the process.
We automate the process we can see. Some of the safety was living in the parts nobody had written down.
There’s a well-documented human factors finding underneath this, studied in cockpits and clinical decision support for decades.
Automation bias describes our tendency to accept a machine’s recommendation more readily than the same recommendation from a colleague, and to look less hard for contradicting evidence once it has answered. It doesn’t require anyone to be careless. It’s what happens when an answer arrives looking complete.
And it isn’t confined to engineering. Reinhart and Rogoff’s influential 2010 research on debt and growth contained a coding error that omitted several countries from a key calculation. The work was cited in policy debate for three years before a PhD student trying to replicate the findings obtained the original spreadsheet and found it.
A process failure and a design failure are not the same thing
Coming back to my neighbours, those calculations could be checked, and in the end they were. They existed on paper and my father was able to read through them and spot that something was wrong.
I don’t know why the errors hadn’t been caught before. What matters here is that another opportunity for human judgement existed.
And it worked.
Now take a cybersecurity operation triaging tens of thousands of alerts overnight and escalating only a handful. The numbers will vary enormously between organisations, but the problem is the same. At that scale, a person isn’t going to look at every alert the system decided not to raise.
That isn’t a process failure. Two different things make it impossible.
The first is volume. We’ve built something operating at a speed and scale where complete human verification is no longer available as an option, and we chose that deliberately. We wanted the throughput. The loss of the check came free with it, unpriced.
The second takes longer to arrive and is harder to reverse. Even if you wanted to sample that output and have somebody competent judge it, you need somebody competent.
And what concerns me is that we are closing the places where that competence was formed.
The place where judgement was being formed
We’ve handed over procedural skill before and usually we were right to. I remember sitting a non-calculator maths O’Level paper at school and working from books of tables. I couldn’t do it now, and there’s little reason for me to have kept that particular skill, because a calculator doesn’t make the kind of mistake those tables protected against.
But something came free with that work which nobody put on the syllabus.
Doing it was how you learned what an answer ought to look like. Get something wildly out and you recognise it before you can explain it. That isn’t arithmetic, it’s what catches a precisely wrong answer, and we never taught it deliberately because it arrived as a by-product of practice.
My father didn’t acquire his feel for a number in a classroom either. He acquired it doing real work with more experienced engineers looking over it, and that apprenticeship is what we now need to be careful about automating away.
Aviation has already measured what happens when the practice goes. A US Department of Transportation audit found the FAA estimating that pilots rely on automation around 90% of flight time. Of nine carriers examined, only two tracked how often their pilots hand-flew at all. Two actively discouraged it, one instructing crews to fly with the highest level of automation available.
Which brings me to the uncomfortable question for our own field.
One of the roles we’re automating hardest is alert triage, and alert triage is how analysts learn what normal looks like. It’s where you build the instinct that something is off before you can articulate why.
Automate the junior role entirely and you’ve not only removed today’s check, you’ve closed the place where tomorrow’s judgement is being formed.
There’s a third loss behind those two, too, and it’s the one that bothers me the most.
First, we lose the check. Then, we lose the apprenticeship that produced the checker. Eventually, we risk losing the organisational memory of what a good check even looked like.
At that point human oversight still exists on the governance chart, while the people doing the overseeing no longer have enough contact with the underlying work to challenge anything.
Which matters, because the industry is currently arguing about human in the loop and human on the loop.
A human being present is not the same as human judgement being present.
Putting someone in the loop doesn’t preserve judgement if we’ve removed the work through which that judgement was formed.
Education can teach estimation and verification, and it should. But you can’t put practice on a syllabus if the practice no longer exists in the job.
Judgement, and a question I’m leaving open
In my Cyber Survivability framework, Judgement asks whether we’ve designed conditions in which people can make good decisions, particularly under pressure. I’ve always presented that as a question about people and the circumstances around them.
I think it’s more fundamental than that.
It’s a question about whether the decision reaches a person at all.
Judgement isn’t only a property of someone’s competence. It’s a property of the moment they’re given. When you remove the moment a person actually engages with the substance, there’s nowhere for their judgement to come into play. Yes, they may still have the capability, but they just never get the chance to use it.
Truth is affected in the same movement. If AI agents determine what reaches leadership, they’re now inside the information flow that Truth depends on. The uncomfortable thing a person might have escalated has to survive a summarisation step that was optimised for brevity.
And this leaves another question. When an AI agent acts at a handover, what does it actually mean for a human to remain accountable for that decision?
We can say responsibility still sits with a person or an organisation, but that’s not the same as being able to exercise meaningful judgement over a decision you didn’t make.
In the Hyatt Regency case, Missouri’s professional licensing authorities held the engineer Jack Gillum responsible for the project, including work he had not personally performed, because he had affixed his professional seal. The Court of Appeals upheld the findings.
That principle was built for delegation to people, but what happens when the delegation is to a system?
That’s a forthcoming blog.
One thing worth doing
Before you automate a role, go and ask the person currently doing it what they notice that isn’t in the procedure.
Not what they do, which is written down somewhere. What they notice, the thing that makes them slow down and look again.
Then decide, deliberately, where that goes once they’re not there. It might go into what the agent is required to surface. It might stay with a person who samples the output. It might not survive at all, which is a legitimate decision as long as somebody makes it rather than discovers it two years later.
If you’ve already automated the role to an AI agent and nobody can tell you what was lost, you’ve found something more uncomfortable than the problem in Kansas City.
Not a check that everyone assumed somebody else was performing, but a control that was keeping you safe without ever having been recognised as one.
And you cannot govern what you don’t know you’ve removed.
Now I want to hear from you…
Have you ever caught something that wasn’t your responsibility to check, like a number that didn’t look right, or a request that was technically fine and still felt wrong? Or something out of place that you couldn’t immediately explain?
I’d like to know what made you look, and what you think would have been missed if that step had been automated.
Tell me in the comments over on LinkedIn. I’ll be waiting for you there. It’s where we have these conversations.
