Corporate governance was built to manage human decisions, deterministic software, and quantifiable financial risk. It is now being applied, largely unchanged, to AI systems that behave probabilistically, drift silently over time, and fail in ways the old control frameworks never anticipated. That is not a translation problem. It is a structural gap, and most organizations discover it only after falling through it.
The problem is rarely a lack of ambition. Most large enterprises have something in place: a model risk policy, an AI ethics committee, a new AI paragraph in the annual risk report. What they lack is a reckoning with how many of the assumptions baked into those frameworks are not merely incomplete but mismatched when applied to contemporary AI.
This essay names four of those mismatches. Each blind spot follows from a specific assumption that holds for deterministic systems and human actors and breaks for probabilistic AI. For the wider regulatory picture pushing organizations toward this work, see the companion piece, Navigating the Wave: How Corporate GRC Is (or Isn't) Keeping Pace.
Reported AI incidents are climbing. The AI Incident Database, which catalogs verified real-world failures in commercial and public-sector deployment, has recorded a marked year-over-year rise, and the trend has been consistent rather than a single bad quarter. Exact counts are a moving target, because incidents are added retroactively as they come to light, so the direction of travel is the reliable signal rather than any particular figure.
The count matters less than the composition. The incidents span factual fabrication, discriminatory output, autonomous action with unintended consequences, fraud-detection failure, and outright regulatory violation. Across every category, one structural feature repeats: the organizations involved were not ungoverned. Most had risk-management functions, internal audit, board oversight, and formal control frameworks. What they applied were frameworks built for a different class of system, the control vocabulary of deterministic software and human-run processes, pointed at probabilistic models. They found the mismatch only in the aftermath.
Regulation is catching up unevenly. The EU AI Act remains the most comprehensive legislative response, but its timeline has moved. Key high-risk obligations were originally set to apply from 2 August 2026. Under the Digital Omnibus, adopted as Regulation (EU) 2026/1744 and in force from 27 July 2026, obligations for stand-alone high-risk systems under Annex III now apply from 2 December 2027, and those for AI embedded in regulated products under Annex I from 2 August 2028. Not everything moved: the Article 50 transparency rules took effect on 2 August 2026 as planned. The direction is unchanged even as the dates slip. Legislation creates obligations; it does not create understanding. The organizations that navigate enforcement well will be the ones that diagnosed, in advance, why their existing frameworks fall short.
In 2025, a Deloitte team delivered a report to the Australian Department of Employment and Workplace Relations that contained references to academic work that did not exist, and a fabricated quotation from a Federal Court judgment. The fabrications were scattered through a deliverable tied to a contract worth roughly AUD 439,000. Australian academics identified the errors after publication, Deloitte issued a corrected version, and the firm refunded part of the fee, reported at around AUD 97,000. The matter drew sustained coverage in the Australian press.
Traditional quality control assumes failure comes from insufficient human diligence, so the fix is more supervision, more review, tighter sign-off. That assumption fits human error. It does not fit AI hallucination, which is an architecturally predictable property of inference-based systems, not a lapse of care.
The response to human error is review. The response to hallucination is architectural: grounding and retrieval, output verification pipelines, confidence thresholds, and domain-specific validation before anything ships. Bolt a human review layer onto AI output and you get a reviewer who cannot catch a fabricated citation buried in a forty-page document without independently checking every reference, which nobody does. Worse, the misclassification hides the redesign that would actually work: treating AI output as its own artifact class that needs AI-specific verification, not enhanced review of the same page.
Adopt failure taxonomies that separate model failure from process failure from human-operator failure, and attach a distinct control to each. This also reaches your contracts. Organizations that classify AI output errors as professional-services errors try to hold vendors to negligence standards, while most enterprise AI agreements explicitly disclaim liability for output accuracy and warrant only availability and security. That gap belongs in the risk register, not in the post-incident report. For the vocabulary problem underneath all of this, see the companion essay, The 2026 GRC-AI Lexicon.
A European bank's AI fraud-detection model is validated at deployment: precision and recall sit within thresholds, predictions are tested against three years of history, and the model risk team signs off on a standard protocol. Eighteen months into production, performance has quietly degraded. New merchant categories, changed customer behaviour after a product launch, and evolving fraud tactics have shifted the live data, and the model has got worse without tripping a single alarm.
A note on this example. This is a representative composite drawn from a pattern that recurs across financial-services model risk practice, not an account of one identified institution. It is included because the mechanism is well documented and instructive, and it is labelled so that nobody mistakes it for a specific case.
Governance assumes a validated control stays valid. That holds for a segregation-of-duties policy or an access-control matrix, which do not change behaviour on their own. AI models do change. It is not a defect; it is a property. Deployed models live in environments that move, and the data they meet in production drifts away from the data they trained on.
Two mechanisms do the damage. Concept drift is when the relationship between inputs and the correct answer changes, as when fraud tactics evolve and old signals stop predicting. Data drift is when the input distribution shifts even though the relationship holds, as when a business expansion changes the geographic mix. A framework that only checks outputs at deployment sign-off sees neither until harm shows up. And a point-in-time model review that certifies compliance without continuous monitoring starts decaying the moment it is signed.
You cannot make a large model's internals legible at production scale; interpretability research is not there yet. So compensate at the control level, and give the control real teeth:
Treat every validation as time-stamped: it certifies behaviour as of a date under set conditions, and it expires. Audit committees should read model attestations the way they read a fire inspection, not a deed.
Australia's Robodebt scheme ran from 2016 to 2019 and issued roughly AUD 1.8 billion in debt notices to welfare recipients using automated income averaging. A 2023 royal commission found it unlawful, documented harm to hundreds of thousands of people, and led to criminal referrals; compensation and settlements ran to hundreds of millions. Robodebt was not a large language model. It was an automated calculation. That is exactly why it belongs here.
Risk appetite frameworks define acceptable residual risk as likelihood times impact, usually on a matrix. What they almost never carry is a scale modifier. An error rate judged acceptable when trained officers review individual cases becomes something else entirely when the same rate runs across millions of cases with no human in the loop.
The language of the risk statement stays identical while the consequence moves by orders of magnitude. Approve a fraud model with a documented 3 percent false-positive rate and, at ten thousand reviewed transactions a year, that is 300 errors, likely inside tolerance. Point the same model at fifty million annual transactions and it is 1.5 million wrongly flagged accounts. The statement reads the same. The harm is four orders of magnitude apart. Automated systems also tend to lack the exception handling that human review provides at the margin, so the harm can be worse than linear.
Rebuild risk appetite statements for automated decision systems to include volume-adjusted impact as standard, not per-instance impact alone. This is not only a public-sector concern. Consumer-facing models that shape credit, insurance, employment, or tenancy decisions operate at scales where bias or error becomes an aggregate civil-rights problem, not a product defect, and both the CFPB and FTC have signalled they read model behaviour at scale through a population-harm lens. The EU AI Act's tiered classification points the same way, placing scaled, consequential decisions in higher-risk tiers, but the Act creates the obligation without building your internal mechanism for you. For the enforcement angle on scaled errors, see The Fight for AI Credit Justice.
On 16 January 2026, California Attorney General Rob Bonta sent a cease-and-desist letter to xAI over Grok's image tools, demanding the company halt the creation and distribution of nonconsensual intimate imagery and child sexual abuse material. The action followed an investigation opened days earlier into sexually explicit material produced with the model, including reports of images of children. The letter required written confirmation by 20 January 2026 that the conduct had stopped. (Source: California Office of the Attorney General.)
The board that approved deploying Grok's image generation did not approve, and could not have approved, the specific outputs that drew enforcement. The behaviour was emergent: not programmed, not tested for, arising from the model, its training data, and the inputs users supplied in production. Board oversight assumes the consequential decisions happen on a cadence the board can see. AI breaks that assumption.
Boards run on a clock built for financial and strategic change: quarterly earnings, annual strategy, periodic risk and audit review. That cadence was adequate for the risks boards were historically asked to hold. It is not adequate for a system that can shift its behavioural profile, in ways that create legal exposure, between two meetings. Adding a technically credentialled director does not fix this. A sophisticated board member still cannot watch an output distribution move at two in the morning on a Tuesday. The fix is structural, not biographical: the board needs different information, not just differently trained people.
This is also where an accountability gap opens. When a model produces output that triggers litigation or a regulator, the question of who authorized the behaviour is hard to answer through normal governance. No resolution approved it. No policy forbade it, because nobody anticipated it. The new AI governance roles emerging in 2026, model risk officers and responsible-AI leads, do create named accountability, but a title is not a system. The named officer needs continuous monitoring, defined escalation triggers, and a real reporting line to make the accountability mean anything.
None of this means the existing machinery is worthless, and an honest argument should say so. Several traditional controls carry over and remain load-bearing under AI. Audit trails still establish who did what and when. Risk registers are still the right home for the vendor-liability gap and the scale modifier once you add them. Vendor due diligence still matters, arguably more, given how much AI capability is bought rather than built. Segregation of duties still limits who can push a model to production or override its output. The argument here is not to discard these. It is to stop assuming they are sufficient on their own for a class of system they were never designed to govern.
The four blind spots share one root: a governance assumption that is valid for deterministic, human-operated systems, applied without modification to probabilistic, autonomously-behaving AI. The fix in each case is not to patch the framework but to replace the underlying assumption.
For misclassification: adopt AI-specific failure taxonomies and attach distinct controls to model, process, and operator failures.
For opacity and drift: make continuous monitoring a first-class obligation, and treat every validation as a time-stamped certification with a defined revalidation cadence.
For the risk appetite paradox: add a scale component to risk appetite statements for all automated decision systems, so a risk that is acceptable at review scale is re-examined at automated scale.
For board oversight: build a standing operational oversight layer, continuous monitoring, anomaly detection, defined escalation, that sits between the model and the board and bridges the gap between how fast AI behaves and how often boards meet.
These are not modest adjustments. They ask governance functions to rethink logic that has held for decades, and the external pressure to do it, EU enforcement, state-level action in California and beyond, guidance from the FTC and CFPB, is only building. Treat it as a compliance checklist and you stay one incident behind. Treat it as real modernization and it becomes an advantage.
This piece is part of a series on governing AI inside corporate accountability frameworks. See also The 2026 GRC-AI Lexicon on the vocabulary the work requires, and The Intelligent Plagiarism on the intellectual-property questions AI systems raise. For roles at the frontier of this discipline, explore the AI governance leadership landscape.
A risk or failure mode that legacy GRC frameworks cannot detect, because they were built for deterministic systems and human actors. Four are structural: misclassifying AI failures as human error, missing model drift in ongoing control validation, applying static risk thresholds to scaled automation, and relying on board cycles too slow for emergent behaviour.
In 2025 a Deloitte team delivered a report to the Australian Department of Employment and Workplace Relations that contained references to academic work that did not exist and a fabricated quotation from a Federal Court judgment. The contract was worth about AUD 439,000. Australian academics identified the errors after publication, and Deloitte issued a corrected report and refunded part of the fee, reported at around AUD 97,000. The lesson is that quality controls designed for human error do not catch AI fabrication, which needs architectural verification instead.
Traditional quality control assumes failure comes from insufficient human diligence, so the remedy is more supervision and tighter sign-off. That fits human error. It does not fit hallucination, which is an architecturally predictable property of inference-based systems rather than a lapse of care. A reviewer cannot catch a fabricated citation buried in a forty-page document without independently checking every reference, which nobody does. The response has to be architectural: grounding and retrieval, output verification pipelines, and domain-specific validation before anything ships.
Model drift is the degradation of an AI model's performance as production data diverges from the data it was trained on. Concept drift is when the relationship between inputs and the correct answer changes. Data drift is when the input distribution shifts even though the relationship holds. Traditional controls do not drift once established: an access-control matrix behaves the same next year. Models do, so a point-in-time validation certifies behaviour that starts decaying the moment it is signed.
It names an owner, a trigger, a metric and an escalation path. For example: owner is the model risk lead for the fraud platform; trigger is a population stability index above 0.2 on any top-ten input feature, or a rolling 30-day precision drop of more than three points below the validation baseline; metric is weekly out-of-sample precision and recall plus feature-drift scores; escalation is a revalidation ticket within 48 hours reported to the risk committee, with two consecutive breaches pausing automated actioning pending human review.
Risk appetite frameworks define acceptable residual risk as likelihood times impact, but they rarely carry a scale modifier. An error rate that is acceptable when trained officers review individual cases becomes something else entirely when the same rate runs across millions of cases with no human in the loop. A model with a 3 percent false-positive rate produces 300 errors across ten thousand reviewed transactions and 1.5 million across fifty million. The risk statement reads identically while the harm moves by four orders of magnitude.
Australia's Robodebt scheme ran from 2016 to 2019 and issued roughly AUD 1.8 billion in debt notices to welfare recipients using automated income averaging. A 2023 royal commission found it unlawful and documented harm to hundreds of thousands of people. Robodebt was not a large language model, it was an automated calculation, which is exactly why it belongs in an AI governance discussion: the failure was scale without a scale-adjusted risk appetite, not model sophistication.
Boards run on a cadence built for financial and strategic change: quarterly earnings, annual strategy, periodic risk and audit review. That was adequate for the risks boards were historically asked to hold. It is not adequate for a system that can shift its behavioural profile between two meetings. Adding a technically credentialled director does not fix it, because a sophisticated board member still cannot watch an output distribution move at two in the morning. The fix is structural rather than biographical: the board needs different information, not differently trained people.
The AI Act remains the most comprehensive AI legislation, but its high-risk deadlines have moved. Under the Digital Omnibus, adopted as Regulation (EU) 2026/1744 and in force from 27 July 2026, obligations for stand-alone high-risk systems under Annex III shift from 2 August 2026 to 2 December 2027, and obligations for AI embedded in regulated products under Annex I shift to 2 August 2028. Not everything was deferred: the Article 50 transparency rules took effect on 2 August 2026. Confirm the current position for your specific obligations before relying on any date.
Four changes, one per blind spot. Adopt AI-specific failure taxonomies that attach distinct controls to model, process and operator failure. Make continuous monitoring a first-class obligation and treat every validation as a time-stamped certification with a defined revalidation cadence. Add a scale component to risk appetite statements for all automated decision systems. Build a standing operational oversight layer that sits between the model and the board and bridges the gap between how fast AI behaves and how often boards meet.
Follow GRC Careers on LinkedIn
New essays, the frameworks that actually move, and the governance roles worth knowing about. Written by practitioners, published as it happens.
Follow on LinkedIn →