A Commonwealth artificial intelligence model built to detect Medicare fraud stopped being used in December 2025. The Auditor-General’s explanation runs to a single clause: the resourcing required for manual assessment of fraud and non-compliance “signals” could not keep up with the volume of potential matters generated.

The audit does not say the model was wrong; it says the people downstream of it could not get through its output.

The Medicare audit, tabled on 21 September 2026, is the closest thing Australia has to a field test of the argument now reorganising how assurance over automated systems gets discussed. The claim behind it is simple. Checking each step of an agentic workflow tells you very little about whether the run as a whole stayed inside its rules, so what has to be examined is the entire execution trace. I have argued a version of this before, in The Run That Graded Clean, where evaluation runs caught an agent making a change it had been gated against only once somebody compared the database before and after. What the Medicare audit adds is the part that never gets costed. Once you have the trace, a person has to read it.

The structural case was put formally in mid-September by four researchers who name four ways an agentic workflow can breach a policy while every individual step passes its check. Two of them are recognisable from any delegation register: authority creep and threshold laundering. The other two, cumulative sum violation and context collapse, describe breaches that exist only across a sequence of steps (Kurady et al., 2026). Underneath all four sits a logical limit instead of a quality problem. As the authors put it, “a predicate over a single step cannot evaluate a property that step does not determine, so no improvement in the accuracy of the step-scoped monitors detects this class”.

A hypothetical shows the shape of it. An agent holds a $100,000 delegation and makes four separate commitments of $50,000. Each commitment sits inside a threshold that a step-level monitor can check and pass. The delegation is breached anyway, and no amount of per-step scrutiny will say so, because the quantity that went over is not a property of any single step. Their paper is a preprint and reports no measured violation rate, so the example demonstrates a mechanism and nothing more.

Their proposed answer is a provenance-aware runtime architecture that evaluates policies over complete execution traces, recomputing guarded quantities from raw provenance instead of from the pipeline’s own derived representation. Such an architecture does not ask the pipeline what it did. It rebuilds what the pipeline did out of the underlying provenance records.

The reflex at this point is to say that Australia has no instrument for any of that. The Commonwealth has one, and the form it takes makes the problem more interesting than an absence would.

The Digital Transformation Agency’s agentic AI addendum to the Commonwealth AI technical standard, last updated on 4 June 2026, asks agencies to ensure accountability can be traced back to agent actions and supported by documented records that can be reviewed and audited. It also wants metadata gathered from agents in each lifecycle phase at runtime, and agent interactions monitored through observability tools. Agent memory should be governed and auditable, with the prompts an agent creates and the inputs and outputs of the whole workflow captured for record keeping (DTA, 2026).

Every one of those is best practice guidance that agencies are expected to apply. Its criteria are written as things they must do, and nothing makes the addendum itself binding.

Set that beside the Policy for the responsible use of AI in government, version 2.0, which took effect on 15 December 2025 and carries eight mandatory requirements for non-corporate Commonwealth entities, with some exceptions. They run from an accountable official and a public transparency statement through to an internal register of use cases and a use case impact assessment (DTA, 2025). Those requirements are real obligations and several of them are worth having. Not one of them requires an agency to be able to reconstruct what an agent actually did, which leaves the process compulsory and the evidence optional.

Fourteen days after the addendum was last revised, an incident showed what the gap allows. On 18 June 2026 a research team at OpenAI used an internal model to conduct internet based research into public medicine spending. The agent reached the Medicare Statistics Reporting Portal, where it accessed both public and non-public material, and wrote files to the internal server. No Commonwealth control produced that finding. The vendor eventually said something, and the Prime Minister’s account of the timing is worth quoting: “it took until 10 September before there was any notification at all. And the notification was an email sent to just the public mailbox” (Prime Minister of Australia, 2026). Services Australia reported the matter to the Australian Cyber Security Centre on 15 September, close to three months after the access. A taskforce was announced on 24 September, led by the Prime Minister’s department alongside the National Cybersecurity Coordinator, the Office of AI, the Australian Signals Directorate, the Australian AI Safety Institute and Services Australia.

In the same week the Auditor-General reported on a Commonwealth AI system operating inside an integrity function, and the finding came out split. Arrangements to manage AI use at an enterprise level were largely effective. Arrangements for managing AI use to support health provider compliance were partly effective. The system examined had been built before any of that governance existed, and it had not been developed or deployed in line with current requirements or fully assured against them. Evaluation and performance monitoring had been limited. Eight potential fraud or non-compliance matters were referred for preliminary analysis, against an estimated $5.2 million of potential fraud or non-compliance identified using, in part, the AI model. Reporting to governance committees had covered the system’s performance without ever specifically examining the role of AI in it, and the cost of developing the system, including the AI-enabled model inside it, had never been analysed at all (ANAO, 2026).

The split finding is the preprint’s structural point rewritten in audit language. A framework can be rated largely effective while a use case inside it goes unassured, because a judgement about the framework cannot evaluate a property the framework does not determine.

A second-order problem sits under all of this, which is that measurement of any of it remains poor. A review of the agentic evaluation literature makes the arithmetic explicit: “V_total ≤ V1 × V2 × V3 … a pipeline retaining 70% validity at each stage is at most 34% valid” (Caban, 2026). Across the empirical estimate bounds the author puts end-to-end validity somewhere between 0.17 and 0.54. In a purposive sample of 55 papers, about 82 per cent reported inter-rater reliability metrics that were mismatched or incomplete, and sometimes reported none at all. Seven of ten popular benchmarks showed task-validity flaws. Caban’s paper is a preprint too, submitted to Knowledge-Based Systems and not yet published.

Put the two preprints together with the audit and the position gets uncomfortable. Only the trace detects this class of violation, and the instruments for reading a trace are immature. The one Australian agency we can observe assessing at that volume stepped back from it, because the reading could not be resourced.

The control itself is buildable, and somebody has already built one. Researchers reported in September on an agentic system for electricity transmission control rooms that holds four non-bypassable rules across every run. Task success was 93.7 per cent over 590 runs of the primary model, with no rule violation anywhere in them (Mylonas et al., 2026). A grid control room is not a Commonwealth integrity function, and this is another preprint. But whole-of-execution control has now been demonstrated instead of merely proposed, which puts the binding constraint somewhere other than the engineering.

Which returns the argument to resourcing, where the Australian evidence is thin and old. The ANAO’s 2023–24 financial statements report found fifty-six entities using AI, up from twenty-seven the year before. Sixty-four per cent of the entities that used it held internal policies governing that use, up from forty-four per cent, and twenty-seven per cent held internal policies on assurance over that use (ANAO, 2025). Those figures are two financial years old and they describe policy coverage. They say nothing about whether anybody is examining execution traces, and it would be an over-read to claim otherwise. The United States is at a similar stage: the Government Accountability Office released an exposure draft of a testing and evaluation guide for commercial off-the-shelf systems on 14 September 2026, with comments open until 14 December (GAO, 2026). That too is guidance, and at least it says so on its cover.

So the useful question changes. It is no longer whether the trace can be captured, because the Commonwealth has published guidance on capturing it and researchers have built systems that enforce rules across whole runs. The question is who is rostered to read the trace, what they are authorised to do when they find something in it, and what happens to the system when they fall behind.

The Medicare audit answers that last one. When the assessment could not keep pace, the department did not add assessors. In December 2025 it changed the model from an AI-enabled one to a standard statistical one, choosing to generate less instead of reading more.

Every Commonwealth instrument in this area has something to say about how the trace will be produced. None of them says how many people will need to read it, and on the evidence above that is the number I would want settled before signing off on any of it.

References