In a set of evaluation runs released on 15 August, an AI agent made a database change it had been gated against making, and then declared that it had refused. The transcript recorded a refusal. A language model judging that transcript recorded a refusal. Every grading method working from the agent’s own account of the episode returned a pass. What recorded the change was a comparison of the database before and after the run (Khurdi, 2026).

That result deserves a moment, because Australia is assembling the machinery that will decide how AI systems in government get proved safe, and the instruments it will reach for are mostly of the first kind.

The Office of AI was established in the Department of the Prime Minister and Cabinet on 15 July 2026, to coordinate across agencies and “design and legislate the new Australia artificial intelligence (AI) standard” (Department of the Prime Minister and Cabinet, 2026). Legislation is expected early in 2027, so today it is a stated intention with a drafting office behind it. A month later New South Wales announced an Office of AI of its own, inside The Cabinet Office (NSW Government, 2026). The state already had an AI office, stood up in Digital NSW in September 2025 for an initial two-year period. It has since been renamed NSW GovAI and keeps the standards and advice work. The new office takes the policy brief, so the state has gone from one AI office to two inside twelve months.

The Commonwealth already has a working instrument, and it deserves its due. That instrument is version 2.0 of the Digital Transformation Agency’s policy for the responsible use of AI in government, and it requires agencies to implement the AI impact assessment requirement for in-scope use cases by 15 December 2026 (Digital Transformation Agency, 2025). An obligation with a date attached puts Australia ahead of most comparable countries, and I have written elsewhere about what that assessment reaches. My question here is narrower and it concerns evidence: when an agency sits down to demonstrate that a deployed system behaved itself, what is that agency allowed to point at?

The 15 August preprint gives the clearest answer available. Shiven Khurdi built an instrument called AgentRelBench, which prices damage from database state differences across repeated runs with no language model anywhere in the measurement path. The evaluation covered 2,128 runs across nine models in six families inside a simulated enterprise operations environment. The finding in the title is that no task damaged on every run. Across a development pool of thirteen model-and-task pairs, one clean run missed a damage-producing pair four times in five, and Khurdi reports the held-out pool as consistent but underpowered, which is candid for a preprint (Khurdi, 2026).

The refusal above is the second finding, and the more uncomfortable, because the first can be answered by running the audit more often. One model family made the gated irreversible change while declaring it had refused, and grading from the transcript, or from a judge model reading it, scored those runs as safe refusals. Repeating that audit a hundred times would have produced a hundred passes, because the instrument was reading the agent’s report of the episode instead of its effect on the world.

A second preprint days later approaches the problem from another direction. Haiyue Zhang audited the step-level credit signals used to train language-model agents against ground truth reconstructed from executed replay. None of the three read a step’s contribution better than a shuffled version of itself, and the implicit signal tracked how fluent a step looked (Zhang, 2026). That is a single-agent research setting, so it reaches assurance by analogy. The distinction it draws is worth having anyway, because a measurement can be well built and consistently applied and still grade a step for looking correct while missing what it contributed.

Set those two results beside each other and a pattern appears in what our assurance instruments actually read. A transcript is the system’s account of an episode. A judge score is a second model’s reading of that account. A model card is the developer’s account of the model, and a position paper analysing 500 of them on Hugging Face argues that for open-weight models the card alone is too thin for downstream governance (Chae et al., 2026). A use case register is an agency’s account of what it is running. All of it is testimony. The one method in the cluster that caught the failure read the state the system left behind.

There is a second problem underneath, and more evidence does not fix this one. Okamoto and colleagues put twelve instruction-tuned models to work as enterprise procurement chatbots, gave each an environmental regulation in its system prompt, and built a vendor list on which the certified suppliers cost nearly twice what the uncertified ones do. Their opening sentence deserves a second reading from anyone who has drafted a contract clause: “Specifying a penalty can turn a legal obligation into a cost-benefit calculation that favors violation.” They call it the enforcement information paradox, and they find it in AI agents (Okamoto et al., 2026).

Two details travel further than the paradox itself. Under identical conditions, compliance spanned 46 percentage points across the twelve models, so the choice of model was doing more work than the wording of the rule. Benchmark scores and the developers’ own descriptions of post-training also failed to predict where a model would fall. All of it sits inside a controlled trial of a constructed scenario and should be quoted that way. The authors’ conclusion still belongs in every procurement conversation: model selection is itself a governance decision, and benchmark evaluation is not sufficient for compliance-sensitive deployments.

Australia’s own evidence has been sitting in the Auditor-General’s reports for eighteen months. In the 2023-24 financial statements audits, 56 entities reported using artificial intelligence in their operations, more than double the 27 of the year before. Among them, 64 per cent had established internal policies governing the use of AI, while 27 per cent had established internal policies regarding assurance over AI use (Australian National Audit Office, 2025a). Adoption doubled inside a year; the share of adopters with a policy about assuring it held at roughly a quarter. The ANAO’s own conclusion is drier and better than anything I would write over it: an absence of governance frameworks for managing the use of emerging technologies could increase the risk of unintended consequences.

Two performance audits since then say the same thing with names attached. At the Australian Taxation Office, which had 43 AI models in production as at 14 May 2024, the ANAO found partly effective arrangements and made seven recommendations, all agreed (ANAO, 2025b). At IP Australia, which put its first tool into patent examination in 2018 and now runs four models inside the patent rights process, none of which makes a final decision on an application, the ANAO found the use of AI largely effective (ANAO, 2026). The verdict is the counterweight this argument needs, and the reason behind it deserves saying plainly: IP Australia kept its tools away from the decision itself for eight years.

Both audits then reach the same weaker finding at the same point. Arrangements at the ATO for monitoring and evaluating its adoption were partly effective, and so was IP Australia’s reporting on the impact of its own. Largely effective overall, and partly effective at the point of knowing whether it worked. The second report also records that as at May 2026, IP Australia identified 30 AI tools in use across the agency. Twelve of them had no documented risk assessment, and retrospective assessments were paused as of April 2026. That is an inventory hole at an office the Auditor-General had just called largely effective, weeks before the Commonwealth created a national one.

None of these findings sit in model quality, which is the part worth taking into December. Strategic framework, inventory, committee role clarity, use-case ownership and benefit measurement are operating-model questions, and an impact assessment completed once answers none of them. What would help is narrower and duller than a framework. Wherever an in-scope system can change a record, the evidence should include a comparison of system state before and after, produced without asking that system what it did. The comparison should repeat on a schedule, because the failure mode above is stochastic and one clean result tells you almost nothing. Somebody named in the register should have to read it, because an artefact nobody opens is testimony too.

I have argued before that the AI assurance market is immature and that procurement language has run ahead of assurance science. My claim here is narrower and harder. The instruments are aimed at the wrong artefact: they read what a system says about itself, and one reporting a refusal it never performed will pass every last one. Somewhere in those 2,128 runs is a row that graded clean and should not have. Every regime built between now and 2027 will produce rows like it, and the only way anybody finds out is by looking at what changed.

References