A survey published in late July went through 2,086 studies on artificial intelligence across the wildfire management lifecycle and counted how many used spatially blocked validation. Fewer than one per cent did. The authors draw the obvious conclusion, which is that most published fire-prediction models are probably overstating how well they would perform in the field (Artificial Intelligence Across the Wildfire Management Lifecycle, 2026).

The technical point behind that number is worth expanding, because it decides what the number means. Fire data is spatially autocorrelated: conditions at one point on a landscape resemble conditions fifty metres away, and a fire that burnt through one pixel almost certainly burnt through its neighbours. When a modeller splits such a dataset at random into training and test sets, adjacent pieces of the same fire end up on both sides of the split. The model is then scored on its ability to recognise something it has effectively already been shown. Spatially blocked validation holds out whole regions instead, so the model has to generalise to country it has never seen. That is the harder test, and it is the test that matches what an agency actually needs, which is a tool that works on the next fire instead of the last one.

For a fire agency reading vendor material, the consequence is immediate. Accuracy figures across the market were mostly generated by a method that inflates them, and inflates them unevenly depending on how each dataset happened to be structured. Two products quoting the same number are not necessarily comparable, and the product quoting the lower number may well be the better one, because a vendor who validated honestly is penalised in the comparison. Procurement panels have no reliable way to tell those cases apart from a specification sheet.

A second paper from the same week argues the measurement problem runs deeper than the split. It contends that standard machine learning metrics such as the F1-score are the wrong instrument for wildfire risk systems altogether, because they score prediction accuracy while agencies need to know whether a risk signal orders the landscape in a way that matches operational demand (Risk Is Not the Target, 2026). A model can be accurate and still rank two districts in an order that sends the tanker to the wrong one. The paper proposes a monotonic evaluation framework that asks whether the ordinal ranking of risk scores tracks real fire management dynamics, which is a much closer question to the one an incident controller is actually asking.

None of that is an argument against machine learning in fire management, and I have made the opposite case at some length. The models are not the constraint, as I argued in The Model Is Not the Bottleneck, and the capability gap sits in integration and adoption more than in accuracy, which was the subject of Fire Tech Has Outrun the Fire Agencies. The validation finding adds a third constraint to those two, and it is the one most directly under an agency’s control. An agency cannot fix a research field’s validation habits. It can absolutely refuse to buy on numbers produced by them.

Underneath the evaluation problem sits an older and more embarrassing one. Researchers and fire service practitioners agreed at a July webinar that the country still lacks a single usable national bushfire database, and that the data largely exists without being integrated or harmonised across jurisdictions (BNHCRC, 2026a). Both the 2020 Royal Commission into National Natural Disaster Arrangements and the 2020 NSW Independent Bushfire Inquiry found that Australia had no national centralised source capable of showing trends in fire intensity and extent, or how far mitigation reaches and how well it works. Six years later the position is a scoping study through Natural Hazards Research Australia, presented at this year’s research forum, which is genuine work but is not a working system. I described that gap as a governance problem wearing a technical costume in A Governance Problem Dressed as a Data Problem, and nothing since has changed the diagnosis.

The two problems compound in a specific way. A model validated on one jurisdiction’s records cannot be checked against a national picture, because there is no national picture to check it against. Harmonised multi-jurisdictional data is precisely what a spatially blocked validation needs, since holding out a region only tests generalisation if the held-out region was recorded to the same standard as the training regions. Australia is asking its fire agencies to buy predictive tools while withholding the evidentiary base that would let anyone judge them.

The timing gives this an edge it would not otherwise have. The Conversation published a piece in late July tracing the climate drivers behind California’s recent fires and warning that Australia may enter spring with a highly flammable landscape (The Conversation, 2026). A Centre-funded project has built four scientifically plausible future scenarios, including events worse than Black Summer, for emergency management leaders to stress-test their preparedness against (BNHCRC, 2026b). Separate work on defendable space has flagged growing risk at the rural-urban interface (BNHCRC, 2026c). Changing vegetation and rising residential density there are producing fire behaviour that older planning assumptions do not cover. Agencies will be making procurement decisions and preparedness decisions in the same months, using an evidence base that cannot properly support either.

The remedy here is unusually concrete for a governance argument, and it belongs in contracts more than in research agendas. Require spatially blocked validation results as a condition of tender response, and require them on Australian fire history rather than on a Californian or Mediterranean benchmark. Ask vendors to report performance against an operational ranking criterion of the kind the monotonic framework describes, and treat an F1-score offered on its own as an incomplete answer. Ask which jurisdictions’ records the model was trained on, and be sceptical of any product that cannot say. Those are three answerable questions, and any vendor who cannot answer them has told a buyer something worth knowing.

There is a broader point in here about how research quality reaches operations. Fire agencies are not going to read 2,086 papers, and they should not have to. They need a purchasing standard that encodes the field’s own methodological findings, so that a validation problem identified in the literature in July becomes a tender requirement by summer. Australia has the institutions to do that. Natural Hazards Research Australia and AFAC could publish a common evaluation standard for fire-prediction products with the state agencies inside a matter of months, and it would do more for operational capability than another round of model development.

Australia is entering a fire season with a research field that has just admitted its accuracy numbers are inflated, a national dataset that six years of inquiries have failed to produce, and a procurement process that asks about neither. The most useful thing anyone could add before summer is a question on the tender form.

References