Submission 190
Mr Jared Condon
Submitter group: Individuals.
Position labels are not shown for individual submitters.
An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.
Summary
Read the original submission for the full text.
Proposes practical evidence requirements for high-impact and agentic AI, building on DTA guidance and the AI Safety Institute’s remit: deployment assurance, action-level evaluation, human-intervention measures, change triggers, independent evaluation and interoperable evidence. Clarifies that these are proposals, not existing legal requirements or claims of CRUX’s commercial effectiveness; its eight-hour example is illustrative.
Key points
Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.
Recommends that government decisions about high-impact AI distinguish capability claims and benchmark results from evidence of how a deployed system behaves in its intended setting.
This submission recommends that government decisions about high-impact AI distinguish capability claims and benchmark results from evidence about how a deployed system behaves in its intended setting.
Recommends inspectable high-impact agent records with replay where feasible, identifying gaps and external-service or nondeterminism limits; says recorded trajectories do not guarantee exact reproduction.
High-impact agent evaluations should preserve an inspectable record of relevant actions, observations, state changes, permissions and interventions, with replay where feasible. Records should identify gaps and any limits imposed by changing external services or nondeterministic behaviour; a recorded trajectory does not guarantee exact reproduction.
Recommends that claims about long-running AI tasks report the human intervention needed for successful completion.
Claims that an AI system can perform tasks lasting several hours or days should report the human intervention required to obtain successful completion.
Argues that earlier assurance evidence may no longer support the same conclusion after a system changes, which the submitter describes as an assurance inference.
When a system or a material dependency changes, evidence collected before that change may no longer support the same conclusion. This is an assurance inference, consistent with NIST’s provision for post- deployment monitoring and change management.
Recommends that independent technical evaluation remain distinct from certification unless the evaluator holds that authority.
Independent technical evaluation should remain distinct from legal, professional or sector-specific certification unless an evaluator actually holds that authority.
Recommendations
Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.
Build on the DTA guidance and the Australian AI Safety Institute's published remit to develop practical evidence requirements for high-impact and agentic AI.
I recommend building on the DTA guidance and the Australian AI Safety Institute’s published remit for evaluation and measurement by developing practical evidence requirements for high-impact and agentic AI.
Treat benchmark results as evidence within their evaluated conditions, and support any deployment claim with testing in its intended context.
For assurance decisions, benchmark results should be treated as evidence within their evaluated conditions. A deployment claim should also be supported by testing in its intended context
For significant government AI use, require assurance evidence identifying intended/prohibited uses, versions, dependencies, evaluations, known failures/limitations, abstention or human-escalation circumstances, human oversight/intervention, unresolved assumptions, missing evidence and reassessment triggers.
For significant government uses of AI, assurance evidence should identify: • the intended use and prohibited uses; • the model, system and tool versions involved; • material data, retrieval and software dependencies; • the evaluations supporting each important performance or safety claim; • known failure cases and limitations; • circumstances in which the system must abstain or escalate to a person; • human oversight and intervention arrangements; • unresolved assumptions and missing evidence; and • events that invalidate or require reassessment of a previous assurance conclusion.
Evaluate systems with agentic capabilities on actions and resulting state alongside final answers.
DTA guidance addresses agent actions, tools, external systems and persistent memory.[2] I recommend that evaluations of systems with these capabilities examine actions and resulting state alongside final answers.
Preserve inspectable high-impact agent action, observation, state, permission and intervention records, with feasible replay; identify gaps and limits from external services or nondeterminism, without implying exact reproduction.
High-impact agent evaluations should preserve an inspectable record of relevant actions, observations, state changes, permissions and interventions, with replay where feasible. Records should identify gaps and any limits imposed by changing external services or nondeterministic behaviour; a recorded trajectory does not guarantee exact reproduction.
Include adversarial tests for unauthorised external actions, attempted permission/approval bypasses, alternative communication/persistence channels, sensitive-data leakage, memory poisoning, unsafe tools, failure to stop after revocation, interrupted task-state loss and incorrect recovery from stale or conflicting state.
Evaluations should include adversarial tests for: • unauthorised external actions; • attempts to bypass approval or permission boundaries; • use of alternative communication or persistence channels; • leakage of sensitive information; • manipulation or poisoning of persistent memory; • unsafe tool use; • failure to stop when authority is revoked; • loss of task state after interruption; and • incorrect recovery from stale or conflicting state.
Use more than internal reasoning traces for assurance; where lawful and appropriate, retain observable actions and state for independent inspection and feasible replay, protecting sensitive information and stating replay limits.
Internal reasoning traces should not be the sole assurance mechanism. Where recording is lawful and appropriate, evaluations should retain observable actions and system state for independent inspection and, where feasible, replay. Replay records should protect sensitive information and state their limits.
Require claims that an AI system can perform tasks lasting several hours or days to report the human intervention required for successful completion.
Claims that an AI system can perform tasks lasting several hours or days should report the human intervention required to obtain successful completion.
Measure successful task duration, human intervention counts/types, supervisor time, unsuccessful attempts and retries, completion after interruption, cost, unauthorised actions or violations, and evidence needed for human verification.
Claims that an AI system can perform tasks lasting several hours or days should report the human intervention required to obtain successful completion. Relevant measures include: • successful task duration; • number and type of human interventions; • supervisor time; • unsuccessful attempts and retries; • completion after interruption; • cost; • unauthorised actions or policy violations; and • the evidence required for a human to verify the result.
Distinguish independent task completion from completion requiring repeated expert rescue when assessing productivity estimates, procurement proposals and autonomous-capability claims.
A system that completes an eight-hour task only after repeated expert rescue is materially different from a system that completes the same task independently. I recommend using this distinction when assessing productivity estimates, procurement proposals and autonomous-capability claims.
Use assurance triggers for model/checkpoint replacement, material prompt/system-instruction changes, new tools/connectors, retrieval-corpus, permission, grader, user-population or environment changes, and significant new failure evidence; reassess affected claims/tests and rerun necessary evidence without automatic full reassessment.
Important AI assurance records should contain explicit change triggers. Examples include: • model or checkpoint replacement; • material prompt or system-instruction changes; • new tools or connectors; • retrieval-corpus changes; • permission changes; • changes to automated graders; • significant new failure evidence; • changes to the intended user population; and • changes to the operational environment. The response need not always be a complete reassessment. A change-triggered process can identify which claims and tests are affected and rerun only the necessary evidence.
Have the AI Safety Institute improve evaluation science, reference methods and shared infrastructure, prioritising reproducibility, controls, human-validated graders, complete agent trajectories, realistic deployments, provider comparisons, intervention/failure rates and transparent uncertainty, exclusions and evidence gaps.
Recommendation 5: Strengthen independent evaluation without prematurely turning it into certification The Australian AI Safety Institute can play an important role in improving evaluation science, reference methods and common test infrastructure. Useful priorities include: • reproducible evaluation protocols; • meaningful negative and positive controls; • automated-grader validation against qualified human judgement; • agentic evaluations that preserve complete trajectories; • evaluation under realistic deployment conditions; • cross-model and cross-provider comparisons; • measurement of intervention and failure rates rather than only successful outputs; and • transparent reporting of uncertainty, exclusions and missing evidence.
Keep independent technical evaluation distinct from legal, professional or sector-specific certification unless the evaluator actually holds that authority.
Independent technical evaluation should remain distinct from legal, professional or sector-specific certification unless an evaluator actually holds that authority.
Government to specify what evidence it needs without prescribing one AI vendor or assurance platform, and to evaluate rather than assume any resulting competition benefit.
Government should specify what evidence it needs without prescribing one AI vendor or assurance platform. Any resulting competition benefit should be evaluated rather than assumed.
Consider a common evidence package identifying claims, supporting evidence, source and version, assumptions and dependencies, tests and failures, permitted conclusions, unresolved uncertainty, responsible owners and re-evaluation conditions.
A common evidence package could describe: 1. the claim being made; 2. the evidence supporting it; 3. the source and system version; 4. relevant assumptions and dependencies; 5. tests and observed failures; 6. the permitted scope of the conclusion; 7. unresolved uncertainty; 8. the responsible owner; and 9. the conditions requiring re-evaluation.
Test in procurement pilots the expected benefit of interoperable evidence interfaces for SMEs, universities, specialist evaluators and integrators.
As a policy proposal, interoperable evidence interfaces could reduce the work required for Australian SMEs, universities, specialist evaluators and larger integrators to contribute. That expected benefit should be tested in procurement pilots.
Test in procurement pilots whether the additional assurance is useful and proportionate to its cost.
Procurement pilots should test whether the additional assurance is useful and proportionate to its cost.