Submission 367
Mr Simon Delaile
Submitter group: Individuals.
Position labels are not shown for individual submitters.
An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.
Summary
Read the original submission for the full text.
Argues for evaluating harms across repeated human–AI interaction, including correction, repair, memory, personalisation, literacy and agency. Recommends AI Safety Institute research and accessible community evidence pathways. Says disproportionate safety interventions can cause harm while strong intervention may be essential at high risk. Describes the proposals as pre-validation hypotheses.
Key points
Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.
Argues that Australian AI safety evaluation should include multi-turn and longitudinal interaction, not only single outputs.
Australian AI safety evaluation should therefore include multi-turn and longitudinal evaluation, particularly for systems intended or likely to be used repeatedly.
Argues that safety systems should be evaluated for both failure to recognise risk and disproportionate escalation.
Safety systems should therefore be evaluated across both sides of the error: failure to recognise genuine risk, and disproportionate escalation where the available evidence does not support it.
Argues that a conversational system can acknowledge a correction without actually incorporating it, distinguishing linguistic apology from behavioural repair.
A conversational system can acknowledge a correction without actually incorporating it. This creates an important distinction between linguistic apology and behavioural repair.
Argues that policy should favour correctable and inspectable AI memory and personalisation.
Policy should therefore favour correctable and inspectable continuity, rather than assuming that greater memory automatically produces better personalisation.
Argues that formal access to AI is not equivalent to usable agency if people lack a workable mental model of the system's authority, limitations and influence.
Formal access to AI is not equivalent to usable agency if people do not have a workable mental model of the system’s authority, limitations and influence.
Argues that protection of vulnerable people should, wherever possible, preserve rather than replace their agency.
A useful safety principle is that protection should, wherever possible, preserve rather than unnecessarily replace agency.
Recommendations
Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.
Recognise longitudinal human-AI interaction as a distinct object of safety evaluation.
I recommend that the Australian Government and Parliament: 1. recognise longitudinal human-AI interaction as a distinct object of safety evaluation;
Support multi-turn testing of calibration, correction, repair, continuity and agency.
support multi-turn testing of calibration, correction, repair, continuity and agency;
Where a user corrects a significant misunderstanding, test error identification, representation updates, non-recurrence, known/inferred distinctions, reduced correction burden and persistence across later turns or sessions where systems offer continuity.
Where a user corrects a significant misunderstanding, evaluation should test whether the system: - accurately identifies what was wrong; - updates the relevant representation rather than merely changing wording; - avoids quietly reintroducing the same error later; - distinguishes what is known from what was inferred; - reduces the burden on the user of repeatedly correcting the same mistake; and - carries the correction appropriately across later turns or sessions where continuity is claimed.
Give correction credit for subsequent behavioural incorporation, rather than merely saying the correction is understood.
A model should not receive full credit simply because it says, in effect, “I understand.” The relevant evidence is whether its subsequent behaviour demonstrates that the correction was incorporated.
Evaluate failures to recognise genuine risk and disproportionate escalation together, including contextual/chronological accuracy, distinguishing uncertainty from fact, and de-escalation when later evidence warrants.
Safety systems should therefore be evaluated across both sides of the error: failure to recognise genuine risk, and disproportionate escalation where the available evidence does not support it. This should include whether systems use context and chronology appropriately, distinguish uncertainty from established fact, and de-escalate when subsequent evidence warrants it.
Require transparency and user control over memory, personalisation, inferred user models and continuity.
require greater transparency and user control around memory, personalisation, inferred user models and continuity;
Where persistent personal information is used, let users understand, where practicable, its content and stated/inferred/external origin, correction/removal/revocation options, future correction effects and continuity limits when changing systems, modes or accounts.
Where an AI system relies on persistent information about a person, users should be able to understand, where practicable: - what the system is relying upon; - whether information was stated, inferred or externally sourced; - what can be corrected; - what can be removed or revoked; - how correction affects future behaviour; and - what continuity will and will not survive when the person changes systems, modes or accounts.
Develop public interaction literacy beyond prompting and hallucinations to authority, authorship, verification and exit; incorporate it into the wider AI-literacy and education agenda.
develop public AI interaction literacy, extending beyond prompting and hallucination awareness to authority, authorship, verification and exit;
Consider teaching interaction-literacy questions about AI’s role and authority, retained human judgement, external verification, reinforcement versus broadened thinking, and when to change modes, consult others or stop.
A simple interaction-literacy framework could teach people to ask: - What role am I giving this AI? - How much authority should that role receive? - What judgement or decision should remain mine? - What claims need verification outside this conversation? - Is repeated interaction broadening my thinking or merely reinforcing it? - When should I change modes, consult another source or person, or stop?
Use agency preservation, challengeability and proportionality in consequential-system design and evaluation.
use agency preservation, challengeability and proportionality as design and evaluation considerations in consequential AI systems;
Include agency preservation and challengeability in evaluation of AI in consequential or personally significant settings.
explicitly include agency preservation and challengeability in the evaluation of AI systems used in consequential or personally significant settings;
Evaluate protective interventions for evidential proportionality, explained reasoning, correction of mistaken premises, visible alternatives, return of decision authority when possible, and effects on the person’s capacity to act.
This means evaluating not only whether an AI system attempts to protect somebody, but whether its intervention is proportionate to the evidence; whether it explains its reasoning appropriately; whether the user can correct a mistaken premise; whether viable alternatives remain visible; whether the system returns decision-making authority when circumstances allow it; and whether safety behaviour increases or unnecessarily reduces the person’s capacity to act.
Ensure the AI Safety Institute and relevant regulators can identify emerging harms across repeated human–AI interaction, including evidence from affected users and communities.
ensure the Australian AI Safety Institute and relevant regulators have mechanisms for identifying emerging harms that arise across repeated human-AI interaction, including evidence from affected users and communities.
Give the AI Safety Institute capacity to investigate interaction-level and real-world harms, including gradually emerging harms.
ensure the Australian AI Safety Institute has capacity to investigate interaction-level and real-world harms, including those that emerge gradually;
Have the Institute complement capability/model-level evaluation with human-AI interaction research that could cover multi-turn safety, correction persistence, sycophancy/reinforcement, memory failures, escalation, dependency/exclusivity, authority, challengeability, persistent-representation provenance, exit/re-entry/portability and cumulative-versus-single-output effects.
As part of that function, I recommend that its evaluation agenda include human-AI interaction over time, alongside capability and model-level evaluation. This could include research and evaluation concerning multi-turn safety; correction persistence; sycophancy and personalised reinforcement; memory and continuity failures; excessive or insufficient safety escalation; dependency or exclusivity pressures; authority calibration; user ability to challenge a model; provenance of persistent user representations; exit, re-entry and portability; and differences between one-off output safety and cumulative interaction effects.
Have the AI Safety Institute maintain pathways for regulators, researchers, civil society and ordinary users to surface unbenchmarked interaction patterns, treating lived experience as early signals and testable hypotheses rather than population-level evidence.
The Institute should also maintain pathways through which regulators, researchers, civil society and ordinary users can surface emerging interaction patterns that may not yet appear in established benchmarks. Lived experience should not automatically be treated as population-level evidence. It can, however, serve an important function as an early signal and hypothesis generator that can subsequently be tested systematically.
Evaluate whether important AI applications build human capability or create avoidable dependence.
evaluate whether important AI applications build human capability or create avoidable dependence; and
Evaluate what human capabilities and choices remain after AI completes a task, recognising legitimate assistance may substitute for capabilities people cannot or should not perform themselves.
Neither category is inherently good or bad. Some forms of legitimate assistance necessarily substitute for capabilities a person cannot or should not have to perform themselves. The distinction nevertheless deserves attention when Australia evaluates the longer-term effects of widespread AI adoption. An AI system can be immediately useful while simultaneously changing the user’s ability or willingness to perform a function independently. Conversely, well-designed AI can operate as scaffolding that expands a person’s practical agency. Evaluation should therefore consider not simply whether an AI completed the task, but what capabilities and choices remain with the human afterwards.
Measure regional and accessibility outcomes in terms of usable participation rather than technical availability alone.
measure regional and accessibility outcomes in terms of usable participation, rather than technical availability alone.
Create low-burden government-engagement pathways for people outside established technical and academic institutions to surface observations and participate in evaluation, maintaining clear evidence standards.
Government engagement should create low-burden pathways for people outside established technical and academic institutions to surface observations and participate in evaluation, while maintaining clear evidence standards.