Require reassessment of high-consequence AI after material changes
4 submissions. Each submission counts once for this shared ask. Counts do not measure importance or merit.
A shared ask groups requests for substantially the same action from three or more submissions when the group was created. Groups remain listed if withdrawals reduce their counts. AI models summarised recommendations, grouped them and checked the groups. The label summarises the requests; it is not a quote. A shared ask does not mean agreement on anything else.
Theme: AI assurance and auditing
Recommendations from submissions
Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Extracts are chosen by an AI model. Submissions are listed in submission number order; recommendations follow their order on each submission’s page.
Chameleon Confidential Solutions Pty Ltd Submission 99
Recommendation 2: establish proportionate high-consequence AI material-change triggers, potentially including technology, functionality, purpose, data, autonomy, regulation, suppliers or dependencies; assess materiality first and reassess proportionately to effects on evidence supporting continued reliance.
Further develop lifecycle-assurance principles by establishing consistent and proportionate material-change triggers for high-consequence AI. Relevant changes could include technology, functionality, purpose, data use, autonomy, regulation, suppliers or critical dependencies. Detection of change should lead first to an assessment of materiality, with reassessment proportionate to its potential effect on the evidence supporting continued reliance.
Dr Rebecca L. Johnson Submission 107
Have the Office of AI, regulators and technical evaluation bodies define review triggers for model, prompt, tool, permission, memory, data-access or workflow changes that could materially affect the previously evaluated property.
The Office of AI should coordinate, with relevant regulators and technical evaluation bodies, clear material-change triggers for reviewing whether previous evaluation claims remain valid. Relevant changes may include a different underlying model, substantial prompt or system-instruction changes, new tools, expanded permissions, new memory functions, altered data access or significant changes to the institutional workflow in which the system operates. The trigger should turn on whether the change could materially affect the property previously evaluated.
Mr Jared Condon Submission 190
Use assurance triggers for model/checkpoint replacement, material prompt/system-instruction changes, new tools/connectors, retrieval-corpus, permission, grader, user-population or environment changes, and significant new failure evidence; reassess affected claims/tests and rerun necessary evidence without automatic full reassessment.
Important AI assurance records should contain explicit change triggers. Examples include: • model or checkpoint replacement; • material prompt or system-instruction changes; • new tools or connectors; • retrieval-corpus changes; • permission changes; • changes to automated graders; • significant new failure evidence; • changes to the intended user population; and • changes to the operational environment. The response need not always be a complete reassessment. A change-triggered process can identify which claims and tests are affected and rerun only the necessary evidence.
Mr Joel Leslie Submission 417
Develop common behavioural-assurance methods reflecting function, consequence, autonomy, human reliance, observability, reversibility and context, with reassessment after material change.
Develop common behavioural-assurance methods. Require assurance intensity to reflect function, consequence, autonomy, human reliance, observability, reversibility and context, with reassessment after material change.