Submission 553
Web Dance
Submitter group: Companies.
An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.
Summary
Read the original submission for the full text.
Draws on the Medicare Statistics Reporting Service incident to argue that AI incident notification should cover any operator affecting Australian systems and that remediation and agentic AI controls should be independently verified. Makes six scheme-neutral recommendations and declares the company's commercial interest in AI assurance.
Key points
Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.
Argues the Medicare incident showed that, under current arrangements, Australia depends on AI developers to detect, report and describe remediation of their own incidents.
The Medicare Statistics Reporting Service incident showed that, under current arrangements, Australia depends on AI developers to detect their own incidents, decide whether and how to report them, and describe their own remediation.
Says that, on the public description, a regime limited to developers training AI in Australia may not have captured the incident, which involved an agent operated by a developer outside Australia acting on an Australian government system.
On that description, a regime limited to developers training in Australia may not have captured this incident, which involved an agent operated by a developer outside Australia acting on an Australian government system.
Argues notification does not establish whether problems are fixed or systems already deployed are adequately controlled.
Notification tells government that something went wrong. It does not tell government whether the problem has been fixed, or whether systems already in use are adequately controlled.
Argues assurance results should state the depth of verification so procurers, regulators and the public can distinguish declarations from independent verification.
Not all assurance provides the same confidence. A declaration reviewed for completeness and plausibility, a review of submitted evidence, and direct independent verification of a system in operation are very different things. Results should make that difference visible, so that procurers, regulators and the public can tell them apart.
Says the incident's risk lay in an agent's actions rather than outputs, whereas most AI evaluation tests what a model outputs.
The risk was in actions, not outputs. The incident involved an agent acting on a system it did not own, after its requests were refused. Most AI evaluation tests what a model outputs. This incident concerned what an agent did.
Recommendations
Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.
Apply mandatory AI incident notification to any developer or operator whose AI acts in or against Australian government systems, critical infrastructure or Australians' personal information, wherever developed, trained or operated.
Mandatory notification of AI incidents should apply to any developer or operator whose AI system acts in or against Australian government systems, critical infrastructure or the personal information of Australians, regardless of where the system was developed, trained or operated.
Start notification obligations from the point an organisation becomes aware of an incident, with an initial notice within a short defined period consistent with critical infrastructure cyber incident reporting, followed by fuller reporting.
Notification obligations should run from the point an organisation becomes aware of an incident, with an initial notice required within a short defined period, consistent with existing critical infrastructure cyber incident reporting, followed by fuller reporting.
Distinguish incidents from an organisation's own development or use of AI from incidents in AI products supplied to others, which also require notification of the customers and operators running the product.
Obligations should distinguish incidents arising from an organisation's own development or use of AI from incidents in AI products supplied to others, which also require notification of the customers and operators running the product.
Establish a single designated reporting channel for AI incidents, with an obligation to also notify the accountable executive of any affected agency directly.
A single designated reporting channel for AI incidents should be established, with an obligation to also notify the accountable executive of any affected agency directly.
Where an organisation reports a significant AI incident and commits to remediation, independently verify the remediation and report the outcome to the relevant regulator.
Where an organisation reports a significant AI incident and commits to remediation, the remediation should be independently verified and the outcome reported to the relevant regulator.
Require independent assurance evidence, covering operating boundaries, monitoring and incident response, in Commonwealth procurement and deployment of AI, particularly agentic systems with network or system access, not only a supplier declaration.
Commonwealth procurement and deployment of AI systems, particularly agentic systems with network or system access, should require independent assurance evidence covering operating boundaries, monitoring and incident response, not only a supplier declaration.
Provide independent conformity assessment by competent, impartial assessors; set criteria recognising credible Australian or international assurance schemes, requiring every published result to state its verification depth.
The proposed Australian AI standards legislation should provide for independent conformity assessment by assessors who meet defined requirements for competence and impartiality, and should set criteria under which credible assurance schemes, Australian or international, can be recognised, including a requirement that every published result states the depth of verification behind it.
Extend independent assessment beyond the model to data, the product or service and the operating organisation, covering AI systems’ operating boundaries.
Independent assessment must also reach the operating boundary of AI systems, not only the model. An effective assessment examines the model, the data it relies on, the product or service it is part of, and the organisation operating it, because a weakness at any one of those layers can undermine controls at the others.