Submission 95
Australian Medical Association (AMA)
Submitter group: Health.
An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.
Summary
Read the original submission for the full text.
The AMA argues that Australia's laws are not yet adequate for AI in healthcare, while supporting safe, clinician-led adoption. It recommends framework legislation working with TGA and Ahpra mechanisms, default high-risk treatment of health AI, a clinical AI governance body, liability allocated by control, and protection of patient data.
Key points
Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.
Says that current laws and regulatory frameworks are not yet adequate for AI in healthcare, a view it describes as broadly held in the sector.
The AMA’s assessment is Australia’s current laws and regulatory frameworks are not yet adequate for AI in healthcare. This is a view broadly held by the healthcare sector.
Recommends default high-risk classification for healthcare AI, with oversight calibrated to autonomy, consequences, reversibility and human oversight.
The AMA recommends AI in healthcare be designated high risk by default, with the intensity of oversight calibrated to the degree of autonomy, potential consequences, the degree to which decisions may be safely revised and the level of human oversight associated with a particular system.
Argues that general-purpose AI used in clinical settings takes on a clinical function and should be governed accordingly.
General-purpose AI (GPAI) used in clinical settings takes on a clinical function by virtue of its use in a clinical context and should be governed accordingly.
Argues that responsibility and liability should be allocated among clinicians, health services and vendors according to their control.
Responsibility and liability must be clearly allocated between clinicians, health services, and technology vendors according to their respective control over clinical decisions, procurement, and system design.
Says patient health information must be protected and that patients, not vendors or health services, must remain the owners of their health data.
Patient health information must be protected through robust privacy, security, and data governance requirements, and patients, not vendors or health services, must remain the owners of their health data.
Recommendations
Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.
Designate healthcare AI high risk by default; calibrate oversight to autonomy, consequences, reversibility and human oversight, allowing streamlined pathways for fully clinician-reviewed tools and strongest oversight for autonomous systems or those materially influencing clinical decisions.
The AMA recommends AI in healthcare be designated high risk by default, with the intensity of oversight calibrated to the degree of autonomy, potential consequences, the degree to which decisions may be safely revised and the level of human oversight associated with a particular system. On this proportionate basis, a fully clinician-reviewed tool can follow a streamlined pathway, while the most intensive oversight is reserved for systems that act autonomously or materially influence a clinical decision.
Establish a clinical AI governing or advisory body to help regulators monitor risks and gaps, consult AI, healthcare, ethics and law experts, and embed practising clinicians in clinical-care decisions.
The AMA recommends the establishment of a dedicated clinical AI governing or advisory body to embed healthcare-specific expertise within Australia’s AI governance arrangements. This body would support regulators to monitor emerging risks and regulatory gaps, convene ongoing consultation with experts in AI, healthcare, ethics and law, and ensure practising clinicians are embedded in decisions affecting clinical care.
Establish overarching framework legislation that integrates with, and is delivered through, TGA risk-based regulation of medical devices and software as a medical device, and Ahpra and Medicare Board practitioner regulation.
reform should establish an overarching framework legislation that integrates with, and is delivered through, Australia’s proven health regulatory mechanisms. Central to this framework are the Therapeutic Goods Administration’s risk-based regulation of medical devices and software as a medical device, and the professional regulation of practitioners through Ahpra and the Medicare Board of Australia.
Govern general-purpose AI used in clinical settings as a clinical function; it should not be treated as low risk because it is marketed as administrative.
General-purpose AI (GPAI) used in clinical settings takes on a clinical function by virtue of its use in a clinical context and should be governed accordingly. It cannot be treated as low risk simply because it is marketed as administrative.
Ensure framework legislation captures AI scribes and 'scope creep' where software updates extend a tool's functionality into medical-device territory, kept in step with clinical governance.
legislation should ensure such tools, and instances of “scope creep” where software updates extend a tool’s functionality into medical-device territory, are captured and kept in step with clinical governance.
Include AI scribes within the definition of software as a medical device.
Currently, AI scribes are not encompassed within the definition of ‘software as a medical device’, and this should be remedied urgently to protect both patients and doctors.
Allocate liability by control over clinical decisions, procurement and system design: health services retain procurement and deployment decisions; vendors answer for defects in design, training, validation, security and autonomous behaviour.
Responsibility and liability must be clearly allocated between clinicians, health services, and technology vendors according to their respective control over clinical decisions, procurement, and system design. At present, accountability tends to settle by default on the clinician, even where the source of harm lies in model design or procurement. The AMA supports a clear principle that liability follows control. Procurement and deployment decisions remain under the purview of health services, but vendors must be responsible for defects in model design, training, validation, security, and autonomous behaviour.
Keep ultimate patient-care responsibility with a human clinician and prevent non-clinical actors from overriding clinical judgement.
Medical practitioners retain ultimate responsibility for patient care, and this agency must always remain with a human clinician. Clinical judgement must never be overridden by non-clinical actors.
Hold software vendors of clinical AI tools to equally robust accountability standards, including being compelled to act when safety and reliability standards are not met.
It is therefore essential software vendors developing and maintaining the AI tools for use in clinical settings are held to equally robust standards of accountability, ensuring their products meet expected safety and reliability standards and that they are compelled to take action when those standards are not met.
Apply governance to the complete AI system (models, data sources, integrations, workflows, autonomous functions), supported by transparency, validation and ongoing monitoring for errors, deterioration and bias.
Governance should also apply to the complete AI system, including underlying models, data sources, integrations, workflows, and autonomous functions, rather than the model alone. Governance should be supported by transparency, appropriate validation, and ongoing monitoring for errors, performance deterioration, and bias.
Protect patient health information through robust privacy, security and data governance requirements, with patients, not vendors or health services, remaining the owners of their health data.
Patient health information must be protected through robust privacy, security, and data governance requirements, and patients, not vendors or health services, must remain the owners of their health data.
Support Australian and, where feasible, onshore data storage and the development of foundation models on Australian data.
The AMA supports Australian, and where feasible, onshore data storage, and the development of foundation models on Australian data as the strongest safeguard for both data security and population relevance.
Strengthen transparency requirements for the foundation models underpinning clinical AI and ensure use of health data is subject to full disclosure and informed patient consent.
Reform should also strengthen transparency requirements for the foundation models that underpin clinical AI and ensure the use of health data is subject to full disclosure and informed patient consent.
Ensure AI supports, rather than replaces, clinical judgement and professional autonomy.
AI supports, rather than replaces, clinical judgement and professional autonomy. Ultimate responsibility for patient care should remain with a human clinician.
Ensure meaningful clinician involvement throughout clinical AI development, assessment and monitoring.
Clinical AI systems are developed, assessed, and monitored with meaningful clinician involvement throughout their lifecycle.
Ensure regulatory frameworks include mechanisms to identify and mitigate algorithmic bias and prevent AI from exacerbating existing health inequities.
Regulatory frameworks include mechanisms to identify and mitigate algorithmic bias and prevent AI from exacerbating existing health inequities.