Submission 344
Mr Stuart Le Gros
Submitter group: Civil society.
An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.
Summary
Read the original submission for the full text.
Argues that safeguards should attach to the consequence of automated systems rather than the AI label, using case studies on aged-care assessment and age-related recruitment screening. Recommends public registers, impact assessments, notice, empowered human review, audit records, outcome testing and procurement conditions.
Key points
Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.
Argues that safeguards should attach to the consequence of automated systems, including rules-based ADM not marketed as AI.
Artificial intelligence is only part of the accountability problem. Many decisions with serious human consequences are influenced by rules-based automated decision-making (ADM), scoring, ranking and screening systems that may not be marketed as “AI”. Safeguards should therefore attach to the consequence of the system, not its label.
Argues that affected people often cannot identify the system used, its decisive factors, error rates or the path to empowered human review.
The two domains are different, but the accountability deficit is the same: the person affected often cannot identify the system used, the factors that mattered, the version or threshold applied, the rate of error or disparate impact, or the path to an empowered human review.
Argues that a nominal human in the loop should not remove a system from scope where staff lack authority, time or information to depart from it.
A nominal “human in the loop” should not remove a system from scope where staff routinely accept its output or lack practical authority, time or information to depart from it.
Argues that a historical assessment, generated classification or queue position should never displace current clinical evidence.
A historical assessment, generated classification or queue position should never displace current clinical evidence.
Recommends that sufficiently large employers and labour-hire/recruitment platforms report age-banded outcomes at every recruitment stage, with privacy safeguards.
Make recruitment outcomes visible. Require sufficiently large employers and labour-hire/recruitment platforms to report age-banded outcomes at applied, screened, shortlisted, interviewed, offered and hired stages, with privacy safeguards.
Recommendations
Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.
Apply safeguards to AI and other automated decision-making that materially influences rights, entitlements, access to care, employment or public services.
Regulate consequential systems, not labels. Apply safeguards to AI and other ADM that materially influences rights, entitlements, access to care, employment or public services.
Require agencies, contractors and large employers to publish registers disclosing consequential systems, their purpose, decision points, owner, vendor, legal authority and current version.
Establish public registers. Require agencies, contractors and large employers to disclose consequential systems, their purpose, decision points, owner, vendor, legal authority and current version.
Require published pre-deployment and material-change impact assessments covering necessity, data quality, accessibility, human rights, age and other protected attributes, security, error and operational failure.
Require impact assessments. Publish pre-deployment and material-change assessments covering necessity, data quality, accessibility, human rights, age and other protected attributes, security, error and operational failure.
Give affected people notice when automation materially influenced a result, with the decisive factors, relevant input categories and accessible reasons.
Give notice and meaningful reasons. Tell affected people when automation materially influenced a result and provide the decisive factors, relevant input categories and accessible reasons.
Guarantee timely human review by a trained person who can examine source information, depart from the system and overturn the result; urgent clinical deterioration must trigger clinical escalation.
Guarantee empowered human review. Provide timely reconsideration by a trained person who can examine source information, depart from the system and overturn the result. Urgent clinical deterioration must trigger clinical escalation.
Retain inputs, outputs, overrides, reasons, thresholds, validation results and model, rules, version and change-control histories for an appropriate statutory period.
Preserve auditable records. Retain inputs, outputs, overrides, reasons, thresholds, validation results and model/rules/version and change-control histories for an appropriate statutory period.
Mandate independent testing for error and disparate impact, with public aggregate results and regulator access to protected technical detail.
Test and publish outcomes. Mandate independent testing for error and disparate impact, with public aggregate results and regulator access to protected technical detail.
Require sufficiently large employers and labour-hire/recruitment platforms to report age-banded applications, screening, shortlisting, interviews, offers and hires, with privacy safeguards.
Make recruitment outcomes visible. Require sufficiently large employers and labour-hire/recruitment platforms to report age-banded outcomes at applied, screened, shortlisted, interviewed, offered and hired stages, with privacy safeguards.
Make transparency, audit access, data portability, incident reporting and human-review capability mandatory conditions of Commonwealth procurement and grants.
Use Commonwealth purchasing power. Make transparency, audit access, data portability, incident reporting and human-review capability mandatory conditions of Commonwealth procurement and grants.
For care and other public services, report approved support’s actual commencement, delay, reasons for non-delivery and outcome after review.
Measure delivery, not approval. For care and other public services, report whether approved support actually commenced, the delay, the reason for non-delivery and the outcome after review.
Apply proportionate safeguards when systems make or materially shape decisions affecting people; nominal human involvement must not exempt systems routinely accepted or lacking practical authority, time or information for departure.
The Committee should recommend a functional test: does the system make, recommend, rank, score, prioritise, filter or materially shape a decision affecting a person? If yes, proportionate safeguards should apply. A nominal “human in the loop” should not remove a system from scope where staff routinely accept its output or lack practical authority, time or information to depart from it.
Require the Hospital Discharge Joint Taskforce work, revised delayed-discharge definition and AIHW data standards to produce public comparable data and accountability for milestones, responsible agencies and actual service commencement.
The Health Ministers Meeting communiqué of 4 September records that delayed discharge is growing at an unprecedented rate. It also records an 18-month Hospital Discharge Joint Taskforce work program, a revised national definition of delayed discharge of an older person, and work by the AIHW on national data standards. These are important steps. They should produce public, comparable data and clear accountability for milestones, responsible agencies and actual service commencement.
In aged care, publish every assessment score or classification used, with the relevant factors and weightings or rules, the data relied upon, the date and version applied, and the consequence for eligibility, priority or funding.
What the Committee should require in aged care Publication of every assessment score or classification used, the relevant factors and weightings or rules, the data relied upon, the date and version applied, and the consequence for eligibility, priority or funding.
Provide a visible pathway for correcting inaccurate or stale information and seeking urgent reassessment after clinical deterioration.
A visible pathway for correcting inaccurate or stale information and seeking urgent reassessment after clinical deterioration.
Provide human clinical review with authority to override a score, with reasons where the system output is accepted or rejected.
Human clinical review with authority to override a score, together with reasons where the system output is accepted or rejected.
Include workforce/provider non-delivery, elapsed time and remedial action in reporting that separates approved funding from commenced care.
Reporting that separates approved funding from commenced care, including workforce/provider non-delivery, elapsed time and remedial action.
Publish changes to the system: what changed, why, validation undertaken, affected cohorts, transition arrangements and retrospective review of adverse outcomes.
Change-control publication: what changed, why, validation undertaken, affected cohorts, transition arrangements and retrospective review of adverse outcomes.
Use mutually exclusive statutory age bands, such as 55-59, 60-64 and 65+, in the accountability dataset for employers above a proportionate size threshold.
For employers above a proportionate size threshold, the minimum accountability dataset should show applications and progression by age band. To avoid double counting, the statutory bands should be mutually exclusive—for example 55-59, 60-64 and 65+—-while campaign communication can retain the simple shorthand “55+, 60+, 65+”.
Publish aggregated, privacy-preserving data, suppressing or combining small cells without excusing non-reporting; give regulators unsuppressed data under confidentiality protections.
Publication should be aggregated and privacy-preserving. Small cells should be suppressed or combined, but suppression must not become a blanket excuse for non-reporting. Regulators should receive the unsuppressed data under confidentiality protections.
Treat frontline-worker evidence as a design and assurance input.
Frontline-worker evidence should be treated as a design and assurance input, not merely an industrial issue.
Before deployment, also require consideration of alternatives, data provenance and quality, accessibility, rights and equality impact, a testing plan and a named accountable officer.
Before deployment Public purpose and legal authority; alternatives considered; data provenance and quality; accessibility; rights and equality impact; testing plan; named accountable officer.
At decisions, provide correction channels, time-bound human review and emergency escalation where safety is involved, alongside notice, explanations, decisive factors and data categories.
At the point of decision Clear notice; explanation of the system’s role; decisive factors and data categories; correction channel; time-bound human review; emergency escalation where safety is involved.
During operation, require outcome and override monitoring, incident reporting, independent audit, workforce consultation, vendor access rights, protected whistleblowing and periodic reauthorisation.
During operation Outcome and override monitoring; incident reporting; independent audit; workforce consultation; vendor access rights; protected whistleblowing; periodic re-authorisation.
When systems change, require reasons, validation and regression testing, affected-cohort identification, notification and capacity to reproduce past decisions, alongside versioned change records.
When the system changes Versioned change record; reason for change; validation and regression testing; affected cohorts; notification; capacity to reproduce past decisions.
When harm is alleged, require preserved records, accessible complaints, regulator powers, remedies, systemic review and public corrective-action reporting.
When harm is alleged Preservation of records; accessible complaint pathway; regulator powers; remedies; systemic review; public reporting of corrective action.
Contracts should guarantee documentation, validation data, logs, inputs, explanations and independent audit. Trade secrets may justify regulator-protected handling, but cannot extinguish rights to reasons or prevent legality, accuracy and non-discrimination assurance.
Contracts should guarantee access to documentation, validation data, logs, source inputs, meaningful explanations and independent audit. Trade-secret claims may justify protected handling by a regulator, but should not extinguish an affected person’s right to reasons or prevent assurance of legality, accuracy and non-discrimination.
Committee inquiry: Identify Commonwealth systems shaping decisions affecting care, benefits, employment or rights, including rules-based systems not described as AI.
Which Commonwealth systems currently make, recommend, score, rank or prioritise decisions affecting care, benefits, employment or other rights, including rules-based systems not described as AI?
Committee inquiry: For each system, establish the accountable senior officer, statutory authority, vendor and dates of the latest impact assessment and independent audit.
For each system, who is the accountable senior officer, what statutory authority applies, what vendor is involved, and when was the latest impact assessment and independent audit?
Committee inquiry: Test access to source data, decisive rules, versions, reasons and empowered human reconsideration without FOI.
Can an affected person obtain the data relied upon, the decisive rules or factors, the applicable version, meaningful reasons and an empowered human reconsideration without resorting to FOI?
Committee inquiry: Examine how many aged-care funding approvals fail to deliver services, delays and workforce, provider, market or administrative causes.
How many aged-care funding approvals do not become delivered services, for how long, and because of which workforce, provider, market or administrative constraints?
Committee inquiry: Test safeguards ensuring changed health or function overrides stale assessments and triggers urgent clinical review.
What safeguards ensure that changed health or functional circumstances override stale assessment data and trigger urgent clinical review?
Committee inquiry: Establish the share of recruitment decisions influenced by automation in the Commonwealth and its funded supply chains.
What proportion of recruitment decisions in the Commonwealth and Commonwealth-funded supply chains are influenced by automated screening, ranking or assessment?
Committee inquiry: Examine age-banded recruitment conversion rates, adverse-impact testing, validation and human-override monitoring.
What are the age-banded conversion rates at each recruitment stage, and what adverse-impact testing, validation and human-override monitoring is undertaken?
Committee inquiry: Test minimum contractual rights allowing agencies and regulators to audit vendor systems and reproduce decisions.
What minimum contractual rights will ensure that agencies and regulators can audit vendor systems and reproduce past decisions?