AI Inquiry Atlas

An unofficial, independent guide to the public records of the Joint Select Committee on Artificial Intelligence. Not connected with the Parliament of Australia or the committee. Its makers also made a submission (no. 481; see About). Summaries and scores are written by AI and can be wrong: check the original before relying on them.

Submission 487

Ms Julie Green

Submitter group: Individuals.

Position labels are not shown for individual submitters.

An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.

Open in the interactive guide

Summary

Read the original submission for the full text.

A social worker argues that predictive and generative AI in child protection and human services carries risks of bias, automation bias and lack of recourse, particularly for Aboriginal and Torres Strait Islander families, and does not believe AI can capture First Nations cultural norms. Recommends mandatory testing, human decision-makers, an AI Safety Institute unit, co-design and extended transparency.

Key points

Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.

  1. Describes how a 2022 Associated Press investigation found the Allegheny Family Screening Tool uses data points that can function as proxies for race and disability.

    An Associated Press investigation in 2022 found the tool uses data points, including disability and mental health diagnoses, that can function as proxies for race and disability [4].

  2. Says Australia has not yet embedded predictive or generative AI into child protection at scale, so rules can be set first.

    Australia has not yet embedded predictive or generative AI into child protection at scale. This committee has a genuine chance to set the rules before that happens rather than after.

  3. Argues that an AI tool trained on data the child protection system already produces would inherit that record for Aboriginal and Torres Strait Islander communities rather than correct it.

    Little is done to address the ongoing effects of colonisation, racism, and discrimination on these communities. An AI tool trained on the data this system already produces would inherit that record, not correct it.

  4. Describes lack of informed consent as a risk: families may not be told a tool was used, cannot see their score or correct the data, and have no meaningful recourse.

    Lack of informed consent, families may not be told a tool was used, cannot see their score, cannot correct the underlying data, and have no meaningful recourse

  5. Recommends a human decision-maker for any AI-informed decision affecting a child or vulnerable person, with AI as a decision-aid only.

    Require a human decision-maker in the loop for any AI-informed decision affecting a child or vulnerable person. AI should never be the decision-maker, at best a decision-aid. That person must be able to see and challenge the tool's reasoning, and the family must be told a tool was used and given a genuine way to contest it.

Recommendations

Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.

  1. Require mandatory testing of any AI tool used in government-funded human services, before deployment and periodically, against realistic scenarios involving children and vulnerable groups, with results published.

    Require mandatory testing of any AI tool used in government-funded human services, before deployment and periodically afterwards, against realistic scenarios involving children and vulnerable groups, with results published.

  2. Require human decision-makers for AI-informed decisions affecting children or vulnerable people; AI remains a decision-aid only. Decision-makers must be able to see and challenge its reasoning; families must be informed and able to contest use.

    Require a human decision-maker in the loop for any AI-informed decision affecting a child or vulnerable person. AI should never be the decision-maker, at best a decision-aid. That person must be able to see and challenge the tool's reasoning, and the family must be told a tool was used and given a genuine way to contest it.

  3. Fund the AI Safety Institute to build a unit with child protection and social work practice experience, able to independently evaluate and certify AI tools before use in child protection, family violence and similar services.

    Fund the Australian AI Safety Institute to build a dedicated unit, staffed by people with child protection and social work practice experience rather than only AI researchers, with the authority to independently evaluate and certify AI tools before they can be used in child protection, family violence, and other services touching vulnerable Australians.

  4. Co-produce AI tools in human services with social workers, frontline practitioners and holistic, diverse stakeholders, including diverse Aboriginal and Torres Strait Islander communities, involved in design from the outset.

    Get social workers and frontline practitioners in from the outset on the system design teams. Consultation after the fact will not be sufficient. Any AI tool used in this space should be built on coproduction principles, with holistic and diverse stakeholders involved from the start, including diverse Aboriginal and Torres Strait Islander communities.

  5. Extend government AI transparency and accountability frameworks to tools used by human services and community sector organisations delivering government-funded services.

    Extend any transparency and accountability framework for government AI use to tools used by human services and community sector organisations delivering government-funded services, not only tools inside Commonwealth agencies.

Read the original

Open in the interactive guide