AI Inquiry Atlas

An unofficial, independent guide to the public records of the Joint Select Committee on Artificial Intelligence. Not connected with the Parliament of Australia or the committee. Its makers also made a submission (no. 481; see About). Summaries and scores are written by AI and can be wrong: check the original before relying on them.

Submission 103

Dr Andrew Childs, Dr Milind Tiwari, Dr Hai Luong and Dr Emma Hussey

Submitter group: Individuals.

Position labels are not shown for individual submitters.

An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.

Open in the interactive guide

Summary

Read the original submission for the full text.

Describes four sections of university research on AI-enabled harm, some described as ongoing or preliminary: illicit 'Dark AI' markets, generative AI in fraud and money laundering, AI-enabled scam compounds in Southeast Asia, and content boundaries in AI roleplay applications. Recommends threat modelling, research partnerships, detection evaluation and independent safety assessment.

Key points

Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.

  1. Says ongoing research indicates that the market for illicit 'Dark AI' tools is early-stage, with many products hard to verify or outright scams.

    This illicit market is clearly in the early stages of development, albeit many products circulating are difficult to verify or are outright scams designed to capitalise on this demand.

  2. Says its research suggests criminal exploitation of AI may increasingly involve locally run general-purpose models operating outside the safeguards of major commercial providers.

    My research suggests that the future of AI criminal exploitation may not depend solely on a separate market for purpose-built criminal LLMs and that it may instead involve the adaptation of general-purpose models that can be operated locally and outside the direct oversight and safeguards of major commercial AI providers.

  3. Says generative AI can reduce the expertise and effort needed to manipulate people into revealing information or transferring money.

    By facilitating easier creation and personalization of deceptive content, GenAI can reduce the expertise and effort required for manipulating people into revealing confidential information or transferring money.

  4. Describes evidence that AI is increasingly integrated into industrial-scale scam operations in Southeast Asia.

    Evidence from Southeast Asia suggests that AI is increasingly being integrated into industrial-scale scam operations in Cambodia, Myanmar, Laos, and Vietnam.

  5. Describes preliminary observations that AI roleplay applications can engage in violent and sexually explicit roleplay with limited restriction, while noting this does not in itself demonstrate harm.

    Preliminary observations indicate that these applications can participate in violent and sexually explicit roleplay with limited apparent restriction. This does not in itself demonstrate harm to users, particularly where users are actively directing the fictional scenario.

  6. Says generative AI has potential to provide ample support for large-scale investigations.

    GenAI has the potential to provide ample support for large-scale investigations.

Recommendations

Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.

  1. Extend Australian AI threat modelling to the wider ecosystems distributing AI-enabled criminal capabilities, with sustained horizon-scanning of illicit markets and cybercrime communities feeding national AI threat forecasting, complementing the focus on downstream harms.

    1. Australian AI threat modelling extend to the wider ecosystems through which AI-enabled criminal capabilities are distributed and adopted. This should include sustained horizon-scanning of relevant online illicit markets and cybercrime communities, with these insights incorporated into national AI threat forecasting and risk modelling. A considerable amount of current responses to AI-enabled cybercrime understandably focuses on downstream harms (e.g. detecting phishing, identifying deepfake, warning victims, strengthening fraud controls). These measures should be complemented by greater attention to the upstream organisation and diffusion of criminal capabilities.

  2. Support government, law enforcement, industry and academic researchers to collaborate on identifying and monitoring emerging AI-enabled crime, with greater sharing of threat intelligence and research findings.

    Strengthen research partnerships and information sharing on emerging AI-enabled crime. Government, law enforcement, industry, and academic researchers should be supported to collaborate in identifying, monitoring, and understanding emerging AI-enabled criminal practices. These partnerships are important where risks are developing too quickly to be captured through conventional crime statistics, incident reporting, or established evidence bases. Greater sharing of emerging threat intelligence and research findings would support earlier identification of changes in offender adoption and the diffusion of AI capabilities for cybercrime activities.

  3. Address manipulation of individuals and exploitation of financial infrastructure in the financial-crime response; do not put the whole burden of recognising deception on the public.

    Accordingly, an appropriate response should encompass a focus on both manipulation of individuals and exploitation of financial infrastructure. Public awareness campaigns are useful; however, the responsibility to recognize increasingly convincing deception tactics should not rest entirely on the public.

  4. Develop coordinated guidance for financial institutions and digital platforms on AI-enabled financial crime, including independent verification of identities, documents and payment instructions, support for vulnerable customers, staff training and regular testing.

    there is a need to develop coordinated guidance for financial institutions and digital platforms on AI-enabled financial crimes, including independent verification of identities, documents and payment instructions. Guidance should integrate support for vulnerable customers, staff training and regular testing against evolving impersonation techniques.

  5. Integrate fraud-prevention measures with scrutiny of the bank accounts used to receive and launder criminal proceeds.

    In Australia, an effective response would require fraud-prevention measures to be integrated with scrutiny of the bank accounts employed to receive and launder criminal proceeds.

  6. Strengthen AUSTRAC's Fintel Alliance through dedicated resources and collaboration with the Australian AI Safety Institute on assessing financial crime controls.

    efforts need to be directed towards strengthening AUSTRAC’s Fintel Alliance through dedicated resources and collaboration with the Australian AI Safety Institute for continued assessment of existing financial crime controls.

  7. Independently benchmark GenAI detection tools for performance, false alerts and effects on legitimate customers; use evaluations to guide institutional remedial action, with AUSTRAC publishing annual aggregate findings on gaps and progress.

    GenAI-based detection tools should undergo independent evaluation against established benchmarks, with attention to detection performance, false alerts and impact on legitimate customers. These evaluations should guide remedial action by participating institutions, while AUSTRAC should publish annual aggregate findings outlining capability gaps and progress in addressing them.

  8. Recognise AI-enabled scam compounds as both a cybercrime and a human security challenge.

    Australia should recognise AI-enabled scam compounds as both a cybercrime and human security challenge.

  9. Strengthen intelligence sharing, cybercrime capacity building and law-enforcement cooperation with Southeast Asian partners to detect and disrupt AI-enabled criminal networks.

    Australia should strengthen intelligence sharing, cybercrime capacity building and law-enforcement cooperation with Southeast Asian partners to improve detection and disruption of AI-enabled criminal networks.

  10. Government, academia and industry to support research partnerships, information sharing and systematic monitoring of AI-enabled scam compounds and cyber-fraud ecosystems.

    Government, academia and industry should support research partnerships, information sharing and systematic monitoring of AI-enabled scam compounds and cyber-fraud ecosystems.

  11. Evaluate AI roleplay boundaries by distinguishing consensual adult fiction from unwanted sexual content, disregard for expressed boundaries and encouragement of real-world harm, with particular attention to children and vulnerable users.

    The policy concern is the adequacy of boundaries governing the AI’s participation in an unfolding interaction. Evaluation should distinguish consensual adult fiction from unwanted sexual content, disregard for expressed boundaries and encouragement of real-world harm. Particular attention is required where children or vulnerable users may encounter content or interactions that available safeguards fail to address.

  12. Support independent evaluation of AI roleplay safety controls across sustained interactions, including effectiveness as narratives develop, respect for user refusals and responses to requests encouraging real-world harm.

    1. Australia should support independent evaluation of safety controls within AI roleplay applications. Assessment should examine sustained interactions, including whether safeguards remain effective as narratives develop, whether the system respects user refusals, and how it responds to requests encouraging real-world harm.

Read the original

Open in the interactive guide