AI Inquiry Atlas

An unofficial, independent guide to the public records of the Joint Select Committee on Artificial Intelligence. Not connected with the Parliament of Australia or the committee. Its makers also made a submission (no. 481; see About). Summaries and scores are written by AI and can be wrong: check the original before relying on them.

Submission 185

NCC Group

Submitter group: Companies.

An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.

Open in the interactive guide

Summary

Read the original submission for the full text.

NCC Group, a cyber security firm, argues that Australia has substantial AI governance machinery and now needs operationalisation and independent technical assurance. It addresses sovereignty as strategic autonomy, supply-chain dependencies, agentic AI security, AI-enabled fraud and the AI Safety Institute, favouring existing structures and targeted gap-filling.

Key points

Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.

  1. Argues that Australia does not primarily lack AI principles or governance bodies and that the next challenge is operationalisation and assurance.

    NCC Group’s central proposition is therefore that Australia does not primarily lack AI principles or governance bodies. The next challenge is operationalisation and assurance: demonstrating that AI systems remain secure and controllable when models are connected to real identities, data, permissions, APIs, tools, cloud infrastructure and automated decision pathways.

  2. Argues that sovereign AI capability should be understood more broadly than data residency.

    Sovereign AI capability should be considered more broadly than whether data is physically stored in Australia.

  3. Argues that governance review of higher-impact systems should be complemented, where appropriate, by independent technical testing of safeguards.

    Expert review can determine whether proposed safeguards and governance arrangements are appropriate; it does not demonstrate that those controls withstand realistic attack, misuse or failure. For higher-impact systems, governance review should therefore be complemented where appropriate by independent technical testing.

  4. Argues that fraud and deepfake resilience should not depend on individuals identifying synthetic content.

    Fraud and deepfake resilience should not depend on an individual correctly identifying synthetic content.

  5. Argues that agentic AI security assurance must cover the whole operational system, not only the model.

    security assurance must extend beyond the underlying model to the software, identities, permissions, data, tools and systems through which an AI agent acts.

Recommendations

Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.

  1. Build on existing AI governance and assurance mechanisms rather than creating additional coordination structures, with clear escalation pathways between agency governance, the AI Review Committee, technical assurance, ASD, the AI Safety Institute and regulators.

    Build on Australia’s existing AI governance and assurance mechanisms rather than creating additional coordination structures, with clear escalation pathways between agency governance, the AI Review Committee, technical assurance, ASD, the AI Safety Institute and regulators.

  2. Translate AI principles and technical standards into testable operational controls, with proportionate independent technical assurance for higher-impact systems.

    Translate AI principles and technical standards into testable operational controls, with proportionate independent technical assurance for higher-impact systems.

  3. Use Commonwealth procurement and critical infrastructure risk management to address material AI dependencies, including supply-chain concentration, FOCI, foreign-jurisdiction exposure, portability, continuity and exit.

    Use Commonwealth procurement and critical infrastructure risk management to address material AI dependencies, including supply-chain concentration, FOCI, foreign-jurisdiction exposure, portability, continuity and exit.

  4. Treat AI sovereignty as a resilience and strategic-autonomy issue, not simply data residency, including understanding where foreign legal, technological and supply-chain dependencies can affect access to critical AI capability.

    Treat AI sovereignty as a resilience and strategic-autonomy issue, not simply data residency. Australia should understand where foreign legal, technological and supply-chain dependencies can affect continued access to critical AI capability.

  5. Retain sufficient Australian technical capability and alternatives to independently evaluate and continue operating nationally significant AI systems where international dependencies are disrupted or constrained.

    Retain sufficient Australian technical capability and alternatives to independently evaluate and continue operating nationally significant AI systems where international dependencies are disrupted or constrained.

  6. Consider foreign-jurisdiction and export-control exposure as economic-resilience and strategic-autonomy risks, beyond compliance.

    Australia should therefore consider foreign-jurisdiction and export-control exposure as economic-resilience and strategic- autonomy risks, not merely compliance issues.

  7. Assess enduring access, operation, assurance, migration and replacement in procurement and assurance for nationally significant AI.

    For nationally significant AI, procurement and assurance should therefore consider Australia’s enduring ability to access, operate, assure, migrate from and replace critical technology.

  8. Pursue sovereign capability where it materially improves strategic resilience; deliberately accept understood international dependencies with credible alternatives.

    Australian sovereign capability should be pursued where it materially improves strategic resilience, while international dependencies should be understood, deliberately accepted and supported by credible alternatives.

  9. Make clear when a matter stays within agency accountability, needs AI Review Committee scrutiny, warrants independent technical testing, or is escalated to ASD, the AI Safety Institute, a regulator or national-security authorities.

    Clear pathways should determine when a matter remains within agency accountability, requires AI Review Committee scrutiny, warrants independent technical testing, or should be escalated to ASD, the AI Safety Institute, a regulator or national-security authorities.

  10. Complement governance review of higher-impact systems with independent technical testing where appropriate.

    For higher-impact systems, governance review should therefore be complemented where appropriate by independent technical testing.

  11. Have Commonwealth procurement require, in proportion to capability, autonomy, access and consequence, dependency transparency, identity and access controls, human oversight, auditability, incident response, FOCI and jurisdictional risk, portability, continuity and independent assurance.

    Commonwealth procurement should reinforce this architecture through proportionate requirements for dependency transparency, identity and access, human oversight, auditability, incident response, FOCI and jurisdictional risk, portability, continuity and independent assurance. Requirements should scale with capability, autonomy, access and consequence.

  12. Conduct a structured gap analysis of existing regimes against real AI use cases, testing whether responsibility and preventative, monitoring, incident-response and redress obligations are adequate for high-impact AI.

    The appropriate task is a structured gap analysis against real AI use cases, testing whether existing regimes clearly allocate responsibility and provide adequate preventative, monitoring, incident-response and redress obligations for high-impact AI.

  13. Consider targeted statutory duties where material gaps cannot be addressed through existing legislation, guidance, standards or sector obligations, proportionate to capability, autonomy, access, reach and consequence.

    Where material gaps cannot be addressed through existing legislation, regulatory guidance, standards or sector-specific obligations, targeted statutory duties may be warranted, proportionate to capability, autonomy, access, reach and consequence.

  14. Prioritise clarity about how existing bodies interact (Office of AI, DTA, AI Review Committee, AI Safety Institute, ASD, sector regulators) rather than creating additional coordination structures.

    The priority should be clarity about how existing bodies interact, including the Office of AI, DTA, AI Review Committee, AI Safety Institute, ASD and relevant sector regulators, rather than creating additional coordination structures.

  15. Have higher-risk developers and deployers document purpose, limitations, data provenance, evaluations, security, oversight, dependencies and monitoring; share appropriate information while protecting personal, security-sensitive and legitimately proprietary information.

    Developers and deployers of higher-risk systems should maintain proportionate technical documentation covering intended purpose, material limitations, data provenance, evaluation results, security controls, human oversight, material supply-chain dependencies and post-deployment monitoring. Appropriate information should be available to regulators, independent assessors and affected parties while protecting personal, security-sensitive and legitimately proprietary information.

  16. Treat AI-enabled fraud as a shared control problem across financial institutions, telecommunications providers, digital platforms, identity systems and government, rather than primarily content detection.

    AI-enabled fraud should be treated as a shared control problem across financial institutions, telecommunications providers, digital platforms, identity systems and government, rather than primarily as a content-detection problem.

  17. Use trusted out-of-band or multi-channel verification for sensitive actions such as changing payment details, granting high-value account access or releasing sensitive information.

    Sensitive actions, such as changing payment details, granting high-value account access or releasing sensitive information, should use trusted out-of-band or multi-channel verification.

  18. Avoid treating absence of content provenance as proof of malicious content.

    Content provenance and authenticity mechanisms can support trust, but the absence of provenance should not itself establish that content is malicious.

  19. Require pre-deployment and continuous abuse testing covering impersonation, fraud, coercion, manipulation of children and vulnerable users, safeguard evasion, prompt injection and misuse of connected tools.

    Pre-deployment and continuous abuse testing should cover impersonation, fraud, coercion, manipulation of children and vulnerable users, evasion of safeguards, prompt injection and misuse of connected tools.

  20. Have higher-impact AI deployments use separate machine identities, least-privilege permissions, segmentation where appropriate, short-lived credentials, human approval for high-consequence actions, logging, credential revocation and containment and shutdown mechanisms.

    Higher-impact deployments should use separate machine identities, least-privilege permissions, segmented or isolated execution where appropriate, short-lived credentials, human approval for high- consequence actions, comprehensive logging, rapid credential revocation and reliable containment and shutdown mechanisms.

  21. Test the whole operational system: model, prompts, data, permissions, tools, integrations, identities and cloud environment.

    Assurance should test the model, prompts, data, permissions, tools, integrations, identities and cloud environment as one operational system.

  22. Have organisations supporting important government, financial, national-security or critical-infrastructure functions demonstrate through independent technical testing that controls withstand realistic attack, misuse and failure.

    For systems supporting important government, financial, national- security or critical-infrastructure functions, organisations should be able to demonstrate through independent technical testing that these controls withstand realistic attack, misuse and failure scenarios.

  23. Build AI supply-chain assurance on ASD's existing cyber supply-chain approach and Home Affairs' technology-vendor mechanisms.

    AI supply-chain assurance should build on ASD’s existing cyber supply-chain approach and Home Affairs’ technology-vendor mechanisms.

  24. Record higher-impact deployments’ material dependencies: model/cloud providers, datasets, libraries, plugins, APIs, tools, processing locations, administrative access, ownership/control concerns, foreign-jurisdiction exposure, incident notification, portability and exit.

    Higher-impact deployments should maintain an actionable record of material dependencies, including model and cloud providers, datasets, libraries, plugins, APIs, external tools, processing locations, administrative access, ownership or control concerns, foreign- jurisdiction exposure, incident-notification terms, portability and exit arrangements.

  25. Consider material AI dependencies within existing SOCI Act obligations where they materially affect critical assets’ and services’ confidentiality, integrity, availability or continuity.

    For entities subject to the SOCI Act, material AI dependencies should be considered within existing critical-infrastructure risk-management obligations where they can materially affect the confidentiality, integrity, availability or continuity of critical assets and services.

  26. Have SOCI Act operators test supplier compromise, model unavailability, corrupted data, unsafe automated action, loss of connectivity, concentration failure, export restrictions and sudden service withdrawal.

    For entities subject to the SOCI Act, material AI dependencies should be considered within existing critical-infrastructure risk-management obligations where they can materially affect the confidentiality, integrity, availability or continuity of critical assets and services. Operators should test supplier compromise, model unavailability, corrupted data, unsafe automated action, loss of connectivity, concentration failure, export restrictions and sudden service withdrawal.

  27. Address physical and functional-safety consequences in assurance where AI interacts with operational technology.

    Where AI interacts with operational technology, assurance must also address physical and functional- safety consequences.

  28. Expand intelligence-led threat testing and joint government-industry exercises for high-impact AI supporting critical services, including scenarios involving loss or restriction of external dependencies.

    Intelligence-led threat testing and joint government-industry exercises should be expanded for high- impact AI supporting critical services, including scenarios involving the loss or restriction of material external dependencies.

  29. Have the AI Safety Institute complement ASD, APS assurance and sector regulators through independent capability and risk evaluation, Australian translation of international testing, and identification of risks requiring regulatory, cyber or national-security action.

    Australia’s AI Safety Institute should complement the technical and operational responsibilities of ASD, APS assurance mechanisms and sector regulators. Its distinctive value should include independent evaluation of emerging capabilities and risks, translation of international testing into the Australian context, and identification of risks requiring regulatory, cyber security or national-security action.

  30. Include autonomy, long-horizon tasks, cyber capability, prompt injection, tool manipulation, data leakage, containment/shutdown, multi-agent behaviour, fraud/impersonation and Australian linguistic, cultural and operational performance in AI Safety Institute evaluation.

    Relevant evaluation should include autonomy and long-horizon task completion, cyber capability, prompt injection and tool manipulation, data leakage, containment and shutdown controls, multi-agent behaviour, fraud and impersonation risks, and performance in Australian linguistic, cultural and operational contexts.

  31. Distinguish model failures from permissions, integrations, credentials, containment, monitoring, oversight and governance failures in incident and near-miss lessons; feed lessons into standards, ISM controls, procurement and regulatory guidance.

    Lessons from significant incidents and near misses should distinguish model failures from failures of permissions, integrations, credentials, containment, monitoring, human oversight and organisational governance, and should feed back into standards, ISM controls, procurement and regulatory guidance.

  32. Use Five Eyes, AUKUS and international AI measurement collaboration to reduce duplication and enable shared testing and timely risk information.

    International collaboration through Five Eyes, AUKUS and the International Network for Advanced AI Measurement, Evaluation and Science should reduce unnecessary duplication and enable shared testing and timely risk information.

  33. Retain sufficient domestic technical capability to independently validate international findings and assess AI systems deployed in Australian government, defence and critical infrastructure.

    Australia must retain sufficient domestic technical capability to independently validate international findings and assess systems deployed in Australian government, defence and critical infrastructure.

Read the original

Open in the interactive guide