Submission 65
Institute for Integrated Economic Research - Australia
Submitter group: Research institutes.
An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.
Summary
Read the original submission for the full text.
Argues that cumulative cognitive and ethical effects of AI and foreign-model dependency require a national-security response. Proposes augmented intelligence, retained-capability measures, oversight and sovereign capability, with further measures in its attached report. Describes cognitive decline as a risk hypothesis, not an inevitable outcome; workshop participants need not agree with every statement. Says its concerns arise from using AI, not opposing it.
Key points
Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.
Argues that Australia does not measure or assign responsibility for AI’s cumulative effects on analytical and ethical judgement.
None of them addresses what AI does to the analytical and ethical judgement of the people and institutions using it. That effect is cumulative, population-wide and invisible until well advanced. It is not a category of harm that Australia currently measures, assigns to anyone, or has any means of detecting.
Argues that the listed adoption measures track uptake, skills and ecosystem growth, not whether AI users retain the ability to evaluate outputs.
Australian Bureau of Statistics data, Jobs and Skills Australia reports, the National AI Centre’s AI Adoption Tracker and the National AI Ecosystem Report. Every one of these measures uptake, skills acquisition or ecosystem growth. Not one measures whether the people using AI retain the ability to evaluate what it produces.
Argues that Australia lacks an operational definition of meaningful human oversight, a requirement for documented reviewable automation settings and tests of whether oversight is real.
Australia currently has no operational definition of meaningful human oversight, no requirement that the automation setting for a given function be a documented and reviewable decision, and no means of testing whether an oversight arrangement is real or nominal.
Argues that cognitive decline is a national security vulnerability that influence operations could exploit.
A population less able to detect manipulation is a population more exposed to it. Cognitive atrophy and cognitive warfare are not separate concerns; the first is the vulnerability that the second exploits.
Describes US export restrictions on two frontier models, with access subsequently restored in part on US terms, as demonstrating foreign-model dependency.
A United States export control directive required the suspension of access to two frontier models by all foreign nationals, and the provider disabled them for all customers in order to comply. Access was subsequently restored in part, at Washington’s discretion and on Washington’s terms,
Argues that the advisory AI Safety Institute’s remit does not reach cumulative cognitive effects on the Australian population.
The Australian AI Safety Institute is an advisory body, not a regulator; enforcement remains with existing sector regulators. Its remit covers upstream model capability risks and downstream harms as experienced by users of a system. Neither limb reaches the cumulative effect of AI deployment on the judgement of the Australian population
Recommendations
Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.
Make strengthening human judgement a public-sector AI procurement and deployment principle, reflected in the APS AI Plan and procurement guidance, recording each deployment’s design intent rather than inheriting it from tool selection.
That the Government adopt, as an explicit principle governing public sector AI procurement and deployment, that systems are to be designed to strengthen human judgement rather than replace it. This should be reflected in the AI Plan for the Australian Public Service and in procurement guidance, so that the design intent of each deployment is a recorded decision rather than a by- product of the tool selected.
Give the Office of AI explicit carriage of cognitive security, with a named senior officer accountable for it, reported to Parliament.
That the Office of AI be given explicit carriage of cognitive security, meaning the effects of AI deployment on the analytical and ethical judgement of the Australian population and workforce, with a named senior officer accountable for it, and that this be reported to Parliament.
Extend the measurement framework for the National AI Plan and the APS AI Plan to include indicators of retained analytical capability, so agencies report evidence that decision-makers can still evaluate AI output.
That the measurement framework for the National AI Plan and the AI Plan for the APS be extended to include indicators of retained analytical capability, so that agencies report not only how much AI they use but what evidence they hold that their decision-makers remain able to evaluate its output.
Have the Australian Standards for AI incorporate cognitive impact and define meaningful human oversight operationally; make the automation setting for each significant public-sector decision function a documented, disclosed, reviewable decision rather than a vendor default.
That the Australian Standards for AI incorporate cognitive impact as a design consideration and define meaningful human oversight operationally, specifying the conditions under which a person is positioned to interrogate, override and take responsibility for a system’s output; and that the automation setting for each significant public-sector decision function be a documented, disclosed and reviewable decision rather than a default inherited from the vendor.
Extend AI literacy programmes funded under the National AI Plan from competent use of AI to active maintenance of independent analytical capability.
That the AI literacy programmes already funded under the National AI Plan be extended from competent use of AI systems to the active maintenance of independent analytical capability, so that the programme which accelerates adoption is matched by one that sustains the judgement adoption depends upon.
Extend the AI Safety Institute's remit to population-level and institutional cognitive effects, including the Commonwealth's own adoption programme, and have the Committee examine whether its placement affords sufficient independence.
That the Australian AI Safety Institute’s remit be extended to cover population-level and institutional cognitive effects, including the Commonwealth’s own adoption programme, and that the Committee examine whether the Institute’s placement within the portfolio responsible for AI adoption and investment attraction affords it sufficient independence.
Commission a national risk assessment of AI's systemic and cumulative effects, including cognitive and sovereign dependency, as the evidentiary foundation for the Australian Standards for AI and a National Security Strategy.
That the Government commission a national risk assessment addressing AI’s systemic and cumulative effects, including cognitive and sovereign dependency, to serve as the evidentiary foundation for the Australian Standards for AI and for a National Security Strategy.
Account for differential vulnerability in policy responses rather than assuming uniform exposure.
Policy responses must account for this differential vulnerability rather than assuming uniform exposure.
Maintain humans in the loop for high-stakes military, strategic-policy and crisis decisions, with sufficient cognitive capacity to override AI when context demands.
Critical decision-making: High-stakes decisions (military operations, strategic policy, crisis response) must maintain human-in-loop with sufficient cognitive capacity to override AI recommendations when context demands.
Have defence and intelligence organisations train analysts to function without AI, recognising adversaries will target dependencies.
Skills maintenance: Defence and intelligence organisations should implement training programs that maintain the analysts’ ability to function without AI support, recognizing that adversaries will target AI dependencies.
Require independent human analysis of complex AI-generated intelligence before operational decisions, documenting reasons for agreement or disagreement.
Verification requirements: Complex AI-generated intelligence assessments should require independent human analysis before informing operational decisions, with documented reasoning that explains agreement or disagreement with AI conclusions.
Take a whole-of-government approach to cognitive-warfare research and policymaking.
Because Cognitive Warfare poses significant challenges to both military operators and the general populace, it demands a whole-of-government approach to research and policymaking.
Ensure Australian AI systems reflect domestic legal frameworks, cultural norms and national-security requirements.
Australian systems must reflect Australian legal frameworks, cultural norms, and national security requirements.
Have domestic institutions steer AI toward national objectives rather than imported regulatory or corporate defaults.
Domestic institutions must steer AI toward national objectives, not imported defaults shaped by foreign regulatory environments and corporate incentives.
Consider widening the AI Safety Institute’s cognitive-risk perimeter and funding it accordingly.
The Institute is a well-conceived and well-led first step within an unchanged conceptual perimeter. Widening that perimeter, and funding it accordingly, is the opportunity.
Consider system-prompt regulation in Australian government, education and high-stakes professional settings, using prompts or forcing functions that promote engagement, challenge assumptions and require evidence-based reasoning.
This agentic layer could also enable system prompt regulation, a novel policy intervention proposed during workshop discussions. System prompts guide model behaviour, and it is technically straightforward to configure them to encourage critical thinking rather than passive consumption. Regulation could require that AI systems deployed in Australian government, education, and high-stakes professional settings include system prompts / forcing functions that promote cognitive engagement, challenge user assumptions, and require evidence-based reasoning.
Treat cognitive engagement as a system-level design and governance requirement across AI systems people use; adaptive interfaces should avoid becoming instruments for monitoring or manipulating cognition.
However, cognitive engagement cannot be secured tool-by-tool but should be treated as a system-level design and governance requirement across the AI systems a population actually uses. Such a capability could, however, raise significant governance questions of its own; any system that adapts to a user's cognitive capability must necessarily assess it, which carries clear risks of surveillance, profiling and paternalistic control, and the design challenge is to deliver age- and capability-appropriate interaction without creating a new instrument for monitoring or manipulating the very cognition it is meant to protect.
Use a viral change approach that inserts characteristics useful across plausible futures through leadership, education, cultural change, processes and standards, with ongoing reassessment.
Rather than specifying an end state, the approach proposes the characteristics a future system will need. The test of any candidate characteristic is that it remains useful across a wide range of plausible futures, rather than being optimised for any single one. Those characteristics are then inserted into the current operating model through leadership behaviour, education, active cultural change, and the design of processes and standards. The approach requires ongoing reassessment and adjustment as the effects of those changes become evident.
Design to preserve the ability to slow down and withhold decisions pending human judgement.
The capacity to deliberately slow down, to withhold a decision pending human judgement, is the single characteristic most at risk from the dynamics described in this report, and the one most worth designing to protect.
Implement cognitive-maintenance protocols for agentic-AI users, including regular AI-free analysis, independent verification of complex outputs and rotation through manual analytical roles.
5. Need for Cognitive Maintenance Protocols: Organisations deploying agentic AI systems should implement formal protocols to maintain human cognitive capacity, including regular “AI-free” analysis exercises, mandatory independent verification of complex AI outputs, and rotation of team members through manual analytical roles.
Consider adapting prepare, detect, respond and recover as a response framework; avoid blanket takedowns, bans and platform-wide blocks because suppression risks legitimacy loss and migration to darker channels.
The COVID-19 response was based on four pillars of public health: Prepare, Detect, Respond, and Recover. We could possibly adapt this framework while avoiding problematic analogues. Example actions are detailed below. We should avoid transplanting “hard lockdown” analogues (blanket takedowns or bans) or “border closures” (platform-wide blocks), as widespread content suppression would risk legitimacy loss and migration to darker channels.
Consider a vaccination analogy for preparing citizens and families for AI impacts, and an antiviral analogy for helping people experiencing cognitive atrophy from unintelligent AI use.
A further pandemic analogy that could be worth considering is that of vaccines and antivirals. A “vaccination” model to educate and prepare our citizens for the impact of AI on them and their families, and an ‘anti-viral” to aid individuals who have experienced cognitive atrophy resulting from unintelligent use of AI systems.
Create formal, auditable information-resilience capability baselines across governments, major technology platforms and AI deployers, adapting global health regulations.
Establish capability baselines: By adapting existing global health regulations (such as WHO’s International Health Regulations and Joint External Evaluations), create formal, auditable capability baselines for “information resilience” across governments, major technology platforms, and AI deployers.
Mandate content-provenance standards for ABC/SBS content by Q4 2026, extending to all public-sector media and newsrooms.
Mandate Content Provenance standards for all ABC/SBS content by Q4 2026, extending to all public-sector media and newsrooms to harden authenticity.
Pre-position risk-communication and community-engagement networks through libraries, schools and community centres to build trust before crises.
Pre-position Risk Communication and Community Engagement networks through libraries, schools, and community centres to build trust before a crisis.
Integrate information-literacy curricula from primary to tertiary education, emphasising detection of manipulation tactics and synthetic media.
Develop information literacy curricula integrated into primary through tertiary education, emphasizing detection of manipulation tactics and synthetic media.
Support Australian AI aligned with domestic legal and ethical frameworks, reducing foreign-platform dependence and incorporating regional, rural and Indigenous contexts rather than solely urban data.
Investment in sovereign capability: Support development of Australian AI systems aligned with domestic legal and ethical frameworks, reducing dependency on foreign platforms with misaligned incentives. Ensure these systems are developed with awareness of regional, rural, and Indigenous Australian contexts; not solely urban data environments.
Create real-time dashboards to detect synthetic-media spikes, coordinated inauthentic behaviour and narrative-manipulation campaigns.
Stand up monitoring infrastructure: Create real-time information surveillance dashboards to monitor early signals like synthetic media spikes, coordinated inauthentic behaviour, and narrative manipulation campaigns.
Establish an ASIO information-surveillance dashboard tracking deepfakes, bot networks and coordinated disinformation, with transparent parliamentary-oversight reporting.
Establish ASIO Information Surveillance Dashboard monitoring deepfake spikes, bot networks, and coordinated disinformation campaigns with transparent reporting to parliamentary oversight committees.
Partner with universities and civil society on independent monitoring to avoid concentrating surveillance power.
Partner with universities and civil society to conduct independent monitoring, avoiding concentration of surveillance power.
Require major platforms to report detection of large-scale manipulation campaigns.
Implement mandatory incident reporting by major platforms for detection of large-scale manipulation campaigns.
Use platform friction to slow viral spread without blocking lawful speech: cap mass forwards, require reading or fact-check acknowledgements before sharing, and throttle amplification during detected manipulation campaigns without removing content.
Information distancing (friction): Implement platform-level friction mechanisms that slow viral spread without blocking lawful speech. Example actions include: • Cap mass forwards on messaging platforms. • Implement “speed bumps” requiring users to read articles before sharing, or to confirm they’ve viewed fact-check warnings.. • Throttle amplification algorithms during detected manipulation campaigns, reducing algorithmic boost without removing content
Prioritise prebunking over debunking; consider short inoculation videos to reduce susceptibility to known manipulation tactics.
Cognitive vaccination (prebunking): Prioritize prebunking over debunking. Prebunking acts as cognitive vaccination and could be more effective than brittle, after-the-fact deepfake detection. Use short “inoculation” videos to reduce public susceptibility to known manipulation tactics could assist.
Launch public education campaigns on common deepfake tells, manipulation tactics and verification techniques.
Launching public education campaigns showing common deepfake tells, manipulation tactics, and verification techniques.
Partner with social-media platforms to deliver prebunking before major elections or crises; integrate prebunking into school digital-literacy curricula.
Partnering with social media platforms to deliver prebunking content before major elections or crisis events. • Integrating prebunking modules into school curricula as part of digital literacy education.
For populations with a generational trust gap, consider interventions showing practical financial, employment, health or social consequences of unreliable information rather than assuming intrinsic motivation to distinguish truth.
For populations exhibiting a generational trust gap, consider developing interventions that demonstrate practical consequences of information unreliability in domains these cohorts care about (financial, employment, health, social outcomes) rather than assuming intrinsic motivation to distinguish truth from falsehood.
After acute manipulation campaigns, rebuild trust through transparent post-incident reviews, local journalism and fact-checking investment, and support for Australian AI aligned with democratic values and transparency.
Institutional strengthening: After acute manipulation campaigns, invest in rebuilding trust in democratic institutions and information ecosystems. Example actions include: • Conducting post-incident reviews following major manipulation campaigns, publishing findings transparently. • Investing in local journalism and fact-checking capacity to provide trusted information alternatives. • Supporting development of Australian AI systems that align with democratic values and transparency requirements.
For knowledge work, require manual synthesis before accepting AI summaries, periodic AI-free analysis days, and substantial human refinement of AI starting points.
For knowledge work: o Require manual synthesis before accepting AI-generated summaries. o Implement periodic “AI-free” analysis days to maintain baseline skills. o Use AI outputs as starting points requiring substantial human refinement, not final products.
For decision-making, require human review with documented reasoning, rotation through manual analytical roles, and regular exercises without AI to test reversionary capability.
For decision-making: o Mandate human review of AI recommendations with documented reasoning. o Rotate team members through manual analytical roles to maintain skills. o Conduct regular exercises where AI systems are unavailable to test reversionary capability.
For education, prohibit AI for foundational skills until mastery is demonstrated, require manual problem-solving before comparison with AI, and teach AI limitations and failure modes.
For education: o Prohibit AI use for foundational skill development, allowing it only after mastery demonstrated. o Require students to solve problems manually before comparing with AI approaches. o Integrate AI literacy education that emphasizes limitations and failure modes.
Support C2PA content provenance across public-sector media and major platforms.
Champion C2PA standards adoption: Support content provenance requirements across public-sector media and major platforms.
Advocate Australian AI development aligned with national security and democratic values, explicitly including regional and rural contexts.
Invest in sovereign capability: Advocate for Australian AI development aligned with national security requirements and democratic values, with explicit attention to regional and rural contexts.
Require major AI deployments to document cognitive-impact assessments and mitigation strategies.
Mandate transparency: Require major AI deployments to document cognitive impact assessments and mitigation strategies.