AI Inquiry Atlas

An unofficial, independent guide to the public records of the Joint Select Committee on Artificial Intelligence. Not connected with the Parliament of Australia or the committee. Its makers also made a submission (no. 481; see About). Summaries and scores are written by AI and can be wrong: check the original before relying on them.

Submission 243

Dr Evgenii Arsentev

Submitter group: Individuals.

Position labels are not shown for individual submitters.

An AI model identified the submitter group and wrote the summary, key points and recommendation summaries. Counts do not measure importance or merit.

Open in the interactive guide

Summary

Read the original submission for the full text.

Describes an operator's measurements of AI agent costs and accountability, as an individual not resident in Australia with no commercial interest here, noting the cited reports are not peer reviewed. Recommends per-run cost telemetry and ceilings for SME programs and procurement, machine-readable run records for Commonwealth AI use, and evaluation of deployed systems rather than models.

Key points

Quotes are the submitter’s own words. Text taken from documents, including scanned pages read by text-recognition software, can differ from the PDF. Check the original. Listed in the same order as on the submission’s page in the interactive guide.

  1. Says most modelled agent cost in the author's measured sessions went to conversation context rather than generating text.

    Across 722 agent sessions comprising 150,902 model calls and 34.6 billion tokens, between 86 and 88 per cent of modelled cost went on handling the conversation context — reading the accumulated context back in and writing it to cache — rather than on generating new text.

  2. Says about 3 per cent of sessions in the author's dataset accounted for 80 per cent of total cost.

    Spend is not distributed anything like usage. In the same dataset, about 3 per cent of sessions accounted for 80 per cent of total cost, and the 8 per cent of sessions that ran longer than 200 model calls accounted for more than 90 per cent of the money.

  3. Says 52 per cent of model calls in the author's measurements were made by sub-agents and absent from top-level transcripts.

    In my own measurements, 52 per cent of model calls were made by sub-agents: work that the agent delegated to copies of itself. None of that work appears in the top-level transcript that a user, an auditor or a regulator would normally be shown.

  4. Argues that a conversation transcript is not an adequate audit record of an AI system that acts.

    An AI system that answers questions can be held to account by reading its answers. An AI system that acts — that calls tools, writes to systems, sends messages, moves money — cannot. What an operator can produce after the fact is a conversation transcript, and a transcript is not an audit record.

  5. Says that evaluation happens on the model while harm happens in the workflow.

    Evaluation happens on the model; harm happens in the workflow. A system that scores well in isolation can still mislead because of what it was given to read, what it was allowed to do, and what the person on the other end believed about it.

  6. Says that if Australia builds sovereign capability on open components, licence provenance and a retained copy of what was used are the difference between a capability and a dependency that can disappear.

    If Australia intends to build sovereign capability on open components, licence provenance and a retained copy of what was actually used are not administrative details; they are the difference between a capability and a dependency that can disappear.

Recommendations

Recommendations are shown beside the submission’s own words. Listed in the same order as on its page in the interactive guide.

  1. Disregard superseded figures in the author's first extension-request email and rely only on versions cited in this submission.

    Correction to my earlier correspondence. My email of 15 September 2026 requesting this extension quoted two figures — 83.5 per cent and 92.2 per cent — from version 1.0 of my first report. Those figures were modelled on provider list prices that have since changed, and I recomputed them against the current official prices. Version 1.0 is superseded. I ask the committee to disregard the numbers in that first email and to rely only on the versions cited in this submission.

  2. Require Commonwealth programmes funding SME AI adoption to fund instrumentation before pilots, with recipients reporting per-run cost and call-count telemetry as a condition.

    Any Commonwealth programme that funds AI adoption by small and medium businesses should fund instrumentation before it funds pilots, and should require recipients to report per-run cost and call-count telemetry as a condition.

  3. Publish small-business guidance on agent-run costs based on measured overhead versus output and spend concentration in long sessions, rather than vendor token prices.

    Publish plain-language guidance for small business on what an agent run actually costs, built on the measured structure of that cost — overhead against output, and the concentration of spend in long sessions — rather than on vendor per-token list prices, which describe almost none of the real bill.

  4. Where the Commonwealth procures AI services, make a default, customer-visible per-run cost and call ceiling a procurement requirement.

    Where the Commonwealth procures AI services, make a default, customer-visible per-run cost and call ceiling a procurement requirement.

  5. Include model identities, versions and timestamps; tool arguments and outcomes; sub-agent calls; token and cost accounting; individual check results; and human authorisation steps in run records.

    • the identity and version of each model used, with a timestamp — vendors update models continuously, and a review of a past decision against a model that has since silently changed reviews nothing; • which tools were invoked, with their arguments and their outcomes; • which calls were made by sub-agents rather than by the top-level agent, so that the invisible half of the work is on the record; • token and cost accounting for the run; • the checks that were run, with a result recorded for each check — not a single aggregate pass or fail, which discards exactly the detail an auditor needs; • what human authorisation, if any, the run had, and at which step.

  6. Require machine-readable per-run records for Commonwealth AI use instead of policies or checklists, capturing model version, tool actions and authorisation.

    For Commonwealth use of AI, require a machine-readable per-run record rather than a policy statement or a self-assessment checklist. A published policy describes an intention; a run record is evidence. The three facts that matter most — which model version acted, what it did through tools, and who or what authorised it — cost essentially nothing to capture at the time and are impossible to reconstruct afterwards.

  7. Prefer an open, interoperable schema for the run record over an agency-specific one, and have Australian agencies participate in standards work at the IETF and W3C.

    Prefer an open, interoperable schema for that record over an agency-specific one, and have Australian agencies participate in the standards work where it is happening (IETF and W3C)

  8. Evaluate the deployed system rather than the model, and publish agency evaluations with per-check results rather than a headline score.

    Evaluate the deployed system, not the model. Benchmark scores describe a model in isolation; what fails in production is the assembly around it — the retrieval, the prompt, the tools, the escalation path. Agency evaluations should be published with per-check results rather than a headline score.

  9. When checks are needed and budgets limited, use a second independent AI instructed to disprove the first, as a practitioner's method that does not replace consequential human review.

    Where a check is needed and budget is short, a second, independent AI system instructed to disprove the first is a cheap and genuinely useful control: independent systems rarely fail in the same way, and the disagreement itself marks where to look. I offer this as a practitioner’s method, not as a substitute for human review of anything consequential.

  10. For clinically-relevant AI tasks, suggests each should have a check the system can run that produces a recorded result.

    every clinically-relevant task should have a check that the system itself can run and that produces a recorded result

  11. For clinically-relevant AI output, suggests recording the model version behind it at the time.

    the model version behind any clinically-relevant output should be recorded at the time

Read the original

Open in the interactive guide