Measure AI cost per accepted task, not just token price.
A practical AI evaluation method for including retries, human review and rework in cost per accepted task. Use matched workloads and explicit acceptance gates.
Reviewed by Aeon AI Risk Management.
The short answer
Cost per accepted task is total evaluation or operating cost divided by the number of outputs that pass a predeclared acceptance gate. Include failed attempts and human review. Cheap tokens can still produce expensive outcomes.
A reproducible evaluation ledger
| Area | Evidence or method | Your verification step |
|---|---|---|
| Define acceptance first | Use the same bounded task set, input data, tools and acceptance criteria for each candidate. [1] | Record correctness, security and latency requirements before running the comparison; do not move the gate after seeing results. |
| Count all attempts | Include rejected outputs, retries, tool calls and reruns in the cost numerator. [1] | Record task ID, attempt count, usage and actual billed cost. Do not equate a subscription allowance with metered API pricing. |
| Include review and rework | Add reviewer hours at a stated rate, correction cost and allocated infrastructure or subscription cost. [1] | Keep assumptions explicit and report both direct provider cost and fully loaded cost so readers can reproduce the comparison. |
| Report the denominator | Divide the total by accepted outcomes, not by all attempted tasks. [1] | If none are accepted, report no accepted outcomes rather than a zero unit cost. Publish sample size, exclusions and evidence status. |
Practical decision checklist
- Write the acceptance rubric and run a small matched pilot using non-sensitive fixtures.
- Record actual usage, billable tool costs, review time and correction time for every attempt.
- Formula: (provider + tools + review + rework + allocated infrastructure cost) / accepted tasks.
- Illustrative arithmetic, not a benchmark: $120 total cost / 8 accepted tasks = $15 per accepted task. Attempt count alone is not the denominator.
- Compare the result with task quality, latency and security constraints. Repeat before extrapolating to production volume.
Sources and review method
We reviewed the linked public material on the date above and separated vendor claims from our recommended checks. We did not audit the vendor, test your account or independently verify every contractual control. Confirm current terms before making a decision.
Common questions
Does the Model Economics Lab measure accepted production outcomes?
No. Its sourced benchmark operating points and evidence tiers are inputs to model selection, not a measured result for your workflow. Run an acceptance-gated evaluation before making a business case.
Turn the review into a decision.
Aeon is a Canadian AI consulting firm working across confidential-data controls, private AI, security assessments and implementation. We scope the actual workflow and produce practical correction and decision evidence.