From cost-of-pass to cost per successful task: A task-level accounting protocol for agentic AI products

Abstract

An agentic AI product can make multiple calls to models, use external tools, try things a few times, verify a lot of things, call for human input, etc. to complete a single user request. These are pieces of this process, but do not, individually, represent the cost of a done job. Previous work has shown that cost-of-pass is the anticipated cost to finding a correct solution, and recent studies have begun to associate cost with the outcomes that a successful solution would yield. This paper does not therefore suggest a new economic concept, cost per successful task (CPST). Instead, it defines an accounting protocol at the task level for using outcome based costing for bounded agentic workflows. The protocol specifies a task and its definition of success before execution, assigns variable costs to each task started and unsuccessful tasks to the cost numerator and reports both the cost and the success, latency, human intervention and policy compliance. It splits up the costs of execution and costs of allocated platform and downstream losses. We derive the relationship between CPST and success probability, describe the relationship between bounded retries and that same relationship, and propose an evaluation design to test whether or not the bounded retries change the product decisions based on the cost of that product vs. its model price. The contribution is a specification for a measure, not an empirical measure, and a set of propositions which are testable, not empirical claims.