If you are a VP, a head of function, or a chief of staff trying to answer the question are our agents actually working? — you have probably been handed a number. Something like “35% productivity uplift” or “$1.8M in saved analyst hours.” Take a breath before you repeat that number in a board deck. There is a good chance it is wrong in a direction that matters.

Over the past six months I have compared instrumentation approaches across eleven agent rollouts spanning three industries. The question I kept asking operators was simple: what denominator are you using? And the answers, depressingly often, were variations of “I hadn’t thought about that.” This piece is the framework I wish I had handed each of them on day one.

The failure mode: time-saved, undiscounted

The standard ROI calculation for an agent deployment looks like this:

(Hours of human work the agent handled) × (Blended cost of that human labor)
− (Inference and license cost of the agent)
= ROI.

Everything on the top line is either fiction or flattered. Four problems, in order of severity:

1. Rework is invisible. When an agent delivers a draft that a human has to rewrite, the hours of rework almost never get logged against the agent’s output. They get logged as “normal work.” The honest denominator is hours of human work the agent handled, net of human correction time. In the eleven rollouts I reviewed, median correction time consumed 27% of the “hours saved” figure before anyone ever measured it.
2. Quality displacement is invisible. If the agent’s output is 10% lower quality than a human’s and the quality differential shows up three quarters later as a customer-retention dip, that is a real cost. It is not on the dashboard. It is in the churn report.
3. Selection bias runs the other direction, too. Operators often front-load agents onto the easiest tasks (“let’s start with something simple”), then extrapolate the resulting ROI to all tasks. The easy-task ROI is real; the extrapolation is not.
4. Opportunity cost is missing. The same dollar spent on agent infrastructure could have been spent on a human hire, a training program, or a non-agent tool. The honest comparison is not agent vs. nothing. It is agent vs. next-best alternative.

Three metrics that survive scrutiny

Here are the three I have found to be robust, in the sense that they do not flatter themselves and do not quietly decouple from outcomes:

Metric 1: Net task throughput, quality-gated

Count tasks completed and accepted by a downstream consumer (customer, reviewer, next-stage process) without rework above a threshold. Measure against the same team’s pre-deployment baseline, normalized for headcount. This metric punishes rework and selection bias because the denominator stays honest.

Metric 2: Human-hour reallocation, tracked

Not “hours saved” — where those hours went. If the analyst whose drafts are now written by an agent is spending the freed time on higher-judgment work, that is the real productivity story. If the freed time is going into meetings, ambient context-switching, or filling in for understaffed colleagues, the ROI is on paper only. Require teams to report the reallocation explicitly. This is also the metric your community-management and people-development functions should own jointly; the answer depends on the social reorganization of the team, not just the tool.

Metric 3: Failure-mode telemetry

Instrument your agents to log every time they abstain, escalate, or produce something a reviewer flags. Rate per 100 tasks. Track the distribution over time. This is the metric that catches quality rot early — long before it shows up as a downstream business number. In the rollouts I reviewed, teams that tracked failure-mode telemetry caught regressions 60 to 90 days earlier than teams relying on aggregate satisfaction scores.

A note on the human-agent mix

The most important thing the measurement literature is starting to surface — and where the social-science evidence is most unambiguous — is that the right unit of analysis is the team, not the tool. Agent ROI measured at the tool level routinely disagrees with agent ROI measured at the team level, and the team-level number is the one that predicts retention, satisfaction, and long-horizon output quality. If your measurement framework stops at the agent, you are measuring the wrong thing.

A closing provocation

The field has internalized that you cannot manage what you do not measure. It has not yet internalized that measuring the wrong thing is worse than not measuring at all, because a flattering wrong number is harder to dislodge than no number. If your organization’s agent ROI figure is currently comforting, ask for the denominator. If the answer is soft, the number is soft.

# Field prompt — measuring-agent-roi — paste into your own harness

Audit an agent-deployment ROI calculation against "Why Your Agent ROI Number
Is Wrong" (Substratics, The Operators, April 23, 2026). I will paste
the current calculation (or slide, or dashboard definition) in my next turn.

For the calculation I provide, identify:
  - which of the four denominator failures are present: invisible rework,
    invisible quality displacement, selection bias from easy-task
    front-loading, missing opportunity-cost comparison;
  - for each present failure, the specific line item or assumption that
    carries it;
  - whether the three scrutiny-surviving metrics are measured, partially
    measured, or absent: (1) quality-gated net task throughput,
    (2) tracked human-hour reallocation, (3) failure-mode telemetry
    (abstentions, escalations, reviewer flags per 100 tasks).

Do not recommend a new ROI number. Produce a punch list the operator can take
to the team owning the calculation. Flag any claim that cannot be verified
from the source provided.