21 minute read

The question is never “can you build it?”

Any team with a capable engineer and access to a good model can build an agent in a week. The demo will impress. The Slack channel will celebrate. The project will get a budget.

Then, six months later, it quietly dies. Not because the model was too slow or the tools were unreliable, but because nobody ran the math on whether this task was actually worth automating.

In June 2025, Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Those are economic failures more than technical ones. Teams skip the economics.

This post is a framework for not skipping the economics. It gives you a three-axis scoring system you can apply to any agent idea in about ten minutes. I’ll also map 12 real deployed agent archetypes against it, so you can see where the commercial wins actually cluster.

TL;DR

Most agent projects fail commercially not because the technology breaks but because teams pick the wrong use cases. Three axes determine whether an agent creates or destroys value: task automation leverage (time freed × loaded cost), failure tolerance (cost of a wrong action), and supervision ratio (human oversight time as a share of the work automated). This post builds that framework and maps 12 real deployed archetypes against it. See What are AI agents for the foundations this analysis assumes.

A coordinate measuring machine with its ruby-tipped probe touching the surface of a precision-machined metal part

Why most agent projects fail commercially

The short answer: teams optimize for “technically possible” instead of “commercially justified.” Three failure modes account for most dead agent projects.

The first failure mode is wrong use case selection: automating tasks where the economics never worked. A task that takes a $20/hour data entry clerk 4 minutes to complete has $1.33 of loaded cost per task. Building and maintaining an agent for that probably costs more than just paying the clerk.

The second failure mode is misjudged failure tolerance. Agents make mistakes. The question is what those mistakes cost. An agent that occasionally misroutes a customer support ticket is cheap to fix. An agent that occasionally approves the wrong loan application is not. Teams that use the same confidence threshold across both types of tasks end up either over-supervising low-stakes agents (destroying the margin) or under-supervising high-stakes ones (creating liability).

The third failure mode is what I call the supervision illusion: projecting zero ongoing human oversight into ROI models. In pilots, one engineer watches everything closely. In production, that engineer is watching thousands of actions with partial attention. The hidden cost of ongoing review, prompt maintenance, and edge-case triage is consistently underestimated, and unlike the build cost it recurs every month.

Gartner’s prediction is sobering: over 40% of projects canceled by the end of 2027. Its advice is to pursue agentic AI only where it delivers clear value or ROI. That is the point of this framework: select projects by economics, not by “this would be cool.”

The three-axis evaluation framework

Three numbers determine whether any agent use case will generate value: task automation leverage, failure tolerance, and supervision ratio. Score each axis before writing a line of code.

Axis 1: Task automation leverage

Task automation leverage = (time per task in hours) × (loaded hourly cost) × (task volume per month).

This is the total monthly value at stake. It tells you the ceiling: the maximum possible savings if the agent handles everything perfectly. If the ceiling is low, stop here.

Example: a document extraction task that takes a paralegal 15 minutes per contract, at a $120/hour loaded cost, with 400 contracts per month = $12,000/month in leverage. Worth pursuing. A task that takes a $25/hour clerk 3 minutes, 1,000 times a month = $1,250/month. Much harder to justify build and maintenance costs.

A useful rule of thumb: leverage needs to exceed $10,000/month for a project to recover its costs within 12 months at typical enterprise build and operating costs.

Axis 2: Failure tolerance

Failure tolerance is the inverse of the cost of a wrong agent action. High failure tolerance means wrong outputs are cheap to fix. Low failure tolerance means wrong outputs are expensive or irreversible.

Rate it on four tiers:

  • High tolerance: Wrong action costs < $100 to fix or is purely cosmetic. Customer support misrouting, content draft errors, lead scoring mistakes.
  • Medium tolerance: Wrong action costs $100-$10,000 to fix, with moderate reputational risk. Contract clause extraction errors, invoice discrepancy misses, scheduling conflicts.
  • Low tolerance: Wrong action costs > $10,000 to fix or carries regulatory exposure, but can still be caught and reversed. Compliance flags, loan decisions, legal filings.
  • Catastrophic: Wrong action is irreversible or safety-critical. Executing financial transactions, medical treatment decisions.

Low-tolerance use cases are not impossible, but they need a human to check most actions. That pushes the supervision ratio toward the 0.3 cutoff and beyond, at which point the product is really human-in-the-loop decision support and should be built and priced as such.

Axis 3: Supervision ratio

Supervision ratio = human time spent overseeing the agent in production ÷ human time the automated work used to take.

Put differently, it is the share of the original labor you still pay for: 0.1 means six minutes of review, escalation handling and prompt tuning for every hour of work the agent takes over. Multiply it by task automation leverage and you get the monthly supervision cost.

This is the number teams most often set to zero in their projections and then find is actually 0.2 or 0.3 in production. At 0.3, an agent handling 1,000 one-minute tasks per day still needs 300 minutes (5 hours) of human attention per day for review, escalation handling, and prompt tuning: most of someone’s working day.

A sustainable agent typically has a supervision ratio below 0.1. Below 0.05 is excellent.

Here is the framework as a scoring matrix:

COMMERCIAL VIABILITY SCORING MATRIX
=====================================

Axis 1: Task Automation Leverage (monthly)
  > $50k/month    → 3 pts
  $10k-$50k/month → 2 pts
  $1k-$10k/month  → 1 pt
  < $1k/month     → 0 pts (stop here)

Axis 2: Failure Tolerance
  High            → 3 pts
  Medium          → 2 pts
  Low             → 1 pt
  Catastrophic    → 0 pts (human-in-the-loop tool only)

Axis 3: Supervision Ratio (achievable in production)
  under 0.05          → 3 pts
  0.05 to under 0.10  → 2 pts
  0.10 to 0.30        → 1 pt
  over 0.30           → 0 pts (supervision cost eliminates margin)

Total Score:
  7-9 pts → Strong commercial case, build with confidence
  5-6 pts → Conditional; validate with a narrow pilot before full build
  3-4 pts → Weak; reconsider the use case or scope
  0-2 pts → Stop. The economics don't work.

Overrides (apply before reading the total):
  Any axis at 0 pts → stop as an autonomous agent, whatever the total
  Leverage at 1 pt  → best possible verdict is Conditional
                      (see the $10k/month rule of thumb above)

The framework is intentionally blunt. A 2-point score isn’t a project to “refine”. It’s a project to kill.

Mapping 12 agent archetypes to the matrix

Twelve common agent archetypes cluster into three groups: high-confidence wins, conditional plays, and traps that keep getting funded despite poor economics.

For each archetype, I’ve scored the three axes from public deployment data where it exists and from judgment where it does not, so treat the ranges as estimates. Leverage labels map to the matrix bands: High (or Very High) is over $50k a month (3 pts), Medium is $10k to $50k (2 pts), Low is $1k to $10k (1 pt).

Archetype Leverage Failure Tolerance Supervision Ratio Total Verdict
Customer support deflection High ($50k+) High Low (0.04-0.08) 8-9 Strong
Invoice / receipt extraction Medium-High Medium-High Low (0.05-0.10) 5-8 Strong once scoped
Sales lead qualification High High Low-Medium (0.05-0.15) 7-8 Strong
Code review assistance Medium Medium Medium (0.10-0.20) 5 Conditional
Contract clause extraction High Medium Medium (0.10-0.20) 6 Conditional
Loan application pre-screening High Low-Medium Medium-High (0.20-0.40) 4-6 Conditional only below 0.30 supervision
Meeting notes / action items Low-Medium High Low (0.02-0.05) 6-8 Conditional (Strong if leverage tops $10k)
HR policy Q&A Low High Low (0.02-0.05) 6-7 Conditional (capped by low leverage)
Content first drafts Medium High Medium (0.10-0.25) 6 Conditional
Compliance monitoring High Low High (0.30-0.50) 4-5 Trap
Medical triage routing High Catastrophic Very High 3 Trap
Financial transaction execution Very High Catastrophic Very High 3 Trap

Trap means at least one axis scores 0 across most of the estimated range: the leverage is real, but the overrides stop it as an autonomous agent even when the total looks middling. The low-leverage rows show the other override: HR policy Q&A is safe and cheap to supervise, but under $10k a month of leverage it cannot justify more than a narrow pilot.

The three strong wins in detail

Customer support deflection was the clearest commercial win in the agent space when this was written in early 2026. Klarna said in February 2024 that its AI assistant handled 2.3 million conversations in its first month and cut average resolution time from 11 minutes to under 2. On its Q3 2025 earnings call, the CEO put the savings at about $60 million (CX Dive, November 2025). The same report notes that Klarna began bringing human agents back in May 2025, with the CEO saying customers should always have the option to speak to a person: the failure-tolerance axis at work. Intercom reported an average 51% autonomous resolution rate for its Fin agent across customers in 2025. The economics work because support tickets are high-volume, the loaded cost of human agents is real, and an incorrectly handled ticket costs almost nothing to fix: route it to a human.

Invoice and document extraction has an underappreciated ROI profile. GravityStack, a Reed Smith subsidiary, built an agentic review workflow for a global bank that cut credit agreement review from up to five days to under 24 hours, according to a Vellum case study. More broadly, Parseur, a document-extraction vendor, compiles 2025 benchmarks that put best-in-class accounts-payable teams at about $2.78 and 3.1 days per invoice, against $12.88 and 17.4 days for everyone else. The supervision ratio stays low because output format is deterministic (structured fields) and errors are caught before downstream commitment.

Sales lead qualification is the least documented of the three. Vendors such as Persana AI publish double-digit gains in sales productivity and conversion, but those are self-reported marketing figures, so validate them in your own pilot. The case rests on the structure: failure tolerance is high, because a misqualified lead costs a 15-minute sales call, not a contract dispute.

The three traps in detail

Compliance monitoring, medical triage, and financial execution are traps because the leverage is real but the failure tolerance and required supervision ratios make the economics collapse. These use cases attract teams because the dollar value at stake is enormous, but that dollar value is also the risk if the agent gets it wrong. A compliance miss can trigger regulatory penalties. A triage error has patient safety implications. A financial execution error is often irreversible.

The pattern with traps is that teams model the leverage (big number) but not the failure cost (bigger number). The correct play for these archetypes is usually human-in-the-loop with agent assistance, not autonomous agent execution, which is a fundamentally different product. Why human-in-the-loop beats full autonomy for high-stakes agent actions works through how to design that gate.

The supervision ratio trap: why agents fail after launch, not before

The supervision ratio almost always rises after launch. Teams that don’t model this correctly find themselves maintaining an expensive system that saves less time than they projected, and the problem compounds as scale increases.

Here is how it typically plays out:

In the pilot phase (months 1-2), one engineer handles 50-100 agent tasks per day with close monitoring. The supervision ratio looks excellent: they’re watching every output, but it’s fast. The ROI calculation looks great.

In the initial production phase (months 3-6), volume scales to 1,000 tasks per day. The same engineer is now responsible for reviewing agent outputs at 10 to 20 times the volume. They spot-check. Edge cases slip through. The first escalations arrive from customers or downstream systems. Prompt changes get made reactively. The supervision ratio quietly climbs from 0.05 to 0.20.

In the scaling phase (months 7-12), either the team hires dedicated agent reviewers (which shows up in cost as a line item nobody projected) or the agent quality degrades visibly (which shows up in customer complaints or downstream errors).

Ongoing oversight can take anything from part of one person’s week to several full-time roles, depending on volume and complexity, and most ROI models never include it. Prompt and eval maintenance is a separate line: 15 to 20 hours a month per deployed agent is a reasonable planning assumption until you have your own numbers.

The fix is to model supervision cost explicitly from the start:

True monthly agent cost =
  LLM inference cost
  + infrastructure
  + supervision_ratio × task automation leverage
      (= supervision_ratio × hours per task × task_volume × loaded_hourly_cost)
  + amortized build cost
  + prompt maintenance hours × loaded_hourly_cost
      (~15-20 hours/month per deployed agent; skip if already in the supervision ratio)

For cost management at the infrastructure level, model routing and circuit breakers help. But those reduce inference cost, not supervision cost. Supervision cost is a function of how well the agent handles edge cases, which is an eval problem, not a cost-routing problem. The build-cost line moves too if a managed platform can replace part of the build; Claude Managed Agents: what the first managed agent platform actually buys you covers that buy-versus-build trade.

The teams that succeed long-term obsess over reducing their supervision ratio. They invest in evals that surface edge cases before they hit production (see Agent Evaluation Frameworks). They build output schemas so agents produce structured outputs that downstream systems can validate automatically. They track supervision ratio as a first-class metric, not an afterthought.

How to evaluate your next agent idea in 10 minutes

Three questions, asked in order. If any answer is “no” or “I don’t know,” stop and find out before writing code.

This is the quick-test version of the three-axis framework. It works for initial conversations, not for final go/no-go decisions, but it kills bad ideas fast.

Question 1: What is the loaded cost of the human doing this task today?

Not the salary. The loaded cost: salary + benefits + overhead, divided by productive hours. A $90k/year knowledge worker costs roughly $75-90/hour fully loaded. Multiply by time per task and task volume. If the monthly leverage number is below $5,000, ask whether a simpler automation (a script, a form, a better tool) would work before an agent.

Question 2: What happens when the agent gets it wrong, and how often will that happen?

Think through the worst-case wrong action at realistic volume. If the agent handles 10,000 tasks per month with a 2% error rate, that’s 200 wrong actions monthly. What does each cost to fix? If the answer is “we’d need to audit everything”, that’s a supervision ratio problem. If the answer is “it’s annoying but harmless”, that’s high failure tolerance.

Question 3: What does 12-month ongoing supervision look like?

Name the person. Describe their weekly routine for monitoring this agent. If you can’t describe it concretely, you don’t have an operating model. A concrete operating model includes: who reviews edge cases, how often prompts get updated, how escalations route, and what a “breaking change” from the model provider triggers.

Here is the decision flow:

flowchart TD
    A[New agent idea] --> B{Monthly leverage > $10k?}
    B -- No --> C[Kill or simplify]
    B -- Yes --> D{Failure tolerance: High or Medium?}
    D -- Low/Catastrophic --> E[Human-in-loop model only]
    D -- High/Medium --> F{Supervision ratio < 0.15 achievable?}
    F -- No --> G[Redesign task scope or add output constraints]
    F -- Yes --> H[Passes the screen: score it on the matrix]
    G --> I{After redesign: ratio < 0.15?}
    I -- No --> C
    I -- Yes --> H
    H --> J[Build with eval + supervision cost modeled]

The branch to “Human-in-loop model only” is not a failure. It’s a different product. Agent assistance for low-tolerance tasks is often a very good product. But it should be built and priced as a decision-support tool, not an autonomous agent, and its ROI model looks different.

A note on the “redesign task scope” branch: many agent ideas that initially score poorly can be restructured to score well. Take invoice extraction. Scoped as “process all invoices autonomously”, every wrong field flows straight into payments, so failure tolerance is Medium at best and someone has to check a lot of output. Scoped as “extract structured fields from standard invoices, flag non-standard formats for human review”, most errors are caught before anything is paid, failure tolerance moves toward High, and reviewers only see the flagged minority. That is why the table rates invoice extraction Medium to High on failure tolerance and calls it strong once scoped.

What this means in practice

  • Task automation leverage sets the ceiling. If the math on time × loaded cost × volume doesn’t work, no amount of clever engineering fixes it.
  • Failure tolerance determines the required supervision ratio floor. For catastrophic-failure tasks a human has to check every action, which keeps the ratio high unless the agent acts in an advisory role only.
  • Supervision ratio is the number that kills projects after launch. Model it explicitly, including prompt maintenance time (budget 15 to 20 hours a month until you have real numbers) and edge-case review time.
  • The three archetypes with the strongest commercial track records share a common structure: high volume, high human loaded cost, and recoverable failure mode. Customer support, document extraction, sales qualification.
  • The traps share the opposite structure: enormous leverage that attracts investment, but failure costs that make autonomous operation impossible at any realistic accuracy level.
  • Every agent idea should survive a 10-minute economics check before receiving engineering time.

For the deployment decisions that follow from a positive economics check, see Agent Deployment Patterns.

FAQ

How do you calculate the ROI of an AI agent before building it?

Multiply the time saved per task by the loaded cost of the human doing that task, then subtract the agent’s per-task cost: LLM calls plus infrastructure plus amortized build cost. If the margin is positive and the task volume is high enough to recover build cost within 12 months, the ROI case is strong. The harder variable to estimate is supervision ratio: the share of the original human effort you still spend overseeing the agent after deployment. Most teams set this to zero in projections; budget 0.1 to 0.3 until you have production data.

What is task automation leverage for AI agents?

Task automation leverage is the product of time freed per task and the loaded hourly cost of the person doing that task, multiplied by monthly volume. A task that takes a $120/hour paralegal 15 minutes per document, processed 400 times monthly, has $12,000 in monthly leverage. Tasks with low leverage (high volume but very low loaded cost) rarely justify the build and maintenance cost of a production agent, and simpler automations are usually more appropriate.

Why do AI agents fail after launch rather than before?

The supervision ratio rises after launch. In a pilot, one engineer monitors every agent action closely. In production, that same engineer’s attention spreads across thousands of daily actions. Teams that never model ongoing review cost (catching edge cases, handling escalations, maintaining prompts) find supervision eats the margin they projected. Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls.

Which agent use cases have the best commercial track record?

Customer support deflection, document extraction (invoices, contracts, forms), and sales qualification have the strongest commercial records. Klarna says its support agent has saved about $60 million and cut average resolution time from 11 minutes to under 2. GravityStack cut a global bank’s credit agreement review from up to five days to under 24 hours. These archetypes share three properties: high task volume, high human loaded cost, and low catastrophic failure risk. A wrong action costs a small fix, not an irreversible outcome.

What is supervision ratio and why does it matter for agent economics?

Supervision ratio is the human time spent overseeing an agent in production (reviewing outputs, catching edge cases, handling escalations, maintaining prompts) divided by the human time the automated work used to take. An agent with a 0.3 supervision ratio still costs 18 minutes of human attention for every hour of work it takes over, so 30% of that labor cost stays on the books. Most teams model zero ongoing supervision in their ROI projections; budget 0.1 to 0.3 until you have production data. Sustainable agents target below 0.1.

Related posts