Frontier Radar #2: Why AI productivity gets lost between benchmarks and the balance sheet

TL;DR: Generative AI speeds many discrete knowledge‑work tasks, but those time savings rarely show up in corporate financials unless firms redesign workflows, measurement and incentives—what economists call absorptive capacity (an organization’s ability to adopt and use new technologies effectively).

A customer‑support team I spoke with shaved draft time per ticket by 30% after rolling out a generative assistant. But monthly support costs and CSAT barely budged. Faster drafts didn’t equal fewer escalations, lower headcount or happier customers—because the team spent the saved time verifying outputs, reworking thin responses, and adding performative steps to satisfy managers. That story is common: generative AI and AI agents deliver micro wins; converting those into balance‑sheet impact requires organizational engineering.

Evidence: where generative AI clearly helps (and by how much)

Field trials and benchmarks agree that generative AI improves output on narrow tasks. Below are headline results and what they practically mean for business leaders.

• Customer service (Brynjolfsson, Li, Raymond; QJE): Resolved issues per hour rose ~14–15% with a generative assistant. Practical implication: Less experienced agents benefit most—so targeted deployment on junior teams can raise throughput quickly.
• Professional writing (Noy & Zhang, 2023): ChatGPT shortened completion time and increased quality on average. Practical implication: Content teams can scale drafts, but expect editorial verification to remain necessary.
• Coding assistance (early GitHub Copilot study): ~56% faster completion on a constrained coding task. Practical implication: For small, well‑scoped developer tasks, AI is a true turbocharge; for multi‑module features, the lift is smaller.
• Large randomized trials (Microsoft/NBER; multiple companies): Roughly 26% average increase in completed tasks across many knowledge‑work roles. Practical implication: Cross‑department pilots show broad gains, but translating them to cost savings needs process change.
• Developer trials (Google randomization): Developers were up to ~20% faster with AI. Practical implication: Productivity gains exist but vary by task familiarity and tooling fit.
• Agentic benchmarks (APEX‑Agents, FeatureBench, ResearchGym): End‑to‑end success rates typically fall in the ~11–26% range. Practical implication: Current AI agents often fail on complex, tool‑heavy workflows—human oversight remains essential.
• Survey & macro signals (St. Louis Fed; BIS; Danish registry): St. Louis Fed user‑level time saved ≈5.4% (implying ~1.1% potential economy‑wide productivity); BIS firm‑level study found ≈4% labor‑productivity lift among adopters with complementary investments; Danish registry work detected task shifts but no clear wage/hours effects after two years. Practical implication: Firm size, complementary investments and measurement matter—small pilots alone won’t move GDP.

“Generative AI speeds many tasks, but faster task completion doesn’t automatically create measurable economic value.”

Why headline time savings shrink before they hit the balance sheet

Task‑level wins leak away through a set of predictable frictions. Each is fixable, but many firms treat AI as a plug‑in that doesn’t require process change.

• Verification and remediation overhead: AI can sound confident but be wrong—humans must check outputs. A five‑minute shortcut that triggers ten minutes of review is a net loss. Better quality controls and model‑specific SLAs reduce this leak.
• Wrong metrics and performative work: Many companies still measure activity (emails sent, tickets closed). Visibility can increase without value: workers may produce items that look productive but add little economic value.
• Hidden downstream costs: Low‑quality AI outputs cause rework, customer confusion and cognitive fatigue. Studies report notable remediation time and user exhaustion; heavy AI delegation while learning can also degrade skill formation.
• Organizational inertia and absorptive capacity: Without redesigned workflows, saved time becomes slack, extra meetings, or performative tasks. Turning speed into outcomes requires new accountability, tooling and training.
• Mismatch between benchmarks and messy reality: Benchmarks test narrow tasks; real work chains many subtasks, approvals and external waits—weak links in the chain blunt overall gains.

Where gains reliably stick

Not all functions are equally salvageable. Focus where returns are clearer:

• Template and drafting work (first drafts, email triage, boilerplate contracts): AI reduces repetitive effort and scales output.
• Triage and categorization (support ticket routing, resume screening): AI speeds decisioning that maps directly to downstream routing actions.
• Code suggestions for small tasks: Single‑file edits or autocomplete where ownership and tests are clear.
• Augmented research and summarization: Rapid literature or internal data summaries that are then validated by domain experts.

Where to be cautious: end‑to‑end automation of complex decisions, unsupervised agentic workflows, and any high‑stakes decisions without robust verification and governance.

A practical playbook: convert task speed into measurable business value

Think of AI deployment as process redesign, not a model swap. Below are concrete steps, KPIs and a sample calculation leaders can use within the next 90 days.

5-step executive checklist (90‑day pilot)

1. Pilot with outcome KPIs, not activity counts. Track cycle time, error/rework rate, conversion or CSAT. Example KPI: “Net time per ticket” (see formula below).
2. Measure verification & remediation explicitly. Log human review minutes per AI output and remediation events so gross gains are not overclaimed.
3. Redesign workflows to remove bottlenecks. Move approvals earlier/later, add clear ownership of AI output, and automate low‑risk checks.
4. Invest in tooling and training. Provide prompt libraries, validation UIs and a 3‑month reskilling track: verification skills, prompt engineering, decision oversight.
5. Set governance & SLAs. Define acceptable error rates, human‑in‑the‑loop thresholds and audit windows for each use case.

KPIs and a simple formula

Net time savings per task = Gross AI time saved − Verification time − Remediation time.

Example (per ticket):

• Gross AI saved time: 6 minutes (drafting)
• Verification time: 3 minutes
• Remediation time (per 10 tickets): 1 minute on average = 0.1 minute per ticket
• Net time saved = 6 − 3 − 0.1 = 2.9 minutes per ticket

Multiply net time saved by ticket volume and labor cost to estimate short‑term cost impact. Track net time month‑over‑month as models and workflows improve.

Governance pattern (human‑in‑the‑loop)

• Tier 1: AI drafts + mandatory human validation for public outputs.
• Tier 2: AI suggests diagnostics or routes; auto‑apply if confidence > threshold and logged for audit.
• Tier 3: High‑stakes decisions require human signoff; AI only provides candidates and evidence.

Reskilling roadmap (3 months)

• Weeks 1–2: Verification fundamentals and prompt best practices.
• Weeks 3–6: Role‑specific tooling and governance training (workflows, SLAs).
• Weeks 7–12: Advanced usage—embedding AI into team processes, measuring net impact, and peer reviews.

Case vignette: how one firm converted drafts into lower costs

Background: A mid‑sized SaaS company rolled out a generative assistant to its onboarding support team. Initial measurements showed a 25% reduction in average drafting time.

What they changed:

• Switched KPIs from “tickets closed” to “time to resolution” and “repeat contacts within 7 days.”
• Logged verification time for every AI draft and tracked remediation events.
• Created a prompt library and a short verification checklist embedded in the support UI.
• Allocated saved staff capacity to proactive outreach (reducing churn) rather than letting it evaporate.

Outcome after 6 months: Net time saved per ticket stabilized at ~3 minutes after verification. Repeat contact rate fell by 8%, and net support cost per customer declined by ~5%. The firm captured measurable cost savings and improved retention—because it changed what it measured and how teams spent freed time.

Scenarios: baseline, acceleration, and slowdown

• Baseline: Continued micro gains in many tasks; modest firm‑level productivity improvements over several years unless firms redesign processes and metrics.
• Acceleration: Broad process redesign, standard outcome metrics across firms, and investments in absorptive capacity unlock faster economy‑wide productivity growth.
• Slowdown: Hidden costs (remediation, cognitive load, skill hollowing) accumulate; time savings fail to translate into economic value and may even create net‑negative outcomes in some units.

What to watch next: agentic systems and measurement

Newer agentic benchmarks report end‑to‑end success rates in the low tens of percent for complex tool‑heavy tasks. That means AI agents are useful as supervised assistants today, but not yet reliable autonomous operators for high‑stakes workflows. Two practical things to watch:

• Improving tool‑model integration: Better connectors, test harnesses and visibility into model decisions will shrink verification time.
• Measurement standards: Industry‑level KPIs for AI outcomes (not activity) will make it possible to compare ROI across pilots and accelerate adoption where real value is created.

Key takeaways

• Generative AI boosts task speed—but speed is not the same as value. Firms must measure net outcomes, not gross activity.
• Verification and governance are not optional. Budget time and headcount for review during rollout and quantify that cost.
• Invest in absorptive capacity. Tooling, training and process redesign determine whether AI affects the balance sheet.
• Use pilots to test full end‑to‑end redesigns. Pilots that only swap in a model without changing workflows usually fail to capture downstream value.
• Monitor for hidden costs. Track cognitive fatigue, remediation time and skill maintenance to avoid long‑term regressions.

Frequently asked questions

Does generative AI actually speed up work?

Yes. Multiple field studies show meaningful task‑level speedups in customer service, writing and coding, with some experiments reporting double‑digit percent gains. The size of the gain depends on task scope and verification needs.

Why don’t those speedups show up in financials or GDP?

Verification overhead, measurement choices, organizational inertia and hidden downstream costs often consume headline time savings before they affect revenue or official output metrics.

Are AI agents ready to run end‑to‑end processes?

Not reliably yet. Benchmarks report modest end‑to‑end success rates (roughly 11–26% in recent tests). Use agents with human oversight and checkpoints for now.

Could hidden costs outweigh benefits?

They can, if firms ignore remediation, cognitive load and skill decay. Measuring those costs explicitly prevents net‑negative outcomes.

Sources & further reading

• Brynjolfsson, Li & Raymond — QJE field experiment on customer service (resolved issues per hour +14–15%).
• Noy & Zhang (2023) — ChatGPT impact on professional writing tasks.
• GitHub Copilot early study — constrained coding tasks (~56% faster).
• Microsoft / NBER randomized field trials across many firms (average ~26% task increase).
• Google randomized trial — developer speedups (~20%).
• APEX‑Agents, FeatureBench, ResearchGym — agentic system benchmarks (11–26% end‑to‑end success ranges).
• St. Louis Fed survey — user‑level and workforce‑level time savings estimates.
• Aldasoro et al. / BIS — firm‑level European study (≈4% productivity lift among adopters with complementary investments).
• Humlum & Vestergaard — Danish registry study (task changes but no detectable income/hours effects after two years).
• BCG “AI Brain Fry”, BetterUp/Stanford “Workslop”, Anthropic developer study, Computers in Human Behavior — studies on cognitive load, remediation and learning effects.
• Penn Wharton, OECD, Anthropic — recent productivity projections for AI.

If you want a ready‑to‑use 90‑day pilot template, KPI dashboard or a one‑page checklist for C‑suite adoption teams, we can prepare those tools to help you convert task‑level gains into measurable business outcomes.