LLM Agents for Incident Response: Intuit’s EWOK Pattern for Safe, Deterministic Failovers

At 2:13 a.m., an engineer types: “Failover payments-gateway in production.”

Minutes later traffic is shifted, databases are repointed, a change record is closed, and the incident moves toward resolution, without a long call or manual runbook steps. Intuit reports that teams across the company have used EWOK Agent to run failovers for the past eight months and that, for supported workloads, recovery time moved from “several hours” to about 20 minutes.

That outcome rests on one clear design rule: use a foundation model to interpret intent and choose the action, but keep all authenticated, state-changing work inside deterministic, auditable systems. As the Intuit team puts it:

“The model decides what to do, and the EWOK Agent deterministically executes how.”

How the pieces fit, four layers and a one-line flow

Consumer layer, Engineers send plain-language requests through an Engineering Portal or IDE (integrated via a Model Context Protocol).
Agent layer, Amazon Bedrock (Converse API + Guardrails) runs a bounded reasoning loop, picks skills, and produces structured tool calls.
Skill layer, Typed, versioned skills (Markdown with YAML frontmatter describing I/O schemas and a prompt body) are compiled into toolSpecs the model can call.
Execution layer, EWOK APIs and workload-specific executors make authenticated, deterministic calls to move traffic, reconfigure DBs, update caches, and create change records.

End-to-end example in one sentence: the engineer’s request → model selects the appropriate skill + fills typed arguments → executor validates args and enforces policy gates → EWOK runs the deterministic failover workflow and writes the change record.

What a skill looks like (concrete, typed primitives)

Skills are deliberate artifacts: Markdown files with YAML frontmatter that declare typed input and output schemas, plus a prompt body describing decision logic and step-by-step rules. Example input-schema fields shown by the team include these exact strings:

operation: “‘get-workflows’, ‘invoke-failover’, or ‘get-status'” (required: true)
asset_name: “Name or alias of the asset to act on” (required: true)
environment: “Target environment, e.g. ‘staging’ or ‘production'” (required: true)
incident_number: “Only needed to override an active change-freeze window” (required: false)

Example output schema fields used by the team:

status: “‘success’ or ‘error'”
result: “Operation-specific payload (workflows, execution ID, or status)”

Those typed schemas constrain model outputs, make arguments machine-checked, and simplify compiling skills into Bedrock toolSpecs for function-calling.

The bounded agentic loop, observable and interruptible

Intuit runs a bounded loop using the Amazon Bedrock Converse API. The model may select a compiled tool (a skill) and return structured arguments. The executor runs the corresponding EWOK call and feeds the result back into the loop. The implementation explicitly handles stop reasons such as "end_turn", "guardrail_intervened", and "tool_use" and enforces a hard iteration cap (MAX_ITERATIONS) so the agent can’t spin indefinitely.

The team used the langchain-aws ChatBedrockConverse client to bind tools and simplify message plumbing, but they chose a self-managed loop instead of a managed harness so they could implement custom branching, circuit breakers, and resiliency logic tailored to production failovers.

Security and safety: layered defenses

LLM-driven ops automation changes the attack surface. Intuit layers protections so the model cannot directly subvert production systems:

Credential blinding, The model does not hold AWS credentials. Executors perform authenticated EWOK API calls with request-scoped IAM roles.
Amazon Bedrock Guardrails, Guardrails are attached to each model invocation and the implementation treats guardrail interventions as explicit stop reasons that halt runs.
Executor-side verification, Executors validate tool arguments (service names, Regions, idempotency keys) against known registries before any state change.
Denial-of-service controls, Per-service job queues with deduplication and cooldowns, circuit breakers, and the MAX_ITERATIONS cap prevent runaway automation.
Human-in-the-loop, Critical or irreversible steps require manual authorization; skill prompt bodies can instruct the model to ask for an incident number or emergency justification and re-run exactly once with that value.
Auditability, Runs open change records and write immutable audit logs anchored by change records, providing a tamper-evident trail of decisions and actions.
Least privilege and anti-replay, Strict IAM scoping, API rate limiting, nonces/timestamps, and other controls limit blast radius and replay risk.

These are complementary controls: Guardrails reduce prompt-injection risk at the model layer, and executor-side checks and request-scoped credentials protect the execution boundary.

Operational experience, what Intuit reports and what they don’t

Intuit reports eight months of production use across teams and a stated improvement for supported workloads from “several hours” down to about 20 minutes. That is a meaningful improvement if representative. Platform teams should treat it as an operator-reported outcome rather than a universal guarantee.

Notably, the public write-up does not list certain telemetry that matters to decision-makers. Missing or single-source items include:

Which specific foundation models (model IDs/providers) were evaluated and chosen in production.
How many failovers ran through EWOK Agent during the reported period and overall success/failure rates.
Distributional metrics: mean, median, P95/P99 recovery times; average agent iteration counts; frequency of guardrail interventions.
Monetary cost breakdown (dollars per failover, monthly spend) for Bedrock inference and Guardrails versus human-hours saved.
Skill governance details at scale: review, testing, versioning, and rollback workflows.
Incident history or near-miss disclosures during the eight months of use.

Those gaps aren’t a deal-breaker, but they are the right questions to ask any team proposing LLM-driven ops automation before you adopt the pattern.

Cost considerations, billing vectors to watch

Intuit notes the primary billing drivers are Bedrock model inference (input and output tokens billed per Converse call) and Guardrails (billed based on content evaluated per invocation). The team also notes that Bedrock model access is billed for invocations rather than as a standing idle charge. Platform teams should verify current AWS Bedrock pricing and Guardrails billing before estimating costs for production usage.

Practical checklist before you adopt this pattern

Declare strict, typed tool schemas for every action exposed to models; compile those into toolSpecs the runtime can call.
Keep credentials out of the model. Use a separate executor with request-scoped IAM credentials for all state changes.
Attach Guardrails to model invocations and make the implementation treat guardrail annotations as stop signals.
Validate every model output again in the executor before performing any action (service name checks, Region checks, idempotency keys).
Require human approval for destructive or high-impact operations and log every step in immutable audit trails anchored by change records.
Set hard iteration caps, per-service queues with deduplication, cooldowns, and circuit breakers to limit blast radius.
Instrument telemetry: total runs, success rate, guardrail interventions, iteration counts, mean/median/tail recovery times, and approval latency.
Define a skills governance workflow: authoring checklist, automated tests, peer review gates, and rollback procedures.

Key takeaways, questions you might ask

How much faster does this make failovers?

Intuit reports that, for supported workloads, recovery time improved from “several hours” to about 20 minutes. Treat this as an operator-reported figure and ask for telemetry (sample size, mean/median/P95) to validate how representative it is.
Does the model ever perform state-changing operations directly?

No. The model selects typed skills and returns structured arguments; an authenticated executor with request-scoped IAM performs the deterministic EWOK API calls and writes change records.
How are prompt-injection and model misbehavior addressed?

Amazon Bedrock Guardrails are attached to each invocation and the implementation treats guardrail interventions as stop reasons; executor-side validation, credential blinding, and human approvals provide additional protection.
What drives the bill?

Model inference tokens per Converse call and Guardrails content evaluation are the primary billing vectors. Platform teams should verify current AWS Bedrock pricing for exact costs.
Can I reuse the sample code as-is?

No. The team’s code excerpts are illustrative and are not published as a complete, runnable implementation or CloudFormation stack.

What platform leaders should ask next (concrete requests)

Provide last 90-day telemetry: total EWOK Agent runs, success rate, P50/P95/P99 recovery times, guardrail intervention count, and average iteration count per request.
Supply a model-selection memo: which foundation models were tested (model IDs/regions), why the chosen model was selected, latency and token-cost trade-offs, and the plan to detect and remediate model drift.
Share a skills governance workflow: authoring checklist, automated unit/integration tests, peer-review gates, and a rollback playbook for bad skill versions.
Show cost vs. benefit: estimated Bedrock + Guardrails spend per failover and monthly, plus human-hours saved (and how those savings were measured).
Explain concurrency and chaos plans: how the system behaves under mass incidents, queue/backoff policies, and results from stress or chaos tests.

Agentic orchestration isn’t a shortcut around platform discipline, it magnifies it. When you separate decisioning (LLM) from execution (authenticated, audited EWOK workflows), codify operations as typed primitives, and instrument every step, you get faster, more repeatable incident response. Skip those guardrails and you risk fast, repeatable failures instead of fast recoveries.