When an agent repeats the same spreadsheet mistake, make it write down the fix
Models keep tripping on procedural edge cases: a misplaced formula, a skipped normalization step, a brittle chain of API calls. Retraining a model for every subtle failure is costly and risky. Google Research suggests another path: don’t change the model, have the agent keep a notebook of lessons it can consult and expand.
Matthias Bastian reported for The Decoder on Aug 29, 2026 that Google Research’s WikiSkill framework does that. WikiSkill pairs agents with a persistent, wiki‑style knowledge base that records full execution traces, documents failure patterns and successful tactics, and distills them into reusable procedural modules called “Agent Skills.” The underlying paper and figures are attributed to Tang et al., 2026 (Google Research).
How WikiSkill is organized, three tiers and two automated roles
The framework is built as a clear pipeline with the exact labels used in the paper:
- “Raw Layer”, immutable execution traces: tool calls, returned values and full step-by-step logs.
- “Wiki Layer”, distilled summaries: documented failure patterns, heuristics that worked, and structured instructions. The authors describe the wiki as append-only for their experiments.
- “Skill Layer”, active procedural instructions the agent executes. Skills are modular and can be swapped or rolled back.
Two automated components run the evolution loop: the “Wiki Maintainer” analyzes traces and writes distilled findings into the wiki, and the “Skill Proposer” suggests updates to the active skills. A gating mechanism evaluates proposed changes on a held‑out validation set. Successful proposals replace the active skills, while failed proposals are rolled back. The wiki itself remains intact.
Why this matters: iterative behavior improvement without weight updates
True continuous learning, safely updating model weights as new experience arrives, remains a hard open problem. As The Decoder summarizes, “the agent, aka the model, doesn’t truly learn in a continuous sense, which remains an unsolved problem.” WikiSkill sidesteps that by compiling agent experience into external procedural instructions instead of changing model parameters. That keeps the base model stable while letting behavior improve over time.
What the experiments show
Google Research evaluated WikiSkill across five benchmark domains (LiveMath for math reasoning, SealQA for web QA, SpreadSheet for spreadsheet manipulation, OfficeQA for document QA with long contexts, and ALFWorld for an interactive virtual environment) using several models: Qwen (4B, 9B, 27B), Gemma‑4‑31B, and Gemini‑3.5‑Flash.
Reported example improvements (all values are averages across three independent runs, as noted in the paper):
- Gemini‑3.5‑Flash: average score rose from 49.5% to 68.1%.
- Qwen‑3.6‑27B: average score rose from 39.4% to 63.3%.
- Gemini‑3.5‑Flash on LiveMath: from 33.0% to 72.6%.
- Gemini‑3.5‑Flash on SpreadSheet: from 50.5% to 76.6%.
The paper notes that bold table values indicate top scores and that multiple bold values in a column are not statistically different from each other. Larger models tended to gain more from evolved skills, but smaller models with WikiSkill sometimes matched larger models that didn’t use the framework. Skills also often transferred between models and, in some cases, outperformed skills the receiving model had developed itself, though transferability is case‑dependent.
Patterns enterprises should care about
- Procedural tasks are where WikiSkill helps the most: multi-step routines like math and spreadsheets saw the largest gains.
- It offers a practical safety tradeoff: behavior improves while model weights stay unchanged, which simplifies certification, rollback and compliance.
- Validation gating matters: every skill proposal is tested on a held‑out set before promotion; failed proposals are rolled back.
- Transferability speeds rollout across models, but it is not guaranteed, expect variation by task and by model.
Operational and governance risks you must address
WikiSkill turns agent experience into persistent operational knowledge. That’s powerful, and it creates new attack surfaces and governance responsibilities:
- Privacy and data leakage: Immutable traces can capture PII or proprietary inputs and outputs. Put redaction, strict access controls, and retention policies in place before storing traces at scale.
- Poisoning and adversarial edits: A writable knowledge base that shapes agent behavior is a target. Treat the wiki as a security boundary: authenticate writers, log edits, and require review for skills that change high‑impact behavior.
- Drift and brittle heuristics: The wiki is append‑only in the experiments; left unchecked it could collect brittle or incorrect heuristics. Add versioning, automated sanity checks, and periodic human audits.
- Validation gaps: The paper describes a gating mechanism but gives few operational details on how held‑out validation sets are built and how promotion thresholds are defined. Teams must design robust validation protocols that match their risk profile.
- Compute and storage cost: Storing full execution traces and running continual distillation pipelines adds storage and compute overhead that scales with traffic and trace detail.
- Legal/IP questions: Moving skills between models or teams raises ownership and licensing issues; make provenance and licensing explicit in your processes.
Try a pragmatic prototype
You don’t need Google‑scale infrastructure to test whether a skills wiki helps your agents. Build a lightweight loop that mirrors the core elements:
- Capture a structured trace schema for every agent run: timestamps, tool name, inputs, outputs, and execution context.
- Store traces in an object store with a metadata database to index events and queries.
- Use a rule‑based or LLM‑assisted distiller to summarize repeated failures and candidate fixes into a small, versioned wiki.
- Implement a gated Skill Layer: run proposed skills on a held‑out validation set and promote only those that improve your operational metrics without regressions; log rollbacks.
- Require human sign‑off for any promoted skill that affects safety, compliance or customer data.
Track these experiment metrics: task accuracy, latency, rollback rate, number of wiki edits, and percentage of promoted skills that later required manual correction. Those signals help you decide whether to expand the system.
Key takeaways, questions you’d ask, and honest answers
-
What is WikiSkill and why should my team care?
WikiSkill is a Google Research framework that pairs agents with a persistent, wiki‑style knowledge base to distill execution traces into reusable procedural modules called “Agent Skills.” It improves agent behavior without changing model weights, which simplifies governance and rollback compared with continuous weight updates.
-
Does WikiSkill update the model’s parameters?
No. The system leaves model weights untouched. It evolves behavior by producing external procedural instructions in the “Skill Layer.”
-
How big are the reported gains?
Reported averages (across three independent runs) include Gemini‑3.5‑Flash rising from 49.5% to 68.1% overall and Qwen‑3.6‑27B from 39.4% to 63.3%. The largest per‑task gains appeared on LiveMath and SpreadSheet. The paper notes table entries are averages and highlights top scores, but readers should consult the paper for full variance details.
-
Can skills be reused across models?
Often. The paper reports that skills developed by one model frequently transfer to others and sometimes outperform the receiving model’s native skills, though transferability depends on task and model differences.
-
What are the biggest operational risks?
Main risks are privacy/PII leakage from traces, wiki poisoning or malicious edits, accumulation of brittle heuristics, unclear validation protocols in production, compute and storage costs, and legal/IP considerations for transferred skills.
Parting note
WikiSkill treats agent improvement as knowledge engineering: a writable, inspectable set of procedures rather than opaque weight updates. That is a practical, lower‑risk way to get better-behaving agents quickly, especially for procedure‑dense business tasks like spreadsheets, workflows and tool orchestration. It is not a universal fix: governance, validation, privacy, and adversarial resilience are mandatory engineering workstreams, not optional extras.
As Andrej Karpathy framed it, the “LLM Wiki” idea puts experience where humans can audit and improve it. WikiSkill shows how to operationalize that insight for agents, a useful pattern to adopt and harden in production systems.