EvoSkill: Automated Skill Induction from Agent Failures

Tldr; We introduce EvoSkill, a task-agnostic framework that induces non-trivial agent skills from agents’ execution failures. EvoSkill analyzes execution failures to automatically discover non-trivial structured, reusable skills, improving coding agents’ (Claude Code, Codex, etc…) performance by up to 50% accuracy and transferring zero-shot to unseen benchmarks.

---

Coding agents such as Claude Code, Codex, and OpenHands have become highly capable general-purpose problem solvers. By treating code as a flexible interface to tools, APIs, and external environments, they can tackle a wide range of tasks with minimal task-specific engineering. As the scope and complexity of tasks grow, particularly for long-horizon problems (e.g. tasks requiring multiple reasoning steps, tool invocations, and cross-referenced outputs) a single monolithic agent is overloaded with different tasks causing errors to compound silently across steps and domain-specific expertise that a human practitioner would bring to bear is absent.

A powerful abstraction that has emerged to address this lack of specialization is Skills: structured, reusable protocols that inject specialized knowledge into an agent at task time. Rather than expecting one agent to be an expert at everything, the skill paradigm delegates domain-specific procedures to composable sub-routines while the main agent orchestrates. More concretely, a skill can be defined as:

• A human-readable Markdown playbook with explicit step-by-step procedures and verification checks
• Optional bundled scripts (Python, Bash) and reference artifacts that the agent can invoke during execution

In this work, we introduce EvoSkill: an open-source task-agnostic framework that automatically discovers agent skills to improve performance. EvoSkill runs an agent over a task, extracts the trajectories of failed episodes, and synthesizes candidate skill packages.

We evaluate EvoSkill on two benchmarks: OfficeQA, a grounded reasoning task that closely reflects enterprise needs, EvoSkill improves Claude Code (Opus 4.5) from 60.6% to 67.9% exact-match accuracy (+7.3%) using only a small training subset (10% of the dataset). Similarly, on SealQA, a search-augmented QA benchmark with noisy retrieval, EvoSkill yields a 12.1% absolute improvement (26.6% → 38.7%). More notably, skills from SealQA transfer zero-shot to BrowseComp with no modification, improving accuracy by 5.3%. This transfer suggests that EvoSkill learns reusable capabilities rather than task-specific heuristics.

---

EvoSkill runs an evolutionary loop with three specialized agents working together: an Executor, a Proposer, and a Skill Builder.

1. Execute and observe failures: The base agent (Executor) attempts a batch of benchmark questions using its current configuration of a system prompt + whatever skills it has accumulated so far. Answers are scored against ground truth.

2. Analyze what went wrong: Failed attempts are handed to the Proposer, which receives the full trajectory (e.g. user messages, assistant responses, tool calls, tool outputs), intermediate artifacts and the agent's final answer. The Proposer:

• Identifies recurring failure patterns across the batch of failed episodes
• Cross-references those patterns against a feedback history of previous proposals (including what worked and what was discarded)
• Proposes a targeted skill to address the most impactful common gap

3. Build the skill: The Skill Builder takes the Proposer's high-level description and implements it as a structured skill file. Skills live as Markdown files in a .claude/skills/ directory, making them inspectable and version-controlled.

4. Evaluate the candidate: The new agent configuration (base prompt + all current skills, including the new one) is scored on a held-out validation set. If the skill improves validation performance, it advances; if not, it is discarded.

5. Update the frontier: EvoSkill maintains a frontier of the top-N performing agent configurations (default N=3), tracked as git branches. Each candidate configuration is stored as a git branch, making the evolutionary history fully reproducible. If the new configuration scores high enough to enter the frontier, it survives to be a parent in the next iteration. If not, it's discarded. All proposals are recorded in the feedback history so future Proposers can learn from the dead end.

This cycle repeats for a configurable number of iterations, iteratively improving validation performance under an explicit selection criterion. The key mechanisms that make it work:

• Failure-driven search: Proposals are grounded in real execution trajectories, not random mutations. The system only proposes changes when it has evidence of a specific capability gap.
• Feedback memory: The Proposer sees what was tried before, what improved scores, and what regressed. This prevents the loop from going in circles.
• Frontier selection: Rather than committing to a single best configuration, EvoSkill maintains diversity by keeping the top-N configurations. Different frontier members can be selected as parents for new mutations, preventing premature convergence.

---

We evaluated EvoSkill on three benchmarks: OfficeQA (document-grounded QA), SealQA (search-augmented reasoning ), BrowseComp (open web research) in a zero-shot setting.

Key findings:

• Cross-category generalization: On OfficeQA, skills discovered for financial table parsing also improved demographic data extraction; merging skills across categories produced the strongest configurations.
• Search strategy discovery: The largest gain (+12.1 pp on SealQA) came from skills the system induced for query reformulation, multi-source verification, and structured search persistence. These ideas have shown up many times in different systems that have led to improvements in overall search performance.
• Zero-shot transfer: SealQA skills improved BrowseComp by +5.3 pp with no adaptation — the skills encode general research strategies (how to search persistently, how to verify across sources), not task-specific patterns.

---

We believe that skills are a better abstraction to optimize over compared to just prompts or code (e.g. AlphaEvolve or GEPA). Prompts can shift priorities but can't encode a multi-step procedure; code evolution produces opaque artifacts that are hard to share across frameworks. Skills sit in the sweet spot — structured enough to encode real procedures with branching logic and verification steps, readable enough that a developer can open one, understand it, edit it, and hand it to a different agent on a different model. The zero-shot transfer results hint at where this is heading: agents sharing skill libraries the way developers share packages.

Where we're headed:

• Broader domains: We want to test whether failure-driven skill induction can drive meaningful gains in other settings such as coding, multimodal reasoning, and other interesting domains where the gap between general capabilities and domain-specific expertise widens with increasing complexity.
• Continual adaptation: The current loop runs offline over a fixed dataset. A natural next step is online skill induction, where agents accumulate and refine skills on the fly as they encounter new failure modes in production; closing the gap between deployment and model improvement.
• Model training with skills: Induced skills are structured, validated, and grounded in real execution traces — properties that make them a compelling source of training signal. We see an opportunity to use skill libraries not just as inference-time context, but as supervision for fine-tuning, distillation, or reinforcement learning, creating a feedback loop where better skills produce better base models that in turn discover better skills.

[1]: See the paper preprint for the full evaluation protocol and ablations.

---

* Browsecomp was run on a stratified subset of 128 examples using BrowseComp-Small recipe