<!--
Sitemap:
- [Installation](/installation): Install the Bento daemon on macOS or Linux, then write your first pipeline to run a coding agent on a GitHub, Linear, or schedule event.
- [Upgrading](/upgrading): Version-specific steps for upgrading an existing Bento install.
- [Concepts](/concepts)
- [Build your first pipeline](/tutorials/pipeline-args)
- [Target a specific issue or PR from a URL](/tutorials/url-targeting)
- [Keep state across runs](/tutorials/pipeline-state)
- [Fire a pipeline on a schedule or on demand](/tutorials/schedule-and-fire)
- [Deploy a box to Railway](/tutorials/deploy-to-railway)
- [Operate a hosted daemon](/tutorials/operate-a-hosted-daemon)
- [Configuration](/configuration): Configure the Bento daemon with daemon.yaml: service name, port, webhooks, sandboxes, knowledge base, and environment variables.
- [Members](/members)
- [Knowledge base](/knowledge-base/)
- [Method and delivery](/knowledge-base/modes): Choose how Bento retrieves knowledge (BM25, vector, or hybrid) and how it delivers that knowledge to a coding agent.
- [Config](/knowledge-base/config)
- [MCP](/knowledge-base/mcp): Give coding agents your team knowledge over MCP. The Bento daemon serves an ask_knowledge search tool and per-file resources.
- [Pipeline configuration reference](/pipelines/config): Reference for Bento pipeline YAML: triggers, agents, skills, instructions, filters, sandboxes, structured results, and outputs.
- [Filters](/pipelines/filters): Filter GitHub, Linear, and webhook events by repository, label, and other fields so only the right Bento pipelines run.
- [Triggers](/triggers/): Start a Bento pipeline from a GitHub, Linear, webhook, schedule, or manual trigger. Compare the trigger types and when each one fires.
- [GitHub trigger](/triggers/github): Run a coding agent on GitHub pull request, issue, and other webhook events. Configure HMAC verification and event patterns.
- [Linear trigger](/triggers/linear): Run a coding agent on Linear issue and comment events. Configure the Linear webhook, signature verification, and Hookdeck relay.
- [Webhook trigger](/triggers/webhook): Fire a Bento pipeline from any HTTP POST to /webhooks/{name}, with bearer-token authentication declared in daemon.yaml.
- [Schedule trigger](/triggers/schedule)
- [Manual trigger](/triggers/manual): Fire any Bento pipeline on demand with bento trigger fire, and pass typed inputs to a pipeline that runs only on request.
- [Traces](/pipelines/traces)
- [Slack](/integrations/slack)
- [Public access](/public-access): Expose the Bento daemon to GitHub webhooks through a Cloudflare Quick Tunnel or Tailscale Funnel, and compare the two options.
- [Context engineering](/context-engineering): Assemble each coding agent prompt from personas, policies, skills, and pipeline instructions. Learn which file each instruction belongs in.
- [Best practices](/best-practices)
- [Troubleshooting](/troubleshooting)
- [Architecture](/architecture/vision)
- [Workspaces](/workspaces)
- [Authentication](/authentication)
- [Identity](/identity)
- [Security](/security)
- [References](/references)
- [Changelog](/changelog): Bento release history.
- [CLI reference](/cli/)
- [Setup](/cli/setup)
- [Secrets](/cli/secrets)
- [Lifecycle](/cli/lifecycle)
- [Sandbox image](/cli/image)
- [Sandboxes](/cli/sandbox)
- [Observability](/cli/observability)
- [Diagnostics](/cli/diagnostics)
- [Triggers](/cli/triggers)
- [Workbench](/cli/workbench)
- [Auth](/cli/auth)
- [Knowledge](/cli/knowledge)
- [Evals](/cli/evals)
- [Bento docs: run coding agents from version-controlled pipelines](/index): Bento runs coding agents from pipelines you keep in your repository. Declare the trigger, sandbox, skill, and guardrails in YAML.
- [Runtime wrapper](/architecture/runtime-wrapper)
- [Skill evolve](/architecture/skill-evolve)
-->

# Skill evolve

Skill-evolve uses session history to evaluate proposed improvements to Bento skills and agent prompts.

Inspired by [Meta-Harness](https://github.com/stanford-iris-lab/meta-harness) (Stanford IRIS Lab) — which searches over model harnesses by proposing candidates, benchmarking them, and tracking a Pareto frontier. We apply the same pattern to our own skills and agent definitions, using real session logs as the evaluation dataset.

## The idea

Skills (SKILL.md) and agent personas (PERSONA.md) are hand-written today. They encode operational knowledge — how to review a repo, how to do a PR review, how to structure context for a project. But they are static. They do not learn from whether sessions using them actually went well.

Skill-evolve closes this loop:

```
Sessions happen → logs accumulate → evaluate outcomes →
propose skill improvements → validate offline → promote winners
```

The "dataset" is session JSONL files. The "harness" is skill/agent definitions. The "evaluator" is whether improved skills would have produced better outcomes on past sessions. The "proposer" is a Claude Code session that reads failure patterns and drafts improvements.

## Architecture

```
┌─────────────────────────────────────────────────┐
│                  Cron (weekly)                   │
│                                                  │
│  1. Collect    session logs from ~/.pi/sessions  │
│  2. Evaluate   score sessions by outcome signal  │
│  3. Analyze    identify failure patterns          │
│  4. Propose    generate skill candidates          │
│  5. Validate   test candidates against held-out   │
│  6. Promote    update frontier, notify            │
└─────────────────────────────────────────────────┘
         │                          │
         ▼                          ▼
   evolution_summary.jsonl    frontier_skills.json
```

### Step 1: Collect

Gather session logs from the past period. Each session JSONL contains the full conversation: user messages, assistant responses, tool calls, tool results, errors. Filter to sessions that used a specific skill or agent.

### Step 2: Evaluate

Score each session. The outcome signal could be:

* **Completion** — did the session reach its goal? (heuristic: did the user say "thanks", approve a PR, or move on to a new topic without frustration?)
* **Efficiency** — tool call count, token usage, number of correction cycles ("no not that", "try again")
* **Error rate** — how many tool calls failed, how many retries
* **User corrections** — explicit feedback like "do not do X" or "that is wrong"

This is the hardest part. Unlike meta-harness's text classification accuracy or terminal-bench pass rate, our signal is noisy and subjective. Early versions use simple heuristics. Later versions use an LLM as judge.

### Step 3: Analyze

A proposer session reads:

* `evolution_summary.jsonl` — what skill variants have been tried, what worked
* `frontier_skills.json` — current best skill versions
* Session logs with low scores — what went wrong?
* Session logs with high scores — which patterns to preserve?

The proposer identifies recurring failure modes. Examples:

* "The review-repo skill does not tell the agent to check for monorepo workspace configs, leading to missed context in 4/12 sessions"
* "The reviewer agent keeps suggesting changes to generated files because the persona does not mention ignoring dist/"
* "Project context injection is missing repo branch conventions, causing agents to push to wrong branches"

### Step 4: Propose

The proposer generates skill candidates — concrete diffs to SKILL.md or PERSONA.md files. Each candidate has:

```json
{
  "name": "review-repo-v3",
  "base": "skills/review-repo/SKILL.md",
  "hypothesis": "Adding monorepo detection step will reduce missed-context errors",
  "diff_summary": "Added workspace detection step between structure and architecture sections",
  "file": "candidates/review-repo-v3/SKILL.md"
}
```

The proposer stores candidates in a staging area. It never applies them directly.

### Step 5: Validate

Replay past sessions against candidates. For each candidate skill:

* Take N held-out sessions that used the base skill
* Run them through a simulated evaluation: "given this session's initial request and the candidate skill, would the outcome improve?"
* This can be an LLM-as-judge comparison: show both the original session trace and what the new skill would have produced, ask which is better

Validation runs offline and changes no real sessions.

### Step 6: Promote

If a candidate beats the current frontier on held-out sessions:

* Update `frontier_skills.json`
* Append to `evolution_summary.jsonl`
* Notify via Telegram/Slack: "Skill review-repo updated: added monorepo detection (improved on 3/5 held-out sessions)"
* The actual skill file is NOT auto-updated — the human reviews and applies

## State files

Following meta-harness conventions:

### `evolution_summary.jsonl`

One line per evaluated candidate:

```json
{
  "iteration": 3,
  "skill": "review-repo-v3",
  "base_skill": "skills/review-repo/SKILL.md",
  "hypothesis": "Adding monorepo detection reduces missed-context errors",
  "score": 0.73,
  "delta": 0.12,
  "sessions_evaluated": 8,
  "outcome": "0.73 (+0.12)",
  "timestamp": "2026-05-02T10:00:00Z"
}
```

### `frontier_skills.json`

Current best version of each skill:

```json
{
  "review-repo": {
    "version": "v3",
    "score": 0.73,
    "candidate_path": "candidates/review-repo-v3/SKILL.md",
    "promoted_at": "2026-05-02T10:00:00Z"
  },
  "reviewer": {
    "version": "v1",
    "score": 0.61,
    "candidate_path": null
  }
}
```

## What gets evolved

| Artifact | Location | What changes |
|---|---|---|
| Skills | `skills/*/SKILL.md` | Workflow steps, anti-patterns, output format |
| Agent personas | `agents/*/PERSONA.md` | Focus areas, tone, review criteria |
| Project context | `~/.projects/*/project.json` | Workflow guidelines, injected instructions |
| System prompts | Extension-injected prompts | Context framing, constraints |

## Evaluation signals

The quality of skill-evolve depends entirely on the evaluation signal. Possible approaches, from simple to sophisticated:

### Tier 1: Heuristic (start here)

* Session length vs task complexity (shorter is better for simple tasks)
* Number of user corrections / "no" / "try again" messages
* Tool error rate
* Whether the session ended with apparent success

### Tier 2: LLM as judge

* Show a judge model the session transcript and ask: "Rate this session 1-5 on task completion, efficiency, and user satisfaction"
* Compare two sessions side-by-side: "Which skill produced a better outcome?"

### Tier 3: Outcome-linked

* If using Linear: did the linked issue get closed?
* If doing PR review: was the review accepted without revisions?
* If doing code changes: did CI pass on the first push?

## Constraints

* **Human in the loop.** Candidates are proposed but never auto-applied. The user reviews and promotes.
* **Anti-overfitting.** Keep skills general-purpose. No session-specific patches. The same anti-overfitting rules from Meta-Harness apply: no hardcoded knowledge about specific repos, tasks, or users.
* **Held-out split.** Always evaluate on sessions the proposer has not seen. Otherwise the evaluation rewards memorized failure cases.
* **Slow cadence.** Weekly or biweekly. Skills stay stable, because users depend on them. A skill that changes every day is worse than one that is slightly suboptimal.

## Dependencies

Before building skill-evolve, we need:

1. **runtime\_wrapper** — programmatic Claude Code / pi invocation with structured logging (adapted from meta-harness `claude_wrapper.py`)
2. **Session scoring** — even a basic heuristic scorer for session JSONL files
3. **Candidate staging** — a directory structure for skill variants that does not interfere with live skills

## Future: continuous learning

The end state is a closed loop where bento gets better at its job automatically:

```
User works with pi → sessions logged → skill-evolve runs weekly →
better skills proposed → human reviews → skills updated →
next week's sessions are better → repeat
```

This is Layer 3 of the vision (Company Brain) applied to bento itself. The proposed system improves how it applies the knowledge it stores.

## References

* [Meta-Harness paper](https://arxiv.org/abs/2603.28052) — the framework this is based on
* [Meta-Harness repo](https://github.com/stanford-iris-lab/meta-harness) — reference implementation
* [VISION.md](/architecture/vision) — how this fits into the broader bento architecture
