<!--
Sitemap:
- [Installation](/installation): Install the Bento daemon on macOS or Linux, then write your first pipeline to run a coding agent on a GitHub, Linear, or schedule event.
- [Upgrading](/upgrading): Version-specific steps for upgrading an existing Bento install.
- [Concepts](/concepts)
- [Build your first pipeline](/tutorials/pipeline-args)
- [Target a specific issue or PR from a URL](/tutorials/url-targeting)
- [Keep state across runs](/tutorials/pipeline-state)
- [Fire a pipeline on a schedule or on demand](/tutorials/schedule-and-fire)
- [Deploy a box to Railway](/tutorials/deploy-to-railway)
- [Operate a hosted daemon](/tutorials/operate-a-hosted-daemon)
- [Configuration](/configuration): Configure the Bento daemon with daemon.yaml: service name, port, webhooks, sandboxes, knowledge base, and environment variables.
- [Members](/members)
- [Knowledge base](/knowledge-base/)
- [Method and delivery](/knowledge-base/modes): Choose how Bento retrieves knowledge (BM25, vector, or hybrid) and how it delivers that knowledge to a coding agent.
- [Config](/knowledge-base/config)
- [MCP](/knowledge-base/mcp): Give coding agents your team knowledge over MCP. The Bento daemon serves an ask_knowledge search tool and per-file resources.
- [Pipeline configuration reference](/pipelines/config): Reference for Bento pipeline YAML: triggers, agents, skills, instructions, filters, sandboxes, structured results, and outputs.
- [Filters](/pipelines/filters): Filter GitHub, Linear, and webhook events by repository, label, and other fields so only the right Bento pipelines run.
- [Triggers](/triggers/): Start a Bento pipeline from a GitHub, Linear, webhook, schedule, or manual trigger. Compare the trigger types and when each one fires.
- [GitHub trigger](/triggers/github): Run a coding agent on GitHub pull request, issue, and other webhook events. Configure HMAC verification and event patterns.
- [Linear trigger](/triggers/linear): Run a coding agent on Linear issue and comment events. Configure the Linear webhook, signature verification, and Hookdeck relay.
- [Webhook trigger](/triggers/webhook): Fire a Bento pipeline from any HTTP POST to /webhooks/{name}, with bearer-token authentication declared in daemon.yaml.
- [Schedule trigger](/triggers/schedule)
- [Manual trigger](/triggers/manual): Fire any Bento pipeline on demand with bento trigger fire, and pass typed inputs to a pipeline that runs only on request.
- [Traces](/pipelines/traces)
- [Slack](/integrations/slack)
- [Public access](/public-access): Expose the Bento daemon to GitHub webhooks through a Cloudflare Quick Tunnel or Tailscale Funnel, and compare the two options.
- [Context engineering](/context-engineering): Assemble each coding agent prompt from personas, policies, skills, and pipeline instructions. Learn which file each instruction belongs in.
- [Best practices](/best-practices)
- [Troubleshooting](/troubleshooting)
- [Architecture](/architecture/vision)
- [Workspaces](/workspaces)
- [Authentication](/authentication)
- [Identity](/identity)
- [Security](/security)
- [References](/references)
- [Changelog](/changelog): Bento release history.
- [CLI reference](/cli/)
- [Setup](/cli/setup)
- [Secrets](/cli/secrets)
- [Lifecycle](/cli/lifecycle)
- [Sandbox image](/cli/image)
- [Sandboxes](/cli/sandbox)
- [Observability](/cli/observability)
- [Diagnostics](/cli/diagnostics)
- [Triggers](/cli/triggers)
- [Workbench](/cli/workbench)
- [Auth](/cli/auth)
- [Knowledge](/cli/knowledge)
- [Evals](/cli/evals)
- [Bento docs: run coding agents from version-controlled pipelines](/index): Bento runs coding agents from pipelines you keep in your repository. Declare the trigger, sandbox, skill, and guardrails in YAML.
- [Runtime wrapper](/architecture/runtime-wrapper)
- [Skill evolve](/architecture/skill-evolve)
-->

# Observability

## `bento daemon status`

Show daemon process status and probe the local and public-tunnel endpoints for reachability.

```bash
bento daemon status
```

```
daemon: pid=12345
local   : reachable (4ms) — bento uptime=3600s
tunnel  : not configured (no `tunnel:` block in .bento/daemon.yaml)
```

When a tunnel is configured, the `tunnel` line reports the public URL and its latency — see [Public access](/public-access).

## `bento daemon logs [target]`

Show daemon logs, or logs for a specific agent.

```bash
bento daemon logs            # daemon logs
bento daemon logs reviewer   # agent-specific logs
bento daemon logs -f         # follow mode
bento daemon logs -n 100     # last 100 lines
```

| Flag | Description |
|------|-------------|
| `-f`, `--follow` | Tail logs in real time |
| `-n`, `--lines <lines>` | Number of lines to show (default: 50) |
| `--agent` | Read the most recent agent run's stdout, plus any `setup:` step output recorded for the same invocation. Combine with a positional `target` (run ID) to select a specific run. |
| `--render flat` | Collapse stream-json events to readable per-turn text |
| `--render raw` | Raw NDJSON (default) |
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

`--agent` without `--follow` reads through the daemon's artifacts endpoint, addressed by invocation id — it works against a remote daemon (`BENTO_DAEMON_URL`). `--agent --follow` stays a local `tail -f` on the daemon host's stdout file, since streaming a remote tail is out of scope. It only works when the CLI runs on the same host as the daemon. Either mode falls back to the stderr of the run when stdout is empty, because a wedged agent leaves stdout empty and stderr carries the cause.

## `bento trace <trigger-id>`

Print a full lifecycle trace for a trigger: trigger, workload, invocations, agents. The command resolves through the daemon (`GET /trace/:id`), so it works against a remote daemon, but it needs a reachable one.

The id accepts a prefix. Give 6 or more characters after the kind (`trg_`, `run_`, `inv_`, or `tsk_`). This is the length of the run id in the log tag column, which pastes in directly. If the prefix matches more than one trigger, the command shows each candidate and stops. When a prefix resolves, the command prints `resolved <prefix> → <full id>` on stderr, so the trace always shows which id it used.

```bash
bento trace <trigger-id>
bento trace <trigger-id> --agent-output turns
```

| Flag | Description |
|------|-------------|
| `--json` | Output as JSON. |
| `--agent-output none\|final\|turns\|raw` | Agent output to render per invocation: nothing, final result text (default), parsed turns, or raw NDJSON. |
| `--verbose` | Show full results, lifecycle timestamps, metadata, log paths, and per-file coverage. Without it, the command prints a compact summary. |
| `--no-messages` | Deprecated alias for `--agent-output none`. |
| `--notes` | Show what each invocation recorded, and what the thread as a whole knows |
| `--notes-limit <n>` | Newest thread notes to show with `--notes` (default: 20, max: 200) |
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

`--notes-limit` bounds the `Thread notes` block only. The per-invocation block is everything that run recorded. The thread block is the newest `n` notes on the workspace, still rendered oldest-first, so raising the limit adds older notes above the ones already shown.

## `bento graph list`

List the live BIP-32 work graphs the daemon holds. A work graph is a durable plan: nodes are units of work, edges order them, and the daemon executes the plan across runs. The command is read-only. Cancel and retry belong to the driver, and this surface never writes a row.

```bash
bento graph list
bento graph list --pipeline issue-solve
bento graph list --source 'github:1a35e1/bento#582'
bento graph list --source 'github:1a35e1/bento#582' --settled
bento graph list --source 'github:1a35e1/bento#582' --settled --limit 5 --offset 20
```

```
wg_3kqmzz3na316vdm3  github:1a35e1/bento#582
  pipeline=issue-solve owner=agentic status=active frontier=1 age=1h
  nodes: pending=1 completed=1
```

`owner` says who claims the nodes. `agentic` means the owning pipeline declares `options.graph`, so the pipeline claims its own nodes and the daemon only resumes it. `driver` means the daemon dispatches each node itself. The label reads the pipeline configuration the daemon holds now, not the configuration the graph ran under. For a settled graph whose pipeline changed or was removed, the label is wrong. `frontier` is the count of nodes that are ready to claim now, and a settled graph shows `settled=<time>` in its place.

`--settled` shows the finished graphs for a source instead of its live one. Use it after a re-ingest, when you must read the previous attempt. It needs `--source`, because a settled graph has no other address. The history is newest first and paged: `--limit` sets the page size (200 at most, 50 by default) and `--offset` skips that many graphs. A footer reads `showing N of M`, where `M` is every settled graph the source has. Step `--offset` when `N` is short of `M`. The live listing is not paged, so it prints no footer and the two flags do nothing there.

| Flag | Description |
|------|-------------|
| `--pipeline <name>` | Only graphs owned by this pipeline |
| `--source <source>` | Only the graph for this ingest source |
| `--settled` | The source's settled graphs instead of its live one (needs `--source`) |
| `--limit / --offset` | Page the settled history (limit ≤ 200, default 50) |
| `--json` | Output as JSON. |
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

## `bento graph show <id|source>`

Show one graph: its nodes with status, attempt, and claimant, its edges, the frontier, the blocked nodes with their blockers, and the fold. The fold (`Fold:`) is the graph status as of this read. `active` means work is still live. It is not a prediction. `show` renders the same `GraphView` the orchestrator projects through its `read_graph` tool: the nodes, the edges, the frontier, the blocked set, and the fold. It is a pure read. Unlike `read_graph`, it does not repair stranded claims and it does not release due retries. A graph that had no driver pass and no orchestrator read since a run died can therefore show a stranded claim that the next `read_graph` call repairs.

Address the graph by its id (`wg_…`) or by its ingest source (`github:owner/repo#123`). A source resolves to its live graph. If the source has no live graph, the command shows the most recent settled one and says so on stderr.

```bash
bento graph show wg_3kqmzz3na316vdm3
bento graph show 'github:1a35e1/bento#582'
```

| Flag | Description |
|------|-------------|
| `--json` | Output as JSON. |
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

## `bento graph node <id|source> <key>`

Show one node in full: its brief, its `produced` record, its diagnostic, its blockers, and the invocation that claimed it.

The `produced` record is printed whole. It is the hand-off a later run acts on, and a truncated hand-off misleads. The command ends with the `bento trace` line for the claiming invocation. Run that to follow the node into its trigger, workload, invocation, and tasks. The graph surface does not repeat that lineage.

The lineage goes one way only. There is no command that takes a run and lists the nodes it claimed. To read that direction, query the `work_nodes` table with `psql`.

```bash
bento graph node wg_3kqmzz3na316vdm3 752
```

```
  Run: bento trace inv_8f3kq9mz2px4r7nt
```

| Flag | Description |
|------|-------------|
| `--json` | Output as JSON. |
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

## `bento queue status`

Show the Graphile Worker queue state, the job counts, and one table row for each live job.

```bash
bento queue status
```

```
state:     running
active:    1
waiting:   3
delayed:   0
capacity:  host 1/2 busy, remote 0/2 busy

STATE    LANE  PIPELINE         TRIGGER               WORKSPACE                                   AGE    ATTEMPTS
active   host  rebase-stale-pr  trg_5t9dfbjp5d9nrkb   github:1a35e1/bento:fix/986-honor-pipeline-tz  1h11m  1/1
waiting  host  pr-automerge     trg_bdynwzb57vpvbyj5  github:1a35e1/bento:feat/bnt-247-answer       1h2m   0/1
waiting  -     linear-triage    -                     schedule:linear-triage                      1h4m   0/1
waiting  -     heartbeat        -                     schedule:heartbeat                          1h9m   0/1
```

The default summary and table show only active, waiting, and delayed jobs, in that order. Each state is newest-first. Failure counts and rows require `--verbose`.

The `capacity` line shows the busy workers of each lane over the lane's configured concurrency. Each active job keeps one worker busy, and a schedule tick counts against `host`. If a lane has fewer live workers than its configured concurrency, the line also shows how many workers are down, for example `host 0/2 busy (1 down)`. A worker that died shows in this count even when no job runs. A lane with no live worker on a running queue claims no job, so the command prints `no live worker on lane <lane> — jobs on it will not start` for it. `bento doctor` fails its queue check for the same condition, and `/health` reports `degraded`. Run `bento daemon restart` to recover the lane. The line is absent while the queue is paused or not yet claiming, and from a daemon older than the CLI.

```bash
bento queue status --verbose --since 1d --limit 50 --offset 0
```

Verbose output adds the failed count for the selected window, the total exhausted jobs on the queue, and matching failed rows. It also shows consecutive failures when the count is nonzero. The footer reports how many matching failed rows appear on the page.

The history window defaults to 24 hours. It starts at the failure, not at enqueue. `--since` accepts an ISO timestamp or shorthand with `m`, `h`, or `d`, such as `30m`, `24h`, or `1d`. History pages default to 50 rows, with a maximum of 200. These filters apply only with `--verbose`; live jobs remain visible regardless of their age or the history page.

An exhausted job stays on the queue until you run `bento queue purge --exhausted`. The queue deletes completed jobs, so verbose output states that completed history is not retained.

A run job shows its lane, trigger ID, and workspace key. A schedule tick shows `-` for the lane and trigger, and the `schedule:<pipeline>` queue it holds. `AGE` measures time since execution started for active jobs, failure for exhausted jobs, or enqueue for waiting and delayed jobs. `ATTEMPTS` is `attempts/max_attempts`.

| Flag | Description |
|------|-------------|
| `--verbose` | Include failed history, consecutive failure counts, and completion information. |
| `--since <time>` | Failed history window; ISO timestamp or `m`/`h`/`d` shorthand. Default: `24h`. Applies only with `--verbose`. |
| `--limit <number>` | Failed rows per page, from 1 to 200. Default: `50`. Applies only with `--verbose`. |
| `--offset <number>` | Failed rows to skip. Default: `0`. Applies only with `--verbose`. |
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

## `bento queue resume`

Resume the queue after repeated job failures cause the worker to pause.

If startup readiness is pending or failed, the command refuses to resume. The HTTP endpoint returns `409` and names the unmet prerequisite. Resolve the reported problem, then restart the daemon. See [startup readiness](/cli/lifecycle#startup-readiness).

```bash
bento queue resume
```

| Flag | Description |
|------|-------------|
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

## `bento queue reconcile`

Move dead in-flight work to a terminal status: invocations, then workloads, then triggers, in that order so each stage settles the evidence the next one reads. The command never touches work that the running daemon owns, so it is safe on a live instance. The same cascade runs automatically at daemon boot.

```bash
bento queue reconcile
```

```
reconciled 3 stranded row(s)
  invocations: 1
  workloads:   1
  triggers:    1
```

A reconciled trigger reports `error` with an `agent_timeout` code when its workload timed out, and `done` when the workload completed cleanly. Every other case reports `error` with a `trigger_orphaned` code. Each one carries a `reconciled:` note, which `bento trigger list` and `bento trace` show.

| Flag | Description |
|------|-------------|
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

## `bento queue purge --exhausted`

Delete exhausted jobs — `attempts >= max_attempts`, never runnable again. These are the rows `bento queue status` counts as `failed`. The flag is required. The command purges nothing else.

```bash
bento queue purge --exhausted
```

| Flag | Description |
|------|-------------|
| `--exhausted` | Required — purge the exhausted jobs |
| `--instance <name>` | Target daemon instance (default: cwd config, else the running instance) |

## `bento ps`

Show all running bento daemon instances and their agent processes, plus the paths to each instance's config, data, and log directories.

```bash
bento ps
```

```
bento  pid=12345  running
  conf: /path/to/project/.bento
  data: /Users/you/.bento/bento
  log:  /Users/you/.bento/bento/daemon.log
  agent pid=12346  name=reviewer
```

| Flag | Description |
|------|-------------|
| `--instance <name>` | Show a single daemon instance by name. Without it, the command lists every running instance. |

## `bento invocation`

Inspect agent invocations and their attempt logs.

### `bento invocation list`

List persisted agent invocations.

```bash
bento invocation list
bento invocation list --repo acme/web --pr 42
bento invocation list --pipeline pr-review --status failed
```

| Flag | Description |
|------|-------------|
| `--repo <owner/name>` | Filter by repository |
| `--pr <number>` | Filter by pull request number |
| `--pipeline <name>` | Filter by pipeline |
| `--status <state>` | Filter by status: `pending`, `running`, `completed`, `failed`, `cancelled` |
| `--limit <n>` | Max results (default: 50, max: 200) |
| `--offset <n>` | Skip this many results, to page past the limit (default: 0) |
| `--instance <name>` | Target daemon instance |

The last line reports `showing N of M`, where `M` is the number of invocations that match the filters across every page, not the number of rows on this page. When `M` is larger than `N`, raise `--offset` by `--limit` to read the next page.

### `bento invocation status`

Summarize invocation counts by state.

```bash
bento invocation status
```

```
pending:     0
running:     1
completed:   23
failed:      2
cancelled:   0
```

The daemon groups the counts in PostgreSQL and returns them from `GET /invocations/stats`. They cover the invocations `bento invocation list` reports, those whose workload and trigger are still retained, so they stay correct past the 50-row page the list returns. A state the daemon matched no rows for prints `0`. Pass `--format json` for `{ counts, total }`; use `bento invocation list --format json` when you want the rows themselves.

| Flag | Description |
|------|-------------|
| `--instance <name>` | Target daemon instance |

### `bento invocation show <id>`

Show one invocation, its tasks, and log artifact paths.

```bash
bento invocation show inv_abc123
```

| Flag | Description |
|------|-------------|
| `--instance <name>` | Target daemon instance |

### `bento invocation get-run <run-id>`

Read the same structured run record that an agent can retrieve through MCP `get_run`. The default output is the final reply, which is empty while the run is still in flight. Pass `--format json` to print the run ID, invocation ID, lifecycle status, the diagnostic that ended a failed run, final output, bounded result, and durable notes. The command fails only when no run carries that ID. A pending or running run returns a record that carries its status.

The command reads any run on the daemon by design. The `workloads:read` scope is the whole gate, and no workspace scope narrows it. `bento invocation list` and `bento invocation show` are daemon-wide under the same scope.

```bash
bento invocation get-run run_abc123
bento invocation get-run run_abc123 --format json
```

| Flag | Description |
|------|-------------|
| `--format json` | Print the complete structured run record |
| `--instance <name>` | Target daemon instance |

### `bento invocation logs <id>`

Read the stdout log from one invocation attempt. The command reads through the artifacts endpoint of the daemon, so it works against a remote daemon. It addresses the log by invocation id, never by a guessed filesystem path.

```bash
bento invocation logs inv_abc123
bento invocation logs inv_abc123 --attempt 2
bento invocation logs inv_abc123 --task tsk_1
bento invocation logs inv_abc123 --stderr
```

| Flag | Description |
|------|-------------|
| `--attempt <n>` | Attempt number to read (default: 1). Ambiguous on an orchestrated run whose sub-tasks each restart their own attempt counter — use `--task` instead. |
| `--task <id>` | Address one artifact by task id directly, bypassing `--attempt`. `bento trace <trigger-id>` lists the task ids. |
| `--stderr` | Read stderr instead of stdout |
| `--instance <name>` | Target daemon instance |
