Knowledge audits

Tests that prompt a real box agent and check what it knows from the guidance a box loads, or (dev-guidance audits) what a dev session in this checkout knows from the repo's own guidance.

What it is

Tests that verify what agents in a box actually know — by prompting a real box agent, watching its tool use, and checking its response. Periodic execution catches knowledge drift as schemas, prompts, and conventions evolve.

The harness lives in src/dev/:

When to use: Verifying that documentation, agent guides, and conditional rules are working — that the agent has the right information at the right time. Not for testing system behavior. Run them after touching AGENTS.md, box AGENTS.md, schemas, prompts, or anything that changes what an agent should know; the periodic cadence is in maintenance. The latest full-corpus rerun record is docs/reports/knowledge-audit-rerun-2026-10-22.md.

How it works

The runner:

  1. Runs the prompt via bbx prompt in the test box
  2. Parses the session transcript to extract: files read, searches, bash commands, response text
  3. Automated checks: correct_contains (substring match), should_read/should_not_read (file access)
  4. Generates a Markdown report with results + blank assessment field for human review

Reports go to src/dev/reports/audit-report-<timestamp>.md; the durable record is the YAML status comment (see Reading results).

Knowledge levels

When we talk about what the agent "knows," there are distinct phenomena worth naming. These categories mix together where information lives, what retrieval strategy is needed, and what failure modes look like — but that's because they describe the distinct behaviors agents actually exhibit. Some are about the knowledge architecture (knows directly, knows about), some about retrieval effort (discoverable, deducible, researchable), and some about what goes wrong when retrieval doesn't happen (guessable, improvised). They don't form a tidy linear spectrum — each is a phenomenon you might encounter when testing or observing an agent, and a starting point for further investigation.

  1. Knows directly — Can answer without investigation. The information is directly in the agent's loaded context: AGENTS.md → .beebox/agent-guide.md, plus any .claude/rules/ files triggered by the current task. These answers should be immediate and accurate.

  2. Knows about — Knows that something exists and where to learn more. The agent guide references docs or files by path (e.g., "see node_modules/beebox/box-docs/card-doc.md"), so the agent can follow the pointer to get details. May require multiple hops of file reading (e.g., guide → table of contents → specific doc), but each hop is straightforward traversal — the agent knows where to go next without searching or guessing. Reliability depends on whether the agent actually follows the references vs. guessing from the name alone.

  3. Discoverable — Information is available locally but requires search or exploration to locate. For example, grepping docs for a keyword, listing directory contents, or reading config files. The agent isn't told where to look — it has to figure that out. Success depends on search strategy and how discoverable the information is.

  4. Deducible — Requires investigation and reasoning from local artifacts. For instance, reading beebox source code to understand how a feature works, or examining multiple files to piece together a procedure. The agent may not always succeed, and different agents might reach different (plausible) conclusions from the same evidence.

  5. Researchable — The information exists somewhere on the internet but not locally. The agent would need to use web search to find it. Examples: how a third-party API works, what format a particular standard uses, best practices for something the codebase doesn't document. Distinct from deducible because the answer can't be found by reading local files — it requires going outside the environment.

  6. Guessable — Appears to be answerable from general knowledge, but the correct answer for beebox may differ from the common/default answer. Dangerous because the agent will sound confident. Example: guessing how the agentic loop works based on general knowledge of agent systems, when beebox has specific conventions.

  7. Improvised — The agent constructs a plausible approach without checking if there's an established one. Unlike guessing (which is about facts), this is about strategy: the agent builds something that works but misses patterns or tools it should have used. Example: hand-parsing frontmatter with string splitting when beebox/cards already has a parser, or hand-rolling a card file instead of using bbx create. The result may actually function, which makes it harder to catch than a wrong guess — the problem is that it's not the right way, and it'll diverge from conventions. Often a fallback when the agent decides it can figure things out as it goes rather than looking up how things are done.

  8. Knows it doesn't know — The agent is aware of the gap. Something is acknowledged to exist but the agent genuinely lacks access to the information. Better than guessing — the agent can say "I don't have that information" or ask. Example: connector auth secrets live in _config/connectors/*.secret.* — ideally these would be inaccessible to the agent (not just forbidden), so the agent knows connectors need credentials but can't read the actual values. (Today the agent can read these files, which makes this "deducible" rather than true "doesn't know" — a gap in the access model.) Relatedly, a box's .claude/settings.json permission rules don't gate engine-spawned agents at all — runAgent hardcodes permissionMode: "bypassPermissions" (src/core/agent/invoke/run.ts) — so those rules only shape a human's interactive Claude Code session in the box, not the wakeup/procedure/chat runs the engine drives.

  9. Does not know — Beyond the agent's knowledge boundaries. Pursuing the question yields no answer. The agent may have given up while the information was still deducible.

Context shifts knowledge levels

The same information can sit at different levels depending on what the agent is currently doing. In Claude Code, this happens concretely through conditional rules:

This means testing should consider: what was the agent doing when it answered? A question about doc card structure might be "knows directly" mid-procedure but "knows about" in a cold prompt.

How the knowledge chain works in Claude Code

The agent's context is built in layers, each corresponding to a knowledge level:

Running it

npx tsx src/dev/knowledge-audit.ts run --box ~/src/boxes/test1 [--filter <tag-or-id>] [--engine claude|codex|pi]
npx tsx src/dev/knowledge-audit.ts list

--box takes the box root (~/src/boxes/test1) — there is only the one root to pass, and the context-history ledger keys off its basename (src/dev/lib/audit-box.ts). Pass an absolute path (or ~/…), never a bare name like test1, which would resolve inside the monorepo and be refused by the nested-box guard.

--filter takes one value, matched as an audit id, an id substring, or a tag; repeated flags keep only the last. The runner resets and cleans the box between tests, so it refuses a box with uncommitted changes. A worktree's test1 clone usually has such changes from ordinary dev use (an engine install sync, for example). Copy it to a scratch directory, commit there, and audit the copy.

The runner has no box server and no notification channel, and a worktree clone's node_modules/beebox links to the engine outside the box. Three kinds of audit fail for those reasons alone:

By default the runner uses the box's configured engine. --engine overrides that choice for the audit only, so the same definitions can be exercised against Claude and Codex without editing _config/box.json. Reports include the engine and use engine-qualified default filenames. Codex behavior is captured from the validated live app-server event stream; private rollout files are not parsed. Codex does not expose Claude-equivalent per-turn context snapshots on that surface, so Codex reports deliberately omit the context baseline rather than presenting incomparable usage as parity.

A pi audit reads the pi session file under .beebox/pi/sessions/ after the run (src/dev/lib/test-runner/runner/pi-audit.ts). pi's read counts as a file read, ls as a search, and bash as a command, classified as for Codex. The context baseline comes from the usage on each assistant message. pi needs an OpenRouter key granted to the box. To audit a copy of a box that has no grant, point BBX_SECRETS_FILE at a scratch store that grants the copy the placeholder pi-dev-placeholder as openrouter, and pass --pi-dev-key. The runner then gives pi BBX_OPENROUTER_API_KEY (environment or beebox/.env) in memory, in place of the placeholder (src/dev/lib/openrouter-dev-key.ts).

Dev-guidance audits

An entry with surface: dev audits a developer session instead of a box agent: what a Claude Code session in this monorepo checkout learns from the root and package AGENTS.md files, the project skills, and docs/. Its first questions cover the product spirit and design docs, which sessions kept forgetting.

pnpm knowledge-audit run --dev [--model <id>] [--filter <tag-or-id>]

--dev runs only surface: dev entries (a box run skips them) as claude -p --output-format stream-json in the checkout root, default model claude-opus-5-5. The session loads project settings and skills with hooks disabled, no MCP servers, and auto-memory off, so a pass means tracked guidance taught it; the user-level ~/.claude/CLAUDE.md still loads. Its settings deny reads of the answer key (knowledge-audits.yaml, the context ledger, src/dev/reports/, and scratch/). Its tools are Read, Grep, Glob, and Skill. A read counts only when its tool result is not an error, and a Skill call counts as a read of that skill's SKILL.md, so should_read paths are repo-relative (.claude/skills/bbx-design/SKILL.md, beebox/docs/box-work.md). The text checks are unchanged; card, Bash, search, fixture, and context-dir fields are refused at load. Mechanics: src/dev/lib/test-runner/dev-session.ts.

It cannot observe Codex sessions (AGENTS.md, .codex/ agents), what a session learns only by doing (hooks, lint and doc-check failures, tool errors), or interactive-session context such as the developer's memory and / commands. It asks cold questions; a session mid-task may load more. The session reads issues/, so a before-and-after comparison of a guidance change also hides the issue that states the rule.

Writing one

Each entry in knowledge-audits.yaml has these fields:

Example:

tests:
  - id: box-structure-inbox
    prompt: "Where would you look for unprocessed incoming items?"
    expected_level: knows_directly
    watch_for: "Names _content/inbox/ directly without searching"
    correct_contains: ["_content/inbox"]
    should_read: ["node_modules/beebox/box-docs/card-doc.md"]   # optional
    should_not_read: ["some/file.md"]              # optional
    tags: [navigation]

Prompt style effects

How you phrase a prompt changes which knowledge level the agent actually operates at. The same question can produce different behavior depending on whether the prompt encourages speed or thoroughness:

When testing, try the same question with different styles to see where the boundary is between "knows directly" and "knows about" — does adding "be brief" cause the agent to guess instead of looking things up? That reveals which knowledge is genuinely in context vs. just referenced.

Reading results

When running bbx prompt, watch for:

Recording results

After running audits, update the status comments in knowledge-audits.yaml with the date and results. Each test section (e.g., # === Don't Drop Important Information ===) should have a # Status (YYYY-MM-DD): comment noting:

The status comment is the durable record. Reports in reports/ are gitignored and ephemeral — they're a working artifact, not a result log.

Context size

Each report entry shows a Context line — the loaded-context size the box agent carried during that audit, read from the session log's per-turn usage (summed across input + cache_creation + cache_read, since prompt caching leaves raw input_tokens tiny). The initial number is the always-on baseline the box pays every turn (system prompt + agent-guide + box AGENTS.md + tool schemas); peak and added show how much answering grew it. A knows_directly / 0-read audit doubles as a baseline gauge: trim a box's always-on context, re- run, watch initial drop. Logic lives in lib/context-usage.ts.

Every run also appends these numbers to src/dev/context-history.yaml — a committed ledger keyed by box → audit id → entries, each stamped with the date, the box's HEAD, and the monorepo's HEAD. The file's git history is the trend line: an AGENTS.md trim that drops the baseline shows up as a diff. (boxCommit is the box's HEAD before the harness regenerates docs — stable for a current box, but the throwaway worktree box clone is behind upstream templates, so its hash moves each run.) Logic lives in lib/context-history.ts.

Interpreting failures

Common failure patterns and what they mean:

When fixing failures, prefer changing docs/prompts to changing the test — the test is asserting an expectation about agent behavior, and silently weakening it defeats the point.