Bee Box documentation · directory: https://beebox.run/docs/dev/testing/ · index: https://beebox.run/docs/dev/testing/index.md · root: https://beebox.run/llms.txt # Knowledge audits Tests that prompt a real box agent and check what it knows from the guidance a box loads, or (dev-guidance audits) what a dev session in this checkout knows from the repo's own guidance. ## What it is Tests that verify what agents in a box actually know — by prompting a real box agent, watching its tool use, and checking its response. Periodic execution catches knowledge drift as schemas, prompts, and conventions evolve. The harness lives in `src/dev/`: - `knowledge-audit.ts` — CLI entry - `knowledge-audits.yaml` — test definitions - `lib/test-runner.ts`, `lib/report.ts`, `lib/session-report.ts` — internals - `reports/` — gitignored output **When to use:** Verifying that documentation, agent guides, and conditional rules are working — that the agent has the right information at the right time. Not for testing system behavior. Run them after touching AGENTS.md, box AGENTS.md, schemas, prompts, or anything that changes what an agent should know; the periodic cadence is in [maintenance](https://beebox.run/docs/dev/maintenance.md). The latest full-corpus rerun record is `docs/reports/knowledge-audit-rerun-2026-10-22.md`. ## How it works The runner: 1. Runs the prompt via `bbx prompt` in the test box 2. Parses the session transcript to extract: files read, searches, bash commands, response text 3. Automated checks: `correct_contains` (substring match), `should_read`/`should_not_read` (file access) 4. Generates a Markdown report with results + blank assessment field for human review Reports go to `src/dev/reports/audit-report-.md`; the durable record is the YAML status comment (see Reading results). ### Knowledge levels When we talk about what the agent "knows," there are distinct phenomena worth naming. These categories mix together where information lives, what retrieval strategy is needed, and what failure modes look like — but that's because they describe the distinct behaviors agents actually exhibit. Some are about the knowledge architecture (knows directly, knows about), some about retrieval effort (discoverable, deducible, researchable), and some about what goes wrong when retrieval doesn't happen (guessable, improvised). They don't form a tidy linear spectrum — each is a phenomenon you might encounter when testing or observing an agent, and a starting point for further investigation. 1. **Knows directly** — Can answer without investigation. The information is directly in the agent's loaded context: `AGENTS.md` → `.beebox/agent-guide.md`, plus any `.claude/rules/` files triggered by the current task. These answers should be immediate and accurate. 2. **Knows about** — Knows *that* something exists and *where to learn more*. The agent guide references docs or files by path (e.g., "see `node_modules/beebox/box-docs/card-doc.md`"), so the agent can follow the pointer to get details. May require multiple hops of file reading (e.g., guide → table of contents → specific doc), but each hop is straightforward traversal — the agent knows where to go next without searching or guessing. Reliability depends on whether the agent actually follows the references vs. guessing from the name alone. 3. **Discoverable** — Information is available locally but requires search or exploration to locate. For example, grepping docs for a keyword, listing directory contents, or reading config files. The agent isn't told where to look — it has to figure that out. Success depends on search strategy and how discoverable the information is. 4. **Deducible** — Requires investigation and reasoning from local artifacts. For instance, reading beebox source code to understand how a feature works, or examining multiple files to piece together a procedure. The agent may not always succeed, and different agents might reach different (plausible) conclusions from the same evidence. 5. **Researchable** — The information exists somewhere on the internet but not locally. The agent would need to use web search to find it. Examples: how a third-party API works, what format a particular standard uses, best practices for something the codebase doesn't document. Distinct from deducible because the answer can't be found by reading local files — it requires going outside the environment. 6. **Guessable** — Appears to be answerable from general knowledge, but the correct answer for beebox may differ from the common/default answer. Dangerous because the agent will sound confident. Example: guessing how the agentic loop works based on general knowledge of agent systems, when beebox has specific conventions. 7. **Improvised** — The agent constructs a plausible approach without checking if there's an established one. Unlike guessing (which is about facts), this is about strategy: the agent builds something that works but misses patterns or tools it should have used. Example: hand-parsing frontmatter with string splitting when `beebox/cards` already has a parser, or hand-rolling a card file instead of using `bbx create`. The result may actually function, which makes it harder to catch than a wrong guess — the problem is that it's not the *right* way, and it'll diverge from conventions. Often a fallback when the agent decides it can figure things out as it goes rather than looking up how things are done. 8. **Knows it doesn't know** — The agent is aware of the gap. Something is acknowledged to exist but the agent genuinely lacks access to the information. Better than guessing — the agent can say "I don't have that information" or ask. Example: connector auth secrets live in `_config/connectors/*.secret.*` — ideally these would be inaccessible to the agent (not just forbidden), so the agent knows connectors need credentials but can't read the actual values. (Today the agent *can* read these files, which makes this "deducible" rather than true "doesn't know" — a gap in the access model.) Relatedly, a box's `.claude/settings.json` permission rules don't gate engine-spawned agents at all — `runAgent` hardcodes `permissionMode: "bypassPermissions"` (`src/core/agent/invoke/run.ts`) — so those rules only shape a human's interactive Claude Code session in the box, not the wakeup/procedure/chat runs the engine drives. 9. **Does not know** — Beyond the agent's knowledge boundaries. Pursuing the question yields no answer. The agent may have given up while the information was still deducible. ### Context shifts knowledge levels The same information can sit at different levels depending on what the agent is currently doing. In Claude Code, this happens concretely through conditional rules: - `src/schemas/AGENTS.md` is **knows directly** once the agent reads a file in `src/schemas/` (Claude Code loads a directory's `AGENTS.md` then). But when the agent is working on something unrelated, the same information is only **discoverable** — the agent would have to navigate to that directory and find the file. - `.claude/rules/card-doc.md` is **knows directly** when the agent reads or edits a `*.doc.card` file (the `paths:` glob triggers loading). When working on other card types, doc-specific knowledge is **discoverable** at best. - A guide card's compiled rules (e.g., `_content/docs/generated/intake-guide.md`) are **knows directly** when processing an intake job (the job-specific rule file references them). Outside that context, the same information is **knows about** (referenced in the agent guide's card type list) or **discoverable** (via directory listing). This means testing should consider: what was the agent *doing* when it answered? A question about doc card structure might be "knows directly" mid-procedure but "knows about" in a cold prompt. ### How the knowledge chain works in Claude Code The agent's context is built in layers, each corresponding to a knowledge level: - **Always loaded** → *knows directly*: `AGENTS.md` → `.beebox/agent-guide.md` (~128 lines of operational overview, directory layout, command summaries, card type catalog with doc references) - **Conditionally loaded** → *knows directly, in context*: `.claude/rules/*.md` (~28 rules, triggered by `paths:` glob patterns when the agent reads/edits matching files — e.g., `card-doc.md` loads when touching `*.doc.card`). Also, directory-level `AGENTS.md` files (e.g., `src/schemas/AGENTS.md`) are loaded when the agent reads a file in that directory. - **Referenced but not loaded** → *knows about*: engine reference docs in `node_modules/beebox/box-docs/*.md` (full card type specs, command reference, procedure authoring guide, domain guides — see its `README.md` index) and box-compiled docs in `_content/docs/generated/*.md` (guide compilations, personality, box-local card type specs). The agent guide points to these by path. - **Present but not referenced** → *discoverable*: config files, procedure definitions, guide cards. Available in the box but the agent has to find them by exploring. - **Outside the box** → *deducible*: beebox source code (`src/cards/`, `src/schemas/`, etc., or `node_modules/beebox` from inside the box). Accessible if the agent knows where to look, but outside the box. - **On the internet** → *researchable*: Third-party API docs, standards, libraries the codebase depends on but doesn't document locally. ## Running it ```bash npx tsx src/dev/knowledge-audit.ts run --box ~/src/boxes/test1 [--filter ] [--engine claude|codex|pi] npx tsx src/dev/knowledge-audit.ts list ``` `--box` takes the box root (`~/src/boxes/test1`) — there is only the one root to pass, and the context-history ledger keys off its basename (`src/dev/lib/audit-box.ts`). Pass an absolute path (or `~/…`), never a bare name like `test1`, which would resolve inside the monorepo and be refused by the nested-box guard. `--filter` takes one value, matched as an audit id, an id substring, or a tag; repeated flags keep only the last. The runner resets and cleans the box between tests, so it refuses a box with uncommitted changes. A worktree's test1 clone usually has such changes from ordinary dev use (an engine install sync, for example). Copy it to a scratch directory, commit there, and audit the copy. The runner has no box server and no notification channel, and a worktree clone's `node_modules/beebox` links to the engine outside the box. Three kinds of audit fail for those reasons alone: - A fixture `AGENTS.md` that imports a package doc (`@node_modules/beebox/box-docs/…`) loads nothing, because Claude Code does not follow an import out of the box. Run those audits on a copy whose `node_modules/beebox` is a real directory holding a copy of `box-docs/`. - `bbx remind` refuses without a channel that can reach the person; set `BBX_NOTIFY_FAKE=1` ([notifications](https://github.com/ianb/beebox/blob/main/beebox/docs/notifications.md)). - `bbx drive` commands need the server (`BBX_SERVER_URL`), so drive audits check the agent's choice, not the command's result. By default the runner uses the box's configured engine. `--engine` overrides that choice for the audit only, so the same definitions can be exercised against Claude and Codex without editing `_config/box.json`. Reports include the engine and use engine-qualified default filenames. Codex behavior is captured from the validated live app-server event stream; private rollout files are not parsed. Codex does not expose Claude-equivalent per-turn context snapshots on that surface, so Codex reports deliberately omit the context baseline rather than presenting incomparable usage as parity. A pi audit reads the pi session file under `.beebox/pi/sessions/` after the run (`src/dev/lib/test-runner/runner/pi-audit.ts`). pi's `read` counts as a file read, `ls` as a search, and `bash` as a command, classified as for Codex. The context baseline comes from the usage on each assistant message. pi needs an OpenRouter key granted to the box. To audit a copy of a box that has no grant, point `BBX_SECRETS_FILE` at a scratch store that grants the copy the placeholder `pi-dev-placeholder` as `openrouter`, and pass `--pi-dev-key`. The runner then gives pi `BBX_OPENROUTER_API_KEY` (environment or `beebox/.env`) in memory, in place of the placeholder (`src/dev/lib/openrouter-dev-key.ts`). ## Dev-guidance audits An entry with `surface: dev` audits a developer session instead of a box agent: what a Claude Code session in this monorepo checkout learns from the root and package `AGENTS.md` files, the project skills, and `docs/`. Its first questions cover the product spirit and design docs, which sessions kept forgetting. ```bash pnpm knowledge-audit run --dev [--model ] [--filter ] ``` `--dev` runs only `surface: dev` entries (a box run skips them) as `claude -p --output-format stream-json` in the checkout root, default model `claude-opus-5-5`. The session loads project settings and skills with hooks disabled, no MCP servers, and auto-memory off, so a pass means tracked guidance taught it; the user-level `~/.claude/CLAUDE.md` still loads. Its settings deny reads of the answer key (`knowledge-audits.yaml`, the context ledger, `src/dev/reports/`, and `scratch/`). Its tools are `Read`, `Grep`, `Glob`, and `Skill`. A read counts only when its tool result is not an error, and a Skill call counts as a read of that skill's `SKILL.md`, so `should_read` paths are repo-relative (`.claude/skills/bbx-design/SKILL.md`, `beebox/docs/box-work.md`). The text checks are unchanged; card, Bash, search, fixture, and context-dir fields are refused at load. Mechanics: `src/dev/lib/test-runner/dev-session.ts`. It cannot observe Codex sessions (`AGENTS.md`, `.codex/` agents), what a session learns only by doing (hooks, lint and doc-check failures, tool errors), or interactive-session context such as the developer's memory and `/` commands. It asks cold questions; a session mid-task may load more. The session reads `issues/`, so a before-and-after comparison of a guidance change also hides the issue that states the rule. ## Writing one Each entry in `knowledge-audits.yaml` has these fields: - `prompt` — what to ask the agent. - `expected_level` — one of: - `knows_directly` — should answer without reading any files. - `knows_about` — should know which docs to read, then answer. - `discoverable` — should be able to find the answer by exploring the filesystem. - `correct_contains` / `correct_contains_any` — strings that must appear in the response. Keep a literal only for a term the guide itself teaches; grade a free-text answer with `correct_matches`, and a box audit's card write with `cards_contain`. - `correct_matches` — JavaScript regexes, run with the `i` flag, that must match the response (an inline flag such as `(?i)` is a syntax error); use when the required relationship has legitimate wording variation that a list of exact substrings would overfit. Explain-only answers often describe a command without naming it. - `cards_contain` — strings that must appear in card files created by the agent. - `cards_not_under` — box-relative directory prefixes (`_tmp/`) no card the agent created or modified may sit under. - `should_read` — files the agent should read before answering. - `should_read_any` — alternative files, at least one of which the agent should read. Use this when generated guidance and its installed skill are equivalent navigation outcomes. - `read_before_edit` — files the agent must read before its first edit or write. Only pi's reader records edits, so on Claude and Codex the check fails. pi counts a nested `AGENTS.md` that a file tool delivered as a read. A read counts only from an earlier message than the edit (pi runs one message's tool calls in parallel) and only when it did not fail. - `should_search` — `web`, `box`, or `any`: the agent must look something up before answering. `web` is a WebSearch or WebFetch (a Codex provider search counts), `box` a `bbx search` command, `any` either. Use it for the laws that require checking rather than answering from memory. - `max_turns` — override the default 10-turn limit (use for tests requiring multi-step card creation). - `tags` — for filtering with `--filter`. - `surface` — `box` (default) or `dev`; see Dev-guidance audits. - `context_dir` — box-relative subdirectory to run the agent from. Sets the SDK's `cwd` there and adds the box root to `additionalDirectories`, mirroring how a chat session bound to a landmark is spawned. Use to audit that the subdirectory's `AGENTS.md` (and its `@MAP.md` import) actually load into the agent's context at session start. - `fixture` — map of box-relative path → file content, written before the test and removed afterward. Used to stage an `AGENTS.md` (or any other file) without checking it into the box. Combined with `context_dir`, this lets a single audit set up its own landmark-style scratch directory. - `plugins` — plugin names the box must have active for this audit. Before docs generation the runner adds them to `_config/box.json` and writes each missing schema stub; after the audit it resets the box. A fixture cannot do this, because fixtures are written after docs generation. View stubs are not written, so a plugin's card docs and skill load but its views do not render. Mechanics: `src/dev/lib/box-audits/plugins.ts`. - `final_reply` — when true, the text checks run on the text the chat shows: the reply after the last tool call, earlier text with ``, a folded text's callouts, and the last earlier line (text or progress update) when the reply is only acks. The chat folds other text written before a later tool call into the activity row. Use it for audits about how the reply reads. - `chat_mode` — when true, the agent runs with `chatSystemPrompt()` instead of the default working-directory prompt. Use for tests that audit chat-mode knowledge — the agent in a chat session sees the chat prompt's tag descriptions (``, ``, ``, ``, ``, etc.), so a non-chat-mode audit can't legitimately expect direct knowledge of those. If the test prompt declares `narration="on"` via a `` snapshot, `NARRATION_OVERLAY` is also appended — mirroring what `ChatSession.resolveSystemPrompt` does when narration is enabled on the session. - `response_not_contains` — list of substrings that must NOT appear in the agent's response. Useful for testing prompt overrides: e.g. the narration overlay tells the agent not to emit `` even when the user spoke; the audit can assert this directly. - `response_not_matches` — regexes, as in `correct_matches`, that must NOT match the response. Match the command form (`bbx answer <`), not a bare name, or a correct negation ("No `bbx answer`") fails. Example: ```yaml tests: - id: box-structure-inbox prompt: "Where would you look for unprocessed incoming items?" expected_level: knows_directly watch_for: "Names _content/inbox/ directly without searching" correct_contains: ["_content/inbox"] should_read: ["node_modules/beebox/box-docs/card-doc.md"] # optional should_not_read: ["some/file.md"] # optional tags: [navigation] ``` - **Immediately after adding or editing audit entries** — filtered to the new entries (`--filter `). A never-run audit is unverified in both directions: the agent may fail it, or the audit itself may be broken (bad `correct_contains`, wrong `should_read` path). Running is part of authoring, not a follow-up. Before running, test each changed response pattern by hand against a right answer and against one wrong answer that `watch_for` rules out, including a correct negation of the wrong action. A pattern widened only to pass a failing right answer often also passes a wrong one. ### Prompt style effects How you phrase a prompt changes which knowledge level the agent actually operates at. The same question can produce different behavior depending on whether the prompt encourages speed or thoroughness: - **No qualifier** (default) — The agent decides how much effort to spend. Fine for "knows directly" and usually fine for "knows about" — the agent will typically follow a reference when it has one. Less predictable for discoverable or researchable questions, but that's not purely a downside — it also gives the agent room to judge what level of effort the specific prompt warrants. Sometimes the agent correctly decides a quick answer is fine; sometimes it correctly decides to dig deeper. - **"Give me a quick answer"** / **"Be brief"** — Pushes the agent toward answering from what it already has in context. Lower latency, which matters for interactive experiences. Also good for testing whether something is truly "knows directly" — if the agent gets it right without reading files, the knowledge is well-placed. But increases guessing: the agent may confidently answer a "knows about" question without reading the doc, getting details wrong. - **"Think it through"** / **"Take your time"** — Encourages the agent to follow references and reason more carefully. "Knows about" questions should reliably get doc reads. But may still not trigger exploration for discoverable questions — the agent thinks harder about what it already has rather than going looking for new information. - **"Research this"** / **"Investigate thoroughly"** — Most likely to push the agent into exploration mode. Discoverable and even deducible questions have a better shot. But also the most expensive in tokens and time, and may lead the agent down irrelevant paths. When testing, try the same question with different styles to see where the boundary is between "knows directly" and "knows about" — does adding "be brief" cause the agent to guess instead of looking things up? That reveals which knowledge is genuinely in context vs. just referenced. ## Reading results When running `bbx prompt`, watch for: - **Knows directly**: Agent answers directly without reading files → good - **Knows about**: Agent reads the right referenced doc, then answers → good - **Knows about but guesses**: Agent answers without reading the doc → risky, may be wrong on details - **Discoverable**: Agent searches/explores, finds the answer → good but slow - **Guessable**: Agent answers confidently but incorrectly → bad, indicates a documentation gap ### Recording results After running audits, **update the status comments in `knowledge-audits.yaml`** with the date and results. Each test section (e.g., `# === Don't Drop Important Information ===`) should have a `# Status (YYYY-MM-DD):` comment noting: - How many tests pass/fail - Any notable failure patterns (turn limits, tool issues, knowledge gaps) - What fixed previous failures (if relevant) The status comment is the durable record. Reports in `reports/` are gitignored and ephemeral — they're a working artifact, not a result log. ### Context size Each report entry shows a **Context** line — the loaded-context size the box agent carried during that audit, read from the session log's per-turn `usage` (summed across `input + cache_creation + cache_read`, since prompt caching leaves raw `input_tokens` tiny). The *initial* number is the always-on baseline the box pays every turn (system prompt + agent-guide + box AGENTS.md + tool schemas); *peak* and *added* show how much answering grew it. A `knows_directly` / 0-read audit doubles as a baseline gauge: trim a box's always-on context, re- run, watch *initial* drop. Logic lives in `lib/context-usage.ts`. Every run also appends these numbers to **`src/dev/context-history.yaml`** — a committed ledger keyed by box → audit id → entries, each stamped with the date, the box's HEAD, and the monorepo's HEAD. The file's git history is the trend line: an AGENTS.md trim that drops the baseline shows up as a diff. (`boxCommit` is the box's HEAD *before* the harness regenerates docs — stable for a current box, but the throwaway worktree box clone is behind upstream templates, so its hash moves each run.) Logic lives in `lib/context-history.ts`. ### Interpreting failures Common failure patterns and what they mean: - **Hits `max_turns`** — the agent didn't converge. Either the prompt is ambiguous, the right docs aren't loadable from the agent's perspective, or the task is too multi-step for the default 10-turn limit. Try `max_turns: 20` first; if that doesn't help, the prompt or the underlying knowledge is the problem. - **Tool issues** — the agent tried to use a tool that's not available in this box, or used an available tool wrong. Usually means an AGENTS.md or rule file is misleading. - **Knowledge gap** — the agent answered confidently but wrong. The doc that should have taught the right answer either doesn't exist, isn't loaded into the agent's context, or contradicts itself. - **Reads wrong files** — `should_read` is wrong, or the doc structure changed and the agent is following a stale pointer. When fixing failures, prefer changing docs/prompts to changing the test — the test is asserting an expectation about agent behavior, and silently weakening it defeats the point.