Testing

Testing Philosophy

Tests serve three purposes in this project, in order of importance:

1. Forcing decomposition — Making something testable creates clean boundaries. Writing a test first helps identify a function's purpose and isolate it from its surroundings.
2. Documentation — Tests as literate documents that tell a story about how things work. Doctests are the primary format: readable markdown that happens to be executable.
3. Regression anchors — Specific bug prevention at the moment of a fix.

What tests are NOT for: validating types (the type system does that), achieving coverage percentages, or comprehensive verification for its own sake. Types with strict settings already act as smoke tests for structural correctness.

Key principles:

• No mock libraries. Injectability should be built into code. Prefer dependency injection over mocking frameworks.
• No server spin-up for testing. Route tests use Fastify's inject(), not a running server. State-in → render → check output.
• TAP over fancy test runners. TAP's text protocol is agent-readable and zero-dependency.
• Doctests are the default. If it can be explained with examples in markdown, it should be a doctest. Traditional .test.ts files are for things that genuinely need complex setup or meta-testing.

The tap plugin set is built at install time

.taprc disables @tapjs/typescript (plugin: - "!@tapjs/typescript") because tsx and the doctest loader resolve TypeScript here, and the typescript plugin's loader runs ahead of them and cannot resolve an extensionless directory import (ERR_UNSUPPORTED_DIR_IMPORT on src/frontend/src/lib/trpc).

That configured set only takes effect after tap build, which regenerates the Test class in node_modules/@tapjs/test/test-built/. tap does rebuild by itself when the built set differs from the configured one — but it computes the child processes' --import arguments from the already loaded class before it rebuilds, so the run that triggers the rebuild still spawns its children with the default plugins. Every pnpm install re-links @tapjs/test from the store and restores the shipped default build, so "the first tap run after an install uses the wrong loaders" is deterministic, not a flake.

The monorepo root's postinstall therefore runs tap build (~3s) after patch-package, so every workspace install builds it — a fresh worktree, a detached checkout, a deploy. It lives at the root rather than in beebox's own postinstall because beebox is packed and installed as a tarball dependency by boxes: a postinstall there would run in a consumer install that has no tap (a devDependency) and fail it. tap plugin list prints the configured set and never shows this; the built set is what tap versions lists under plugins:.

See issues/closed/bugs/2026-08-25-fresh-checkout-tap-default-plugins.md.

1. Doctests

Location: test/*.doctest.md Runner: TAP with a custom Node.js loader (the monorepo's agent-doctest package — loader hook at agent-doctest/src/doctest-hooks.ts, exposed via the agent-doctest/hooks export) Run: pnpm test (runs alongside traditional tests)

Doctest files are executable markdown documents. The prose explains behavior; fenced code blocks contain examples that are run as tests. A Node.js loader hook transforms them into TAP tests at runtime.

When to use: The default for most testing. Pure functions, template generators, stateful sequences with setup helpers, anything where showing examples is more readable than t.equal() assertions.

Syntax: See doctest syntax for the full reference.

```ts setup
import { initBox, isValidBox } from "../src/core/box/index.js";
```

## Creating a box

`initBox` creates the directory structure:

```
const tmp = await makeTmpDir();
await initBox(tmp, { skipGit: true });
await isValidBox(tmp)
=> true
```

Features:

• t.check() wildcards in expected values: «*» (anything), «date», «int», «name», «name=type»
• ``` continue blocks share scope with the previous block (for prose between related code)
• ``` cleanup blocks register teardown code via t.teardown() — runs after the test even on failure
• Lines ending with ; are statements; the last non-; line is the checked expression
• Setup blocks run at module scope for imports and helpers
• print() — scope-local function for building narrative output (see below)

print() for storytelling:

Each test gets its own print function. Lines accumulate and drain into the next => assertion, combined with the expression result. Useful for building up narrative output across multiple steps:

```
const result = await runProcedure(params);
print(`status: ${result.success}`);
for (const step of steps) {
  print(`${step.id}: ${step.status}`);
};
"done"
=>
status: true
fetch: completed
analyze: skipped
done
```

When print() isn't called, behavior is unchanged — the expression result is checked directly. print() returns void, so print("last line") as the expression adds the line without appending an extra value.

Shared helpers:

• test/helpers/doctest-helpers.ts — makeTmpBox() for filesystem tests. Returns .root, .list(), .read(), .write(), .cleanup(). All output is relative paths (no temp dir names in expected output).
• test/helpers/doctest-server.ts — makeTestServer() for route tests. Returns .inject() (string for check), .request() (parsed object), .seed(), .read(), .commitAll(), .cleanup(). Uses Fastify inject() internally — no socket server. .request()/.inject() prefix URLs with the test box slug (/test); use .rootRequest() to hit a root-level route without that prefix.

Current doctest files:

Testing with Service Fakes

External dependencies (APIs, CLIs) are wrapped in typed service interfaces with fake implementations for testing. Full service layer docs: src/services/CLAUDE.md.

Pattern

Every external service has three parts:

1. Interface — the subset of the API we actually use
2. Real factory — thin wrapper around the library, created from config
3. Fake factory — domain-specific in-memory implementation for tests

// Create a fake with domain-specific constructor params
const tg = createFakeTelegram({ username: "test_bot" });

// Fakes have observable state
await tg.sendMessage(123, "hello");
tg.sent.length  // => 1
tg.sent[0].text // => "hello"

Injecting into routes

Pass fakes via makeTestServer({ services: { ... } }):

const tg = createFakeTelegram({ username: "my_bot" });
const ctx = await makeTestServer({ services: { telegram: tg } });
// Routes that use telegram will get the fake
const res = await ctx.request({ method: "GET", url: "/api/admin/telegram-status" });
// Inspect what the route did via the fake's state
tg.sent  // messages the route sent

Call logging

Wrap any fake with withCallLog() to record method calls:

const tg = withCallLog(createFakeTelegram({ username: "bot" }));
await tg.sendMessage(123, "hello");
printCalls(tg.callLog);
// => sendMessage(123, "hello")

Available fakes

Connector testing pattern

Connector tests use makeTmpBox({ git: true }) to create a temp box with git, seed config files, inject a service fake, and run sync(). Credentials go through the machine secret store (docs/secrets.md), not a seeded box file — setSecret/grantSecret (src/core/secrets/lifecycle.js) put a value in and grant it to the box's slug, same as bbx secrets set/grant would:

const box = await makeTmpBox({ git: true });
await initBox(box.root);
box.commitAll("init box");
await setSecret({ name: `telegram-bot/${slug}`, value: JSON.stringify({...}) });
await grantSecret({ slug, name: `telegram-bot/${slug}`, access: "server" });

const tg = createFakeTelegram({ username: "bot", updates: [...] });
const connector = createTelegramConnector(box.root, tg);
const result = await connector.sync();
// Check result.created, result.updated, result.pushed
// Check tg.sent for outbound messages
await box.cleanup();

Doctest limitations

Doctest blocks are full TypeScript (compiled via esbuild's ts loader) — import type, non-null assertions, and type annotations all work, in setup and test blocks alike. The real limitations are structural: assertions compare serialized output (see the string-comparison rules in doctest syntax), and code blocks can't express trailing newlines.

2. Traditional TAP Tests

Location: test/*.test.ts Runner: tap v21 with tsx Run: pnpm test

Reserved for things that would be circular as doctests: testing the test infrastructure itself.

Current files:

2. Scenario Tests

Location: Definitions in ~/src/boxes/scenarios/<name>/, runner in src/scenario/ Run: bbx scenario list / bbx scenario run <name>

Scenario tests are end-to-end integration tests that run the full system (CLI commands, connectors, agents) against a real box, with stubbed time and HTTP. They verify that the whole pipeline works — from connector sync through agent processing to output generation.

When to use: Testing system behavior that spans multiple components — connector pulls items, wakeup creates jobs, reactor processes them, output appears in the right place. Also useful for testing agent behavior (via prompt: validations) in realistic contexts.

Scenario Definition (scenario.yaml)

name: intake-basic
description: Basic intake-job creation and processing

steps:
  - name: sync
    run: bbx wakeup
    time: "2026-01-20T15:00:00Z"     # sets BBX_TIME for this step onward
    checkpoint: after-sync            # git tag on the step's commit, for manual inspection
    validate:
      - committed: true               # working tree must be clean
      - script: "ls _bookkeeping/jobs/*.intake.job.card | wc -l | grep -q 2"
      - prompt: "Check that intake jobs were created for the seeded inbox items"

  - name: process
    run: bbx reactor
    validate:
      - committed: true
      - script: "ls _bookkeeping/jobs/*.intake.job.card 2>/dev/null | wc -l | grep -q '^0$'"

Stubs (stubs.yaml)

time: "2026-01-15T12:00:00Z"         # freeze BBX_TIME globally

http:
  - pattern: "https://techblog.test/feed.xml"
    response_file: stubs/feed.xml      # relative to scenario dir

  - pattern: "https://techblog.test/feed.xml"
    response_file: stubs/feed-with-articles.xml
    after: "4h"                        # only active after 4h of scenario time

  - pattern: "https://techblog.test/2026/01/ai-consumer-products*"
    response_file: stubs/article-1.html

HTTP stubs intercept fetch() calls. Patterns can use * suffix wildcards. When multiple stubs match, the last one whose after: constraint is met wins. Strict fetch mode (BBX_STRICT_FETCH=1) rejects any un-stubbed non-localhost fetch.

Validation Types

1. committed: true — Working tree must be clean (no staged/modified/untracked files).
2. script: "shell command" — Runs in the box root; passes if exit code is 0. Prefix with ! for negation.
3. prompt: "natural language check" — Sends the prompt to a Claude agent acting as test validator. Response must start with PASS or FAIL.

Runner Mechanics

1. Verifies box is on main with clean working tree
2. Creates test branch test/<name>/<timestamp>
3. Installs fetch stubs and strict fetch mode
4. Runs steps sequentially; failed step skips remaining steps
5. On completion, checks out main (test branch preserved for inspection)

Dry run: bbx scenario run <name> --dry-run

Scenarios always run every step from the beginning — there is no flag to resume from a checkpoint. checkpoint: on a step still tags the commit (scenario/<name>/<checkpoint>) for manual inspection with git checkout, but nothing restores state from it; a prior --from <checkpoint> flag that only skipped steps without restoring their state was removed.

Available Scenarios

Creating a New Scenario

Each scenario is a self-contained directory under ~/src/boxes/scenarios/<name>/ with its own git repo as the test box.

Directory structure: bbx init scaffolds the one-root layout by default (package.json/tsconfig/src/ plus the underscore operational areas at the same root — see docs/box-layout.md), so a freshly-created scenario's box/ looks like:

~/src/boxes/scenarios/my-scenario/
  scenario.yaml      # step definitions (required)
  stubs.yaml         # time/HTTP stubs (optional)
  stubs/             # stub response files (optional)
    feed.xml
    article.html
  setup.md           # human-readable description of what this tests
  box/               # the git repo — a real box initialized with bbx init, boxRoot itself
    _content/inbox/  # pre-seeded test data
    _config/         # connector configs, schedules, etc.
    ...

The existing scenarios in the table above (intake-basic, tick-basic, tick-chain) predate this and haven't been migrated — getBoxShape is strict and rejects any marker without shapeVersion: 3, so these scenarios need bbx migrate run against them (or recreating) rather than being a supported second shape.

Steps to create:

1. Create the directory and initialize a box:

   mkdir -p ~/src/boxes/scenarios/my-scenario/box
   cd ~/src/boxes/scenarios/my-scenario/box
   git init
   bbx init .
   

2. Seed the box with test data. Put cards in _content/inbox/, configure connectors in _config/connectors/, add scheduled scripts, etc. Commit everything — the scenario runner requires a clean main branch as starting state.

3. Write scenario.yaml with steps. Each step runs a shell command (usually a bbx command) and validates the result. See the format description above.

4. Write stubs.yaml if your scenario involves HTTP (connector syncs, article fetches). Freeze time with time: to make timestamps deterministic. Put response files in stubs/.

5. Write setup.md describing what the scenario tests, what stubs are used, and what the expected outcome is. This is for humans, not the runner.

6. Test it:

   bbx scenario run my-scenario --dry-run   # verify steps parse correctly
   bbx scenario run my-scenario             # run for real
   

Design principles for scenarios:

• Each scenario tests one pipeline or behavior. Don't combine unrelated features. intake-basic tests intake jobs only; tick-basic tests scheduled-script behavior only.
• Seed the minimal data needed. The intake-basic box has just 2 memos in inbox — enough to verify the behavior, not so much that agent processing is slow or unpredictable.
• Use --skip-* flags on bbx wakeup to isolate phases when you don't need the full wakeup cycle.
• Use checkpoints on steps that are expensive (agent runs) to tag the resulting commit for manual inspection later. Scenarios always run from the beginning, so a checkpoint doesn't let you skip re-running earlier steps.
• Prefer script: validations for structural checks (files exist, XML contains expected content). Use prompt: validations only for things that require judgment (quality of generated text, correct interpretation of ambiguous input).
• prompt: validations cost money. Each one invokes a Claude agent with up to 5 turns / $0.50. Use them sparingly.

Managing Scenarios

Inspecting a failed run: The test branch test/<name>/<timestamp> is preserved after the run. Check it out to see the state at failure:

cd ~/src/boxes/scenarios/my-scenario/box
git branch                        # list test branches
git checkout test/my-scenario/... # inspect the failed state
git checkout main                 # return to clean state

Cleaning up old test branches:

git branch | grep 'test/' | xargs git branch -D

Updating a scenario's test data: Edit files in the box on main, commit, then re-run. The runner always starts from a clean main.

Scenarios are git repos — you can use standard git operations. The box inside each scenario is a real box; bbx commands work normally when you cd into it.

3. Knowledge Audits

Location: Tests in src/dev/knowledge-audits.yaml, runner in src/dev/knowledge-audit.ts, reports in src/dev/reports/ Run: pnpm knowledge-audit run [--filter <id-or-tag>]

Knowledge audits test what the agent knows rather than what the system does. They run prompts against a Claude agent in a box and check whether the agent answered from loaded context (knows directly), followed a doc reference (knows about), or had to search (discoverable).

When to use: Verifying that documentation, agent guides, and conditional rules are working — that the agent has the right information at the right time. Not for testing system behavior.

See knowledge-taxonomy.md for the full knowledge taxonomy and test prompt guide.

Test Definition

tests:
  - id: box-structure-inbox
    prompt: "Where would you look for unprocessed incoming items?"
    expected_level: knows_directly
    watch_for: "Names _content/inbox/ directly without searching"
    correct_contains: ["_content/inbox"]
    should_read: ["node_modules/beebox/box-docs/card-memo.md"]   # optional
    should_not_read: ["some/file.md"]              # optional
    tags: [navigation]

How It Works

1. Runs the prompt via bbx prompt in the test box
2. Parses the session transcript to extract: files read, searches, bash commands, response text
3. Automated checks: correct_contains (substring match), should_read/should_not_read (file access)
4. Generates a Markdown report with results + blank assessment field for human review

Reports go to src/dev/reports/audit-report-<timestamp>.md.

4. Session Critiques

Location: Report generator in src/dev/lib/session-report.ts, subagent in .claude/agents.json Run: @session-critique <session-id> (or @session-critique latest)

Session critiques evaluate whether CLI tools helped or hindered the agent during real agentic sessions. Unlike knowledge audits (which test what the agent knows), session critiques test whether the tools the agent used gave it good output.

When to use: After observing a session with unusual behavior — the agent took too many turns, used raw git/grep instead of bbx commands, or seemed confused by command output. Also useful as a periodic check on CLI usability.

How it works

1. Report extraction: bbx session <id> --tool-report parses the session JSONL and produces a markdown report containing:

   • User and assistant text messages
   • Bash commands with their full output (the key differentiator from bbx session which skips output)
   • Read/Write/Edit as one-liner summaries for context
   • Grep/Glob with abbreviated results

2. Critique subagent: The @session-critique agent reads the report and evaluates it against five criteria:

   • Unhelpful output — Did a bbx command produce output the agent ignored or misinterpreted?
   • Missing commands — Did the agent cobble together raw commands when a bbx command should have existed?
   • Wrong tool — Did the agent use the wrong tool (e.g., Read for scanning many files)?
   • Bad error messages — Did errors lead the agent to the fix or cause flailing?
   • Wasted effort — Retry loops, redundant reads, unnecessarily complex approaches?

3. Output: Structured findings with evidence, impact, and concrete suggestions (CLI format changes, new commands, .claude/rules/ hints).

Running a critique

# From within a box directory:
bbx session --list                   # find session IDs
bbx session <id> --tool-report       # generate report for a specific session
bbx session --latest --tool-report   # most recent session

# Or use the subagent (from Claude Code in this project):
# @session-critique <session-id>
# @session-critique latest

The subagent runs bbx session itself, so it needs a box directory context.

Acting on findings

Session critiques produce actionable suggestions. The typical workflow:

1. Run a critique on a session that seemed inefficient or problematic.
2. Review findings. Each has a category and suggestion.
3. For unhelpful-output: Modify the bbx command's output format — trim noise, surface key info earlier, add structured markers the agent can parse.
4. For missing-command: Consider whether a new bbx subcommand or flag would help. Only add one if the pattern recurs across sessions.
5. For wrong-tool: Add a .claude/rules/ hint that triggers when the agent is in the relevant context, pointing it to the right tool.
6. For bad-error: Improve the error message in the CLI command. Good errors name what went wrong, what file/card caused it, and what to do next.
7. For wasted-effort: Usually a prompting issue. Check if the system prompt or agent guide is missing guidance for this task type.

Not every session has problems. If the critique comes back clean, that's a positive signal that the tools are working.

Comparison with other test types

5. Card Validator Hook

Location: src/core/sdk-hooks.ts (cardValidatorHook) Trigger: Runs automatically during agent sessions on PostToolUse of Write/Edit

Not a test you run manually, but a live validation hook. When an agent writes or edits a .card file, the hook calls the card linter (src/core/card-lint.ts, built on the card primitives absorbed from the former cardworks package into src/cards/) in-process and feeds any issues back as additionalContext. This catches frontmatter/schema issues during agent work rather than after.

Also enforces directory structure rules (e.g., trick scripts must be in subdirectories of tricks/scripts/).

6. Frontend Dev Stubs

Some frontend bugs only manifest against real layout and measurement — scroll behavior, virtualization, streaming-driven reflow — and can't be reproduced in a doctest. For these, drive the running app with bin/browse (see .claude/skills/browse/SKILL.md, monorepo root) and use a dev stub to make the input deterministic instead of depending on a live agent response.

/fakestream — deterministic chat streaming

Location: src/frontend/src/machines/chat-actors.ts (runFakeStream) Trigger: Send a chat message beginning with /fakestream.

Instead of calling the backend, the chat machine plays a timed script of STREAM_TEXT events, growing the assistant bubble at a controlled rate with no API calls. This reproduces streaming-UI bugs (scroll-follow, layout jitter) frame-for-frame.

/fakestream [chunks] [intervalMs] [chunkLen]

• chunks — total text events to emit (default 200)
• intervalMs — delay between events (default 40)
• chunkLen — approx chars per event (default 25)

Example: /fakestream 2000 30 25 streams ~50k chars over ~60s. The message list is a single scroll container with data-testid="chat-scroller" driven by the useChatScroll controller (components/chat/chat-scroll.ts). Measure scroll state from the browser to assert behavior deterministically — with bin/browse eval --no-wait, since a plain eval waits out the stream and hands you the settled page:

// Under the write-on-user-action model, fromBottom GROWS as the reply streams
// and scrollTop does not move: nothing follows the bottom.
const s = document.querySelector('[data-testid="chat-scroller"]');
({ fromBottom: s.scrollHeight - s.scrollTop - s.clientHeight, scrollTop: s.scrollTop, scrollHeight: s.scrollHeight });

Driving the scroll behavior. The two instruments — the scenario table at /dev/chat-scroll and the step-by-step bin/browse procedure, plus the real-device checklist — live in chat-scroll-testing.md. Note that /fakestream content vanishes at finalize in a server-backed session (the authoritative history has no such turn), so sample while it streams, not after.

The stub is gated purely on the message prefix, so it ships harmlessly — a real message never starts with /fakestream.

Smoke Tier (a real box boots and is walked — a merge gate)

Run: bin/smoke (add --box <slug>; --no-restart when debugging the walk itself). ~30s, hard-fails, never selected by the test graph.

The only tier that answers "does the app actually run". It restarts this checkout's dev-server generation through the router's control socket, waits for the box's backend to answer /api/health (vite serves pages seconds before Fastify is up, and a box child reloads itself after a post-commit CLI rebuild — both looked like a 502 flake until 2026-08-26), then walks it in a browser: the chat page renders its shell, the app bar's place menu opens and lists landmarks, selecting one moves you there, /browse lists the box's real content, a card opens and renders, and the page raised no uncaught errors. No model turns — nothing that spends tokens or waits on an agent.

It exists because three escapes on 2026-08-25/26 passed typecheck, lint and their selected tests while the app was broken: the code was right and the state was wrong (a missing frontend build, broken global ~/.codex state, an SDK item outside its own union). Only a real box on the real machine shows those.

Where it runs. /finish names it on the decision sheet for any diff that touches a deployed path (bin/deployed-paths.ts — the same rule the deploy hook uses to decide whether a commit ships), and bin/finish-verify runs it before bin/land. It is a gate, not a post-merge alarm, because the router runs TypeScript straight off disk and never reloads it: after the merge, the main checkout's running generation is still the old source, so there would be nothing correct to point at. The worktree at that moment already contains main and is byte-identical to what lands.

A failing test file gets a flake re-run; the smoke walk does not. It boots one real box and either that works or it does not.

Post-deploy, deploy/deploy.sh runs the server-side half: hub /healthz, a /healthz/canary that cold-starts one real box, and an assertion that the shipped src/frontend/dist/index.html exists. That last one is the condition registerSpaFallback branches on — without the build, every page navigation 404s while both health checks stay green. It asserts the file rather than probing a URL because the hub redirects an unauthenticated navigation to login before the child is reached, so a URL probe answers 302 either way. It is not duplicated in the local walk: in dev, page requests are served by vite and never reach that handler at all.

Every run is logged, so the tier can be trimmed on evidence. Each walk appends a line to callback-smoke-log.jsonl in the shared git dir (beside the test ledger, and shared by every worktree on the machine for the same reasons): the verdict, which step failed, and every step's outcome and duration. bin/smoke --report folds it into per-step counts — how often each step ran, how often it caught something, what it costs at p50. A step that has never failed across many runs is paying rent out of the budget, and the report names those once there are enough runs for a clean record to mean anything.

Breaking it on purpose: declare it. Proving the tier can still go red means breaking something, and the resulting red is indistinguishable in the log from one the tier caught for real — the first weekly review duly read one as an intermittent worth watching. So say what you broke:

BBX_SMOKE_FAULT_INJECTION="hub throws at import" bin/smoke

The reason is stamped on the run, the walk says so loudly while it runs, and every count in --report and in the weekly review excludes it. A step's forced column counts these separately from failed: firing on demand proves the step is wired up, never that it has caught anything.

Read ran as the denominator, not the run count: the walk stops at the first failure, so a late step has seen fewer runs than an early one. Steps are keyed by a stable id, not their printed name, so rewording a step keeps its history.

A weekly schedule reviews the tier's shape (schedules/smoke-review/). It gathers the log, the window's failures, and the bugs filed that week, then hands them to a session that asks two questions: is every step still earning its place, and is there something the walk should be checking that it isn't. The second half is necessarily after-the-fact — a bug that reached main and was only visible on a running box is what "the walk has a hole" looks like, and the hourly full-suite run's own issues are the sharpest evidence for it. The session files an issue; it does not edit the tier, because trimming a step or adding one belongs to a session that can run the walk and see what happens.

It is disruptive, on purpose. The walk stops this checkout's dev-server generation and closes the shared browse session (tours and interactive bin/browse share one Chrome). If the boxholder has this worktree open in a browser, their tab reloads. That is the cost of testing the code that is about to land rather than whatever was running.

Known gap. A change confined to bin/ gets no smoke walk, because bin/ ships nothing and the "code-related" rule is deliberately the deploy hook's. A bin/router.ts change that breaks the dev router is therefore not gated here.

Tours (the app's walk, written down — kept true weekly, not a gate)

Scripted browser walks (bin/tour <name>, scripts in test/tours/) that produce review artifacts: desktop+mobile screenshots, AX-tree snapshots, axe-core reports, and soft-assertion findings per checkpoint. Ungated — findings never fail an exit code and artifacts are gitignored — but not unrun: schedules/tour-check/ walks every tour weekly, edits a tour when a miss is explained by a deliberate change (a landing, plan, or issue), and files an issue for anything else. That asymmetry with smoke-review (which files and never edits) is by design: a gate goes red on every intended UI change, so tours need a session with judgment rather than a threshold. Full reference — running, reviewing artifacts, writing conventions, the edit-vs-finding rule, and when NOT to use them: tours.md.

Field Tests (agent-operator, expensive, not a gate)

Field tests use a persona operator, a disposable real box, its agents, and the real web UI to test realistic discoverability and end-to-end use. They run weekly or on demand and never gate CI or a merge. See the current field-testing runbook for commands, artifacts, visual-review rules, and issue-triage boundaries. The implemented plan is design history.

Choosing the Right Approach

Overlap: Some things could be tested at multiple levels. Prefer the lowest level that catches the bug:

• A template generating bad XML → unit test (fast, deterministic)
• An agent not using <chat-response> tags → scenario test (needs agent behavior)
• An agent not knowing about a command → knowledge audit (tests documentation)

Adding New Tests

New doctest (preferred)

Create test/<name>.doctest.md. Write prose explaining the behavior, with fenced code blocks containing examples. See doctest syntax for syntax. Runs automatically with npm test.

New traditional test

Create test/<name>.test.ts, import from tap. Use for route integration tests, meta-tests, or anything needing complex setup that doesn't read well as documentation.

New scenario

Create ~/src/boxes/scenarios/<name>/ with scenario.yaml and optionally stubs.yaml + stubs/ directory. Test with bbx scenario run <name> --dry-run first.

New knowledge audit

Add entries to src/dev/knowledge-audits.yaml. Run with --filter <id> to test individually.

New session critique

Pick a session to review (use bbx session --list from a box), then run @session-critique <id> from Claude Code in the beebox project. Review the findings and apply fixes per the "Acting on findings" guide above.

Periodic Checks

Not automated — run these occasionally and fix what they find.

Session critiques

Review recent agentic sessions (intake triage, capture processing, chat handling) for tool quality issues. Pick sessions that seemed slow or where the agent used workarounds. Run @session-critique <id> and act on findings. See § Session Critiques above.

Documentation graph

npx tsx src/dev/doc-graph.ts > docs/doc-graph.md — scans all .md files, extracts cross-references, reports orphans and broken links. Review description quality at each reference site. Fix issues, regenerate, commit.

Supplemental linters

These catch issues the pre-commit hook doesn't:

pnpm lint:oxlint    # Ambiguous constructors, useless spreads, identical branches
pnpm lint:knip      # Unused files, exports, dependencies
pnpm lint:circular  # Value-import circular dependencies (type-only cycles are OK)

Future Directions

Key ideas not yet implemented:

• Self-describing services — Services export description, examples, and properties alongside their functions. Tests get generated from these.
• Property testing — Semantically meaningful invariants (like parse(serialize(card)) === card), not random fuzzing.
• CGI-style route testing — Test routes as pure state→output functions rather than spinning up servers.
• Assertive logging — Soft assertions in production code that surface through the logging system, caught by an error harness in tests.