Bee Box documentation · directory: https://beebox.run/docs/dev/ · index: https://beebox.run/docs/dev/index.md · root: https://beebox.run/llms.txt

# Testing

## Testing Philosophy

Tests serve three purposes in this project, in order of importance:

1. **Forcing decomposition** — Making something testable creates clean boundaries. Writing a test first helps identify a function's purpose and isolate it from its surroundings.
2. **Documentation** — Tests as literate documents that tell a story about how things work. Doctests are the primary format: readable markdown that happens to be executable.
3. **Regression anchors** — Specific bug prevention at the moment of a fix.

What tests are NOT for: validating types (the type system does that), achieving coverage percentages, or comprehensive verification for its own sake. Types with strict settings already act as smoke tests for structural correctness.

**Key principles:**
- **No mock libraries.** Injectability should be built into code. Prefer dependency injection over mocking frameworks.
- **No server spin-up for testing.** Route tests use Fastify's `inject()`, not a running server. State-in → render → check output.
- **TAP over fancy test runners.** TAP's text protocol is agent-readable and zero-dependency.
- **Doctests are the default.** If it can be explained with examples in markdown, it should be a doctest. Traditional `.test.ts` files are for things that genuinely need complex setup or meta-testing.

## The tap plugin set is built at install time

`.taprc` disables `@tapjs/typescript` (`plugin: - "!@tapjs/typescript"`) because
tsx and the doctest loader resolve TypeScript here, and the typescript plugin's
loader runs ahead of them and cannot resolve an extensionless directory import
(`ERR_UNSUPPORTED_DIR_IMPORT` on `src/frontend/src/lib/trpc`).

That configured set only takes effect after `tap build`, which regenerates the
Test class in `node_modules/@tapjs/test/test-built/`. tap does rebuild by
itself when the built set differs from the configured one — but it computes the
child processes' `--import` arguments from the *already loaded* class before it
rebuilds, so the run that triggers the rebuild still spawns its children with
the default plugins. Every `pnpm install` re-links `@tapjs/test` from the store
and restores the shipped default build, so "the first tap run after an install
uses the wrong loaders" is deterministic, not a flake.

The monorepo root's `postinstall` therefore runs `tap build` (~3s) after
`patch-package`, so every workspace install builds it — a fresh worktree, a
detached checkout, a deploy. It lives at the root rather than in
`beebox`'s own `postinstall` because `beebox` is packed and
installed as a tarball dependency by boxes: a `postinstall` there would run
in a consumer install that has no `tap` (a devDependency) and fail it.
`tap plugin list` prints the *configured* set and never shows this; the built
set is what `tap versions` lists under `plugins:`.

See issues/closed/bugs/2026-08-25-fresh-checkout-tap-default-plugins.md.

## 1. Doctests

**Location:** `test/*.doctest.md`
**Runner:** TAP with a custom Node.js loader (the monorepo's `agent-doctest` package — loader hook at `agent-doctest/src/doctest-hooks.ts`, exposed via the `agent-doctest/hooks` export)
**Run:** `pnpm test` (runs alongside traditional tests)

Doctest files are executable markdown documents. The prose explains behavior; fenced code blocks contain examples that are run as tests. A Node.js loader hook transforms them into TAP tests at runtime.

**When to use:** The default for most testing. Pure functions, template generators, stateful sequences with setup helpers, anything where showing examples is more readable than `t.equal()` assertions.

**Syntax:** See [doctest syntax](https://github.com/ianb/beebox/blob/main/agent-doctest/docs/syntax.md) for the full reference.

````markdown
```ts setup
import { initBox, isValidBox } from "../src/core/box/index.js";
```

## Creating a box

`initBox` creates the directory structure:

```
const tmp = await makeTmpDir();
await initBox(tmp, { skipGit: true });
await isValidBox(tmp)
=> true
```
````

**Features:**
- `t.check()` wildcards in expected values: `«*»` (anything), `«date»`, `«int»`, `«name»`, `«name=type»`
- ```` ``` continue ```` blocks share scope with the previous block (for prose between related code)
- ```` ``` cleanup ```` blocks register teardown code via `t.teardown()` — runs after the test even on failure
- Lines ending with `;` are statements; the last non-`;` line is the checked expression
- Setup blocks run at module scope for imports and helpers
- `print()` — scope-local function for building narrative output (see below)

**`print()` for storytelling:**

Each test gets its own `print` function. Lines accumulate and drain into the next `=>` assertion, combined with the expression result. Useful for building up narrative output across multiple steps:

````markdown
```
const result = await runProcedure(params);
print(`status: ${result.success}`);
for (const step of steps) {
  print(`${step.id}: ${step.status}`);
};
"done"
=>
status: true
fetch: completed
analyze: skipped
done
```
````

When `print()` isn't called, behavior is unchanged — the expression result is checked directly. `print()` returns void, so `print("last line")` as the expression adds the line without appending an extra value.

**Shared helpers:**
- `test/helpers/doctest-helpers.ts` — `makeTmpBox()` for filesystem tests. Returns `.root`, `.list()`, `.read()`, `.write()`, `.cleanup()`. All output is relative paths (no temp dir names in expected output).
- `test/helpers/doctest-server.ts` — `makeTestServer()` for route tests. Returns `.inject()` (string for check), `.request()` (parsed object), `.seed()`, `.read()`, `.commitAll()`, `.cleanup()`. Uses Fastify `inject()` internally — no socket server. `.request()`/`.inject()` prefix URLs with the test box slug (`/test`); use `.rootRequest()` to hit a root-level route without that prefix.

**Current doctest files:**

| File | Tests |
|------|-------|
| `test/core/box.doctest.md` | `initBox()`, directory structure, `isValidBox()`, `findBoxRoot()`, metadata |
| `test/schemas/schemas.doctest.md` | Schema registry, card templates (memo, question, intake-job, calendar-review-job) |
| `test/connectors/intake-utils.doctest.md` | `createOrAppendIntakeJob()` — create, append, multi-source |
| `test/connectors/calendar-utils.doctest.md` | ICS parsing, event formatting, timespan parsing, date filtering |
| `test/core/chat-response-extraction.doctest.md` | `<chat-response>` streaming extraction, chunking, multiline |
| `test/connectors/chat-utils.doctest.md` | Chat utilities |
| `test/schemas/scheduled-script.doctest.md` | `isDue()`, `isDueForWakeup()`, `isWithinBudget()`, template generation |
| `test/core/schedule-state.doctest.md` | `pruneRecentRuns()`, `recordRun()` |
| `test/cli/lib/format.doctest.md` | `stripAnsi()` |
| `test/core/procedure/dedent.doctest.md` | `dedent()` |
| `test/serialize.doctest.md` | Value serialization |
| `test/cli/lib/paths.doctest.md` | Card name parsing |
| `test/webapp/routes/routes-scheduler.doctest.md` | Scheduler log and schedules listing API |
| `test/webapp/routes/routes-admin.doctest.md` | Box config admin API |
| `test/webapp/routes/routes-api.doctest.md` | Core data API (status, inbox, cards, browse, debug-log, activity) |
| `test/webapp/routes/routes-commands.doctest.md` | Command listing, details, sync execution, error cases |
| `test/webapp/routes/routes-history.doctest.md` | Git commit log, diffs, session log |
| `test/webapp/routes/routes-actions.doctest.md` | Answer question, create card, validation |
| `test/webapp/routes/routes-clerk.doctest.md` | Clerk extension API (memo, save-to-brief, save-page, tabs, actions) |
| `test/frontend/lib/parse-tags.doctest.md` | XML-like tag parsing (frontend) |
| `test/frontend/lib/patmatch.doctest.md` | Keyword pattern matching (frontend) |
| `test/frontend/lib/speech-parsing.doctest.md` | Speech tag extraction for TTS (frontend) |
| `test/frontend/lib/speech-keywords.doctest.md` | Voice command keyword detection (frontend) |
| `test/print.doctest.md` | `print()` function in doctests (meta-test) |
| `test/core/procedure/procedure-engine.doctest.md` | Procedure engine: shell steps, precheck skip/fail, validation, agent mock, fallback commits |
| `test/cli/lib/git.doctest.md` | Git command helpers (init, commit, log, diff, status, branches, tags) |
| `test/cli/lib/time.doctest.md` | Stubbable time utilities (BBX_TIME env, stubs.yaml, caching) |
| `test/webapp/routes/routes-calendar.doctest.md` | Calendar config routes (list available, get/save config) |
| `test/services/service-call-log.doctest.md` | Generic `withCallLog()` wrapper for recording method calls |
| `test/services/service-telegram.doctest.md` | Telegram service fake (outbox, webhook, polling) |
| `test/services/service-google-calendar.doctest.md` | Google Calendar service fake (calendars, events) |
| `test/services/service-openai-audio.doctest.md` | OpenAI audio service fake (transcription, TTS) |
| `test/service-imap.doctest.md` | IMAP service fake (connect, search, fetch) |
| `test/connectors/connector-telegram.doctest.md` | Telegram connector: extractMessage, webhook processing, full sync, outbound send |

## Testing with Service Fakes

External dependencies (APIs, CLIs) are wrapped in typed service interfaces with fake implementations for testing. Full service layer docs: `src/services/CLAUDE.md`.

### Pattern

Every external service has three parts:

1. **Interface** — the subset of the API we actually use
2. **Real factory** — thin wrapper around the library, created from config
3. **Fake factory** — domain-specific in-memory implementation for tests

```typescript
// Create a fake with domain-specific constructor params
const tg = createFakeTelegram({ username: "test_bot" });

// Fakes have observable state
await tg.sendMessage(123, "hello");
tg.sent.length  // => 1
tg.sent[0].text // => "hello"
```

### Injecting into routes

Pass fakes via `makeTestServer({ services: { ... } })`:

```typescript
const tg = createFakeTelegram({ username: "my_bot" });
const ctx = await makeTestServer({ services: { telegram: tg } });
// Routes that use telegram will get the fake
const res = await ctx.request({ method: "GET", url: "/api/admin/telegram-status" });
// Inspect what the route did via the fake's state
tg.sent  // messages the route sent
```

### Call logging

Wrap any fake with `withCallLog()` to record method calls:

```typescript
const tg = withCallLog(createFakeTelegram({ username: "bot" }));
await tg.sendMessage(123, "hello");
printCalls(tg.callLog);
// => sendMessage(123, "hello")
```

### Available fakes

| Service | Factory | Key constructor params | Observable state |
|---------|---------|----------------------|-----------------|
| Telegram | `createFakeTelegram()` | `{ username }` | `.sent[]`, `.webhookUrl` |
| Claude CLI | `createFakeClaudeCli()` | `{ loggedIn? }` | `.loggedIn` |
| Google Calendar | `createFakeGoogleCalendar()` | `{ calendars?, events? }` | `.calendars[]`, `.events[]` |
| OpenAI Audio | `createFakeOpenAIAudio()` | `{ transcriptionText? }` | `.calls[]` |
| IMAP | `createFakeImap()` | `{ messages? }` | `.connected`, `.lockedMailbox` |
| Google Auth | `createFakeGoogleAuth()` | `{ accessToken? }` | — |

### Connector testing pattern

Connector tests use `makeTmpBox({ git: true })` to create a temp box with git, seed config files, inject a service fake, and run `sync()`. Credentials go through the machine secret store (`docs/secrets.md`), not a seeded box file — `setSecret`/`grantSecret` (`src/core/secrets/lifecycle.js`) put a value in and grant it to the box's slug, same as `bbx secrets set`/`grant` would:

```typescript
const box = await makeTmpBox({ git: true });
await initBox(box.root);
box.commitAll("init box");
await setSecret({ name: `telegram-bot/${slug}`, value: JSON.stringify({...}) });
await grantSecret({ slug, name: `telegram-bot/${slug}`, access: "server" });

const tg = createFakeTelegram({ username: "bot", updates: [...] });
const connector = createTelegramConnector(box.root, tg);
const result = await connector.sync();
// Check result.created, result.updated, result.pushed
// Check tg.sent for outbound messages
await box.cleanup();
```

### Doctest limitations

Doctest blocks are full TypeScript (compiled via esbuild's `ts` loader) — `import type`, non-null assertions, and type annotations all work, in setup and test blocks alike. The real limitations are structural: assertions compare serialized output (see the string-comparison rules in [doctest syntax](../../agent-doctest/docs/syntax.md)), and code blocks can't express trailing newlines.

## 2. Traditional TAP Tests

**Location:** `test/*.test.ts`
**Runner:** [tap](https://node-tap.org/) v21 with tsx
**Run:** `pnpm test`

Reserved for things that would be circular as doctests: testing the test infrastructure itself.

**Current files:**

| File | Tests |
|------|-------|
| `test/check.test.ts` | Wildcard matching, extractions, diff output, serializers, inspect() |
| `test/doctest.test.ts` | Doctest parser and generator (meta-testing) |

## 2. Scenario Tests

**Location:** Definitions in `~/src/boxes/scenarios/<name>/`, runner in `src/scenario/`
**Run:** `bbx scenario list` / `bbx scenario run <name>`

Scenario tests are end-to-end integration tests that run the full system (CLI commands, connectors, agents) against a real box, with stubbed time and HTTP. They verify that the whole pipeline works — from connector sync through agent processing to output generation.

**When to use:** Testing system behavior that spans multiple components — connector pulls items, wakeup creates jobs, reactor processes them, output appears in the right place. Also useful for testing agent behavior (via `prompt:` validations) in realistic contexts.

### Scenario Definition (`scenario.yaml`)

```yaml
name: intake-basic
description: Basic intake-job creation and processing

steps:
  - name: sync
    run: bbx wakeup
    time: "2026-01-20T15:00:00Z"     # sets BBX_TIME for this step onward
    checkpoint: after-sync            # git tag on the step's commit, for manual inspection
    validate:
      - committed: true               # working tree must be clean
      - script: "ls _bookkeeping/jobs/*.intake.job.card | wc -l | grep -q 2"
      - prompt: "Check that intake jobs were created for the seeded inbox items"

  - name: process
    run: bbx reactor
    validate:
      - committed: true
      - script: "ls _bookkeeping/jobs/*.intake.job.card 2>/dev/null | wc -l | grep -q '^0$'"
```

### Stubs (`stubs.yaml`)

```yaml
time: "2026-01-15T12:00:00Z"         # freeze BBX_TIME globally

http:
  - pattern: "https://techblog.test/feed.xml"
    response_file: stubs/feed.xml      # relative to scenario dir

  - pattern: "https://techblog.test/feed.xml"
    response_file: stubs/feed-with-articles.xml
    after: "4h"                        # only active after 4h of scenario time

  - pattern: "https://techblog.test/2026/01/ai-consumer-products*"
    response_file: stubs/article-1.html
```

HTTP stubs intercept `fetch()` calls. Patterns can use `*` suffix wildcards. When multiple stubs match, the last one whose `after:` constraint is met wins. Strict fetch mode (`BBX_STRICT_FETCH=1`) rejects any un-stubbed non-localhost fetch.

### Validation Types

1. **`committed: true`** — Working tree must be clean (no staged/modified/untracked files).
2. **`script: "shell command"`** — Runs in the box root; passes if exit code is 0. Prefix with `!` for negation.
3. **`prompt: "natural language check"`** — Sends the prompt to a Claude agent acting as test validator. Response must start with `PASS` or `FAIL`.

### Runner Mechanics

1. Verifies box is on `main` with clean working tree
2. Creates test branch `test/<name>/<timestamp>`
3. Installs fetch stubs and strict fetch mode
4. Runs steps sequentially; failed step skips remaining steps
5. On completion, checks out `main` (test branch preserved for inspection)

Dry run: `bbx scenario run <name> --dry-run`

Scenarios always run every step from the beginning — there is no flag to
resume from a checkpoint. `checkpoint:` on a step still tags the commit
(`scenario/<name>/<checkpoint>`) for manual inspection with `git checkout`,
but nothing restores state from it; a prior `--from <checkpoint>` flag that
only skipped steps without restoring their state was removed.

### Available Scenarios

| Scenario | What it tests |
|----------|--------------|
| `intake-basic` | Wakeup creates intake jobs for unjobbed inbox items; reactor processes them |
| `tick-basic` | Scheduled script listing, dry-run, execution, skip-if-recently-run |
| `tick-chain` | `create-after-success` chaining between scheduled scripts across ticks |

### Creating a New Scenario

Each scenario is a self-contained directory under `~/src/boxes/scenarios/<name>/` with its own git repo as the test box.

**Directory structure:** `bbx init` scaffolds the one-root layout by default (`package.json`/`tsconfig`/`src/` plus the underscore operational areas at the same root — see `docs/box-layout.md`), so a freshly-created scenario's `box/` looks like:
```
~/src/boxes/scenarios/my-scenario/
  scenario.yaml      # step definitions (required)
  stubs.yaml         # time/HTTP stubs (optional)
  stubs/             # stub response files (optional)
    feed.xml
    article.html
  setup.md           # human-readable description of what this tests
  box/               # the git repo — a real box initialized with bbx init, boxRoot itself
    _content/inbox/  # pre-seeded test data
    _config/         # connector configs, schedules, etc.
    ...
```
The existing scenarios in the table above (`intake-basic`, `tick-basic`, `tick-chain`) predate this and haven't been migrated — `getBoxShape` is strict and rejects any marker without `shapeVersion: 3`, so these scenarios need `bbx migrate` run against them (or recreating) rather than being a supported second shape.

**Steps to create:**

1. **Create the directory and initialize a box:**
   ```bash
   mkdir -p ~/src/boxes/scenarios/my-scenario/box
   cd ~/src/boxes/scenarios/my-scenario/box
   git init
   bbx init .
   ```

2. **Seed the box with test data.** Put cards in `_content/inbox/`, configure connectors in `_config/connectors/`, add scheduled scripts, etc. Commit everything — the scenario runner requires a clean `main` branch as starting state.

3. **Write `scenario.yaml`** with steps. Each step runs a shell command (usually a `bbx` command) and validates the result. See the format description above.

4. **Write `stubs.yaml`** if your scenario involves HTTP (connector syncs, article fetches). Freeze time with `time:` to make timestamps deterministic. Put response files in `stubs/`.

5. **Write `setup.md`** describing what the scenario tests, what stubs are used, and what the expected outcome is. This is for humans, not the runner.

6. **Test it:**
   ```bash
   bbx scenario run my-scenario --dry-run   # verify steps parse correctly
   bbx scenario run my-scenario             # run for real
   ```

**Design principles for scenarios:**

- **Each scenario tests one pipeline or behavior.** Don't combine unrelated features. `intake-basic` tests intake jobs only; `tick-basic` tests scheduled-script behavior only.
- **Seed the minimal data needed.** The `intake-basic` box has just 2 memos in inbox — enough to verify the behavior, not so much that agent processing is slow or unpredictable.
- **Use `--skip-*` flags** on `bbx wakeup` to isolate phases when you don't need the full wakeup cycle.
- **Use checkpoints** on steps that are expensive (agent runs) to tag the resulting commit for manual inspection later. Scenarios always run from the beginning, so a checkpoint doesn't let you skip re-running earlier steps.
- **Prefer `script:` validations** for structural checks (files exist, XML contains expected content). Use `prompt:` validations only for things that require judgment (quality of generated text, correct interpretation of ambiguous input).
- **`prompt:` validations cost money.** Each one invokes a Claude agent with up to 5 turns / $0.50. Use them sparingly.

### Managing Scenarios

**Inspecting a failed run:** The test branch `test/<name>/<timestamp>` is preserved after the run. Check it out to see the state at failure:
```bash
cd ~/src/boxes/scenarios/my-scenario/box
git branch                        # list test branches
git checkout test/my-scenario/... # inspect the failed state
git checkout main                 # return to clean state
```

**Cleaning up old test branches:**
```bash
git branch | grep 'test/' | xargs git branch -D
```

**Updating a scenario's test data:** Edit files in the box on `main`, commit, then re-run. The runner always starts from a clean `main`.

**Scenarios are git repos** — you can use standard git operations. The box inside each scenario is a real box; `bbx` commands work normally when you `cd` into it.

## 3. Knowledge Audits

**Location:** Tests in `src/dev/knowledge-audits.yaml`, runner in `src/dev/knowledge-audit.ts`, reports in `src/dev/reports/`
**Run:** `pnpm knowledge-audit run [--filter <id-or-tag>]`

Knowledge audits test what the agent _knows_ rather than what the system _does_. They run prompts against a Claude agent in a box and check whether the agent answered from loaded context (knows directly), followed a doc reference (knows about), or had to search (discoverable).

**When to use:** Verifying that documentation, agent guides, and conditional rules are working — that the agent has the right information at the right time. Not for testing system behavior.

See [knowledge-taxonomy.md](knowledge-taxonomy.md) for the full knowledge taxonomy and test prompt guide.

### Test Definition

```yaml
tests:
  - id: box-structure-inbox
    prompt: "Where would you look for unprocessed incoming items?"
    expected_level: knows_directly
    watch_for: "Names _content/inbox/ directly without searching"
    correct_contains: ["_content/inbox"]
    should_read: ["node_modules/beebox/box-docs/card-memo.md"]   # optional
    should_not_read: ["some/file.md"]              # optional
    tags: [navigation]
```

### How It Works

1. Runs the prompt via `bbx prompt` in the test box
2. Parses the session transcript to extract: files read, searches, bash commands, response text
3. Automated checks: `correct_contains` (substring match), `should_read`/`should_not_read` (file access)
4. Generates a Markdown report with results + blank assessment field for human review

Reports go to `src/dev/reports/audit-report-<timestamp>.md`.

## 4. Session Critiques

**Location:** Report generator in `src/dev/lib/session-report.ts`, subagent in `.claude/agents.json`
**Run:** `@session-critique <session-id>` (or `@session-critique latest`)

Session critiques evaluate whether CLI tools helped or hindered the agent during real agentic sessions. Unlike knowledge audits (which test what the agent knows), session critiques test whether the tools the agent used gave it good output.

**When to use:** After observing a session with unusual behavior — the agent took too many turns, used raw git/grep instead of `bbx` commands, or seemed confused by command output. Also useful as a periodic check on CLI usability.

### How it works

1. **Report extraction:** `bbx session <id> --tool-report` parses the session JSONL and produces a markdown report containing:
   - User and assistant text messages
   - Bash commands with their **full output** (the key differentiator from `bbx session` which skips output)
   - Read/Write/Edit as one-liner summaries for context
   - Grep/Glob with abbreviated results

2. **Critique subagent:** The `@session-critique` agent reads the report and evaluates it against five criteria:
   - **Unhelpful output** — Did a `bbx` command produce output the agent ignored or misinterpreted?
   - **Missing commands** — Did the agent cobble together raw commands when a `bbx` command should have existed?
   - **Wrong tool** — Did the agent use the wrong tool (e.g., `Read` for scanning many files)?
   - **Bad error messages** — Did errors lead the agent to the fix or cause flailing?
   - **Wasted effort** — Retry loops, redundant reads, unnecessarily complex approaches?

3. **Output:** Structured findings with evidence, impact, and concrete suggestions (CLI format changes, new commands, `.claude/rules/` hints).

### Running a critique

```bash
# From within a box directory:
bbx session --list                   # find session IDs
bbx session <id> --tool-report       # generate report for a specific session
bbx session --latest --tool-report   # most recent session

# Or use the subagent (from Claude Code in this project):
# @session-critique <session-id>
# @session-critique latest
```

The subagent runs `bbx session` itself, so it needs a box directory context.

### Acting on findings

Session critiques produce actionable suggestions. The typical workflow:

1. **Run a critique** on a session that seemed inefficient or problematic.
2. **Review findings.** Each has a category and suggestion.
3. **For unhelpful-output:** Modify the `bbx` command's output format — trim noise, surface key info earlier, add structured markers the agent can parse.
4. **For missing-command:** Consider whether a new `bbx` subcommand or flag would help. Only add one if the pattern recurs across sessions.
5. **For wrong-tool:** Add a `.claude/rules/` hint that triggers when the agent is in the relevant context, pointing it to the right tool.
6. **For bad-error:** Improve the error message in the CLI command. Good errors name what went wrong, what file/card caused it, and what to do next.
7. **For wasted-effort:** Usually a prompting issue. Check if the system prompt or agent guide is missing guidance for this task type.

Not every session has problems. If the critique comes back clean, that's a positive signal that the tools are working.

### Comparison with other test types

| Aspect | Knowledge audit | Session critique |
|--------|----------------|-----------------|
| Tests | What the agent knows | How well tools serve the agent |
| Input | Controlled prompts | Real session logs |
| Automated? | Yes (run suite) | Semi-manual (pick sessions to review) |
| Frequency | Periodic suite runs | After observing issues |
| Fixes | Documentation, agent guide, rules | CLI output, error messages, rules |

## 5. Card Validator Hook

**Location:** `src/core/sdk-hooks.ts` (`cardValidatorHook`)
**Trigger:** Runs automatically during agent sessions on `PostToolUse` of `Write`/`Edit`

Not a test you run manually, but a live validation hook. When an agent writes or edits a `.card` file, the hook calls the card linter (`src/core/card-lint.ts`, built on the card primitives absorbed from the former `cardworks` package into `src/cards/`) in-process and feeds any issues back as `additionalContext`. This catches frontmatter/schema issues during agent work rather than after.

Also enforces directory structure rules (e.g., trick scripts must be in subdirectories of `tricks/scripts/`).

## 6. Frontend Dev Stubs

Some frontend bugs only manifest against real layout and measurement — scroll
behavior, virtualization, streaming-driven reflow — and can't be reproduced in a
doctest. For these, drive the running app with `bin/browse` (see
`.claude/skills/browse/SKILL.md`, monorepo root) and use a dev stub to make the
input deterministic instead of depending on a live agent response.

### `/fakestream` — deterministic chat streaming

**Location:** `src/frontend/src/machines/chat-actors.ts` (`runFakeStream`)
**Trigger:** Send a chat message beginning with `/fakestream`.

Instead of calling the backend, the chat machine plays a timed script of
`STREAM_TEXT` events, growing the assistant bubble at a controlled rate with no
API calls. This reproduces streaming-UI bugs (scroll-follow, layout jitter)
frame-for-frame.

```
/fakestream [chunks] [intervalMs] [chunkLen]
```

- `chunks` — total text events to emit (default 200)
- `intervalMs` — delay between events (default 40)
- `chunkLen` — approx chars per event (default 25)

Example: `/fakestream 2000 30 25` streams ~50k chars over ~60s. The message
list is a single scroll container with `data-testid="chat-scroller"` driven by
the `useChatScroll` controller (`components/chat/chat-scroll.ts`). Measure
scroll state from the browser to assert behavior deterministically — with
`bin/browse eval --no-wait`, since a plain `eval` waits out the stream and hands
you the settled page:

```js
// Under the write-on-user-action model, fromBottom GROWS as the reply streams
// and scrollTop does not move: nothing follows the bottom.
const s = document.querySelector('[data-testid="chat-scroller"]');
({ fromBottom: s.scrollHeight - s.scrollTop - s.clientHeight, scrollTop: s.scrollTop, scrollHeight: s.scrollHeight });
```

**Driving the scroll behavior.** The two instruments — the scenario table at
`/dev/chat-scroll` and the step-by-step `bin/browse` procedure, plus the
real-device checklist — live in
[chat-scroll-testing.md](chat-scroll-testing.md). Note that `/fakestream`
content **vanishes at finalize in a server-backed session** (the authoritative
history has no such turn), so sample while it streams, not after.

The stub is gated purely on the message prefix, so it ships harmlessly — a real
message never starts with `/fakestream`.

## Smoke Tier (a real box boots and is walked — a merge gate)

**Run:** `bin/smoke` (add `--box <slug>`; `--no-restart` when debugging the
walk itself). ~30s, hard-fails, never selected by the test graph.

The only tier that answers "does the app actually run". It restarts this
checkout's dev-server generation through the router's control socket, waits for
the box's **backend** to answer `/api/health` (vite serves pages seconds before
Fastify is up, and a box child reloads itself after a post-commit CLI rebuild —
both looked like a 502 flake until 2026-08-26), then walks it in a browser: the chat page renders its shell,
the app bar's place menu opens and lists landmarks, selecting one moves you there, `/browse` lists the box's
real content, a card opens and renders, and the page raised no uncaught errors.
No model turns — nothing that spends tokens or waits on an agent.

It exists because three escapes on 2026-08-25/26 passed typecheck, lint and
their selected tests while the app was broken: the code was right and the
*state* was wrong (a missing frontend build, broken global `~/.codex` state, an
SDK item outside its own union). Only a real box on the real machine shows
those.

**Where it runs.** `/finish` names it on the decision sheet for any diff that
touches a deployed path (`bin/deployed-paths.ts` — the same rule the deploy
hook uses to decide whether a commit ships), and `bin/finish-verify` runs it
*before* `bin/land`. It is a gate, not a post-merge alarm, because the router
runs TypeScript straight off disk and never reloads it: after the merge, the
main checkout's running generation is still the old source, so there would be
nothing correct to point at. The worktree at that moment already contains main
and is byte-identical to what lands.

A failing test file gets a flake re-run; the smoke walk does not. It boots one
real box and either that works or it does not.

**Post-deploy**, `deploy/deploy.sh` runs the server-side half: hub `/healthz`,
a `/healthz/canary` that cold-starts one real box, and an assertion that the
shipped `src/frontend/dist/index.html` exists. That last one is the condition
`registerSpaFallback` branches on — without the build, every page navigation
404s while both health checks stay green. It asserts the file rather than
probing a URL because the hub redirects an unauthenticated navigation to login
before the child is reached, so a URL probe answers 302 either way. It is not
duplicated in the local walk: in dev, page requests are served by vite and
never reach that handler at all.

**Every run is logged, so the tier can be trimmed on evidence.** Each walk
appends a line to `callback-smoke-log.jsonl` in the shared git dir (beside the
test ledger, and shared by every worktree on the machine for the same reasons):
the verdict, which step failed, and every step's outcome and duration.
`bin/smoke --report` folds it into per-step counts — how often each step ran,
how often it caught something, what it costs at p50. A step that has never
failed across many runs is paying rent out of the budget, and the report names
those once there are enough runs for a clean record to mean anything.

**Breaking it on purpose: declare it.** Proving the tier can still go red means
breaking something, and the resulting red is indistinguishable in the log from
one the tier caught for real — the first weekly review duly read one as an
intermittent worth watching. So say what you broke:

```
BBX_SMOKE_FAULT_INJECTION="hub throws at import" bin/smoke
```

The reason is stamped on the run, the walk says so loudly while it runs, and
every count in `--report` and in the weekly review excludes it. A step's
`forced` column counts these separately from `failed`: firing on demand proves
the step is wired up, never that it has caught anything.

Read `ran` as the denominator, not the run count: the walk stops at the first
failure, so a late step has seen fewer runs than an early one. Steps are keyed
by a stable id, not their printed name, so rewording a step keeps its history.

**A weekly schedule reviews the tier's shape** (`schedules/smoke-review/`). It
gathers the log, the window's failures, and the bugs filed that week, then hands
them to a session that asks two questions: is every step still earning its
place, and is there something the walk should be checking that it isn't. The
second half is necessarily after-the-fact — a bug that reached `main` and was
only visible on a running box is what "the walk has a hole" looks like, and the
hourly full-suite run's own issues are the sharpest evidence for it. The session
files an issue; it does not edit the tier, because trimming a step or adding one
belongs to a session that can run the walk and see what happens.

**It is disruptive, on purpose.** The walk stops this checkout's dev-server
generation and closes the shared browse session (tours and interactive
`bin/browse` share one Chrome). If the boxholder has this worktree open in a
browser, their tab reloads. That is the cost of testing the code that is about
to land rather than whatever was running.

**Known gap.** A change confined to `bin/` gets no smoke walk, because `bin/`
ships nothing and the "code-related" rule is deliberately the deploy hook's. A
`bin/router.ts` change that breaks the dev router is therefore not gated here.

## Tours (the app's walk, written down — kept true weekly, not a gate)

Scripted browser walks (`bin/tour <name>`, scripts in `test/tours/`)
that produce review artifacts: desktop+mobile screenshots, AX-tree
snapshots, axe-core reports, and soft-assertion findings per
checkpoint. Ungated — findings never fail an exit code and artifacts
are gitignored — but not unrun: `schedules/tour-check/` walks every
tour weekly, *edits* a tour when a miss is explained by a deliberate
change (a landing, plan, or issue), and files an issue for anything
else. That asymmetry with `smoke-review` (which files and never edits)
is by design: a gate goes red on every intended UI change, so tours
need a session with judgment rather than a threshold. Full reference —
running, reviewing artifacts, writing conventions, the edit-vs-finding
rule, and when NOT to use them: [tours.md](tours.md).

## Field Tests (agent-operator, expensive, not a gate)

Field tests use a persona operator, a disposable real box, its agents, and the
real web UI to test realistic discoverability and end-to-end use. They run
weekly or on demand and never gate CI or a merge. See the current
[field-testing runbook](field-testing.md) for commands, artifacts, visual-review
rules, and issue-triage boundaries. The
[implemented plan](implemented-plans/agent-field-tests.md) is design history.

## Choosing the Right Approach

| Question | Approach |
|----------|----------|
| Does this function return the right value? | Unit test |
| Does the full pipeline produce the right output? | Scenario test |
| Does the agent know where to find X? | Knowledge audit |
| Did the CLI tools help or hinder the agent? | Session critique |
| Does a card validate after agent edits? | Card validator (automatic) |
| Does the streaming UI scroll/reflow correctly? | Frontend dev stub (`/fakestream` + `bin/browse`) |
| Does the app still boot and work at all? | Smoke tier (`bin/smoke` — runs automatically at `/finish` for a code change) |
| Does this page render sane at both viewports / pass axe? | Tour (`bin/tour <name>` — see [tours.md](tours.md); walked and kept true weekly, not a gate) |
| Is every state of this component reachable and right? | Dev harness route (`/dev/…`, real components over injected fakes) |
| Is this realistically discoverable/usable end-to-end, through the real UI? | Field test (`bbx field-test run <scenario>` — expensive, weekly/manual, never a gate) |

**Overlap:** Some things could be tested at multiple levels. Prefer the lowest level that catches the bug:
- A template generating bad XML → unit test (fast, deterministic)
- An agent not using `<chat-response>` tags → scenario test (needs agent behavior)
- An agent not knowing about a command → knowledge audit (tests documentation)

## Adding New Tests

### New doctest (preferred)
Create `test/<name>.doctest.md`. Write prose explaining the behavior, with fenced code blocks containing examples. See [doctest syntax](../../agent-doctest/docs/syntax.md) for syntax. Runs automatically with `npm test`.

### New traditional test
Create `test/<name>.test.ts`, import from `tap`. Use for route integration tests, meta-tests, or anything needing complex setup that doesn't read well as documentation.

### New scenario
Create `~/src/boxes/scenarios/<name>/` with `scenario.yaml` and optionally `stubs.yaml` + `stubs/` directory. Test with `bbx scenario run <name> --dry-run` first.

### New knowledge audit
Add entries to `src/dev/knowledge-audits.yaml`. Run with `--filter <id>` to test individually.

### New session critique

Pick a session to review (use `bbx session --list` from a box), then run `@session-critique <id>` from Claude Code in the beebox project. Review the findings and apply fixes per the "Acting on findings" guide above.

## Periodic Checks

Not automated — run these occasionally and fix what they find.

### Session critiques

Review recent agentic sessions (intake triage, capture processing, chat handling) for tool quality issues. Pick sessions that seemed slow or where the agent used workarounds. Run `@session-critique <id>` and act on findings. See § Session Critiques above.

### Documentation graph

`npx tsx src/dev/doc-graph.ts > docs/doc-graph.md` — scans all `.md` files, extracts cross-references, reports orphans and broken links. Review description quality at each reference site. Fix issues, regenerate, commit.

### Supplemental linters

These catch issues the pre-commit hook doesn't:

```bash
pnpm lint:oxlint    # Ambiguous constructors, useless spreads, identical branches
pnpm lint:knip      # Unused files, exports, dependencies
pnpm lint:circular  # Value-import circular dependencies (type-only cycles are OK)
```

## Future Directions

Key ideas not yet implemented:

- **Self-describing services** — Services export `description`, `examples`, and `properties` alongside their functions. Tests get generated from these.
- **Property testing** — Semantically meaningful invariants (like `parse(serialize(card)) === card`), not random fuzzing.
- **CGI-style route testing** — Test routes as pure state→output functions rather than spinning up servers.
- **Assertive logging** — Soft assertions in production code that surface through the logging system, caught by an error harness in tests.