Health Checks

Manual runbooks for periodic checks on the deployed beebox server. Each section is self-contained — follow the steps, report the verdict, and suggest a next action if unhealthy. These are designed to be driven by a local Claude Code agent that has SSH access to box.example.com.

Hub health endpoints (/healthz, /healthz/canary)

The hub exposes two diagnostic endpoints, both requiring the BBX_DIAG_API_KEY bearer token (Authorization: Bearer <key>). An unauthenticated caller gets 401 — the endpoints used to be open and leaked slugs/PIDs/ports, so they're now gated like the box server's own /healthz. Both run on the server's localhost (http://localhost:3210); externally they're reachable at https://box.example.com/... with the same key.

GET /healthz — passive verdict. Reports status: "ok" (HTTP 200) or status: "unhealthy" (HTTP 503), plus a state per box. On the server that state is read from the box's systemd unit (src/hub/socket-provider.ts): an idle box's service is inactive and reads stopped, which is its normal resting state and not a fault, so an idle fleet reads ok. The verdict is liveness, not readiness: it's unhealthy only if a box is broken — crash-looping (starting with consecutiveFailures > 0) or crash-budget-latched (unhealthy). restarts is a lifetime counter shown for information; the verdict never keys on it (a box that blipped once long ago would otherwise pin the hub red forever). Derivation lives in src/hub/health.ts. This endpoint has no side effects — safe for an uptime monitor to poll.

GET /healthz/canary — active box check. Cold-starts one box (a connection to its socket starts the unit), then fetches that box's own root /healthz (authenticated) and returns 200 only if the box answers 200 — otherwise 503, naming the slug. The box-healthz step matters: a box that accepts the connection but whose health handler is broken would pass a socket-only check; requiring the box's healthz 200 makes "canary ok" mean the box actually answered. This is what catches a fleet-wide startup break — e.g. a native-module ABI mismatch after a Node major bump — that the passive verdict can't see, because most boxes rest stopped and report nothing. ?box=<slug> names the box to canary; unset picks the first configured slug. It wakes a box (leaving it resident until its idle exit, thirty minutes), so it's a deploy-time / on-demand check, not something a monitor should poll. The deploy runs it automatically (deploying).

Why the canary exists at all: during the 2026-07-16 Node 22→24 upgrade every box child crash-looped on a better-sqlite3 ABI mismatch while the old /healthz returned a constant 200 — the deploy verified "healthy" while no box could serve a request. The passive verdict now catches a crash-looping box; the canary catches a break on boxes that were never started.

Engine quota (waiting is not failing)

When bbx health shows tasks as waiting plus an engine: waiting on <provider> quota until <t> line, the box's agent engine (Claude or Codex) is out of usage quota — a deferred-recoverable condition, not a fault. Nothing needs fixing: scheduled work skips until the reset time, consecutiveFailures does not accrue, and the boxholder was notified once with the reset time. The machine-level record lives at ~/.local/share/beebox/engine-availability.json and self-expires at the reset. Investigate only if waiting persists well past the stated time (the store then shows a fresh episode — quota was exhausted again immediately, which is a capacity problem, not a code problem).

Inconclusive (? is not ✗)

When bbx health shows a task as ? <name> inconclusive with last run's check reached no verdict; the work itself completed, the run did its work and its checker fell short — typically a procedure's validate review hitting its turn cap. Nothing found a defect. Do not redo the work on this signal, and do not read it as a failure: consecutiveFailures does not accrue, no alert fires, and bbx health does not exit 1 for it. The error line under the task names the procedure, the step, and the reason.

What to do with it: if the same task reads inconclusive repeatedly, the fix is a budget or prompt fix on the judge (a turn cap set too low, or a review prompt that wanders), not a fix to the work. The reason tags distinguish the cases — see docs/procedure-implementation.md → Instructions.

health.check snapshot vs fresh

The box-level checks (runHealthChecks — permissions, API keys, annex, nav card, engine) are deep and slow: a subprocess claude auth status, the git-annex doctor over the attachment trees, a tmp-capture/ walk, a dozen serial fs probes. Measured 580–650 ms on prod, and they rode in the dashboard's tRPC batch, so every dashboard load waited on them.

GET /api/trpc/health.check therefore answers from a stale-while-revalidate snapshot (src/webapp/trpc/routers/health/checks/snapshot.ts), per box, per bbx serve process:

What the staleness bound actually is: a change shows up two requests after the TTL expires, not one, and only if something asks again. A client that stops polling never sees the new verdict — which is fine, since nothing is displaying it either, but it means "worst case 60 s" is wrong. For the dashboard (which refetches on every visit) it is 60 s plus one page view. Two things bound the damage: writes are ordered by when each computation started, so a slow background refresh can never overwrite a newer result; and a background refresh that throws is latched, so the next reader recomputes in the foreground and gets the error rather than another serving of the last-known-good report.

Anything that must not be stale asks for a live run:

Every deploy restarts the box units, which empties the cache, so a deploy never serves a pre-deploy verdict.

Box growth (files, directories, and Git history)

The scheduler measures each box at most hourly and stores the latest baseline in .beebox/box-growth-health.json. A native find subprocess streams NUL-delimited type/path pairs so the Node scheduler does not reopen every directory or retain every path. The scan counts files and directories separately, records the largest subtrees, and samples Git commit/object growth. It counts symlink entries as files but does not follow them, and excludes .git, .beebox, and node_modules directories at any depth. The filesystem walk has a 10-second budget. If it reaches that deadline or encounters a traversal error, the state retains the counts and attribution already streamed, marks them as incomplete lower bounds, and warns. Rate checks pause until two complete samples are available, so partial traversal does not look like new growth.

Size alone is never a finding. A fixed level (it was 250 directories or 1,000 files) fired on any box with an email connector and could not clear, which taught the boxholder to ignore the dashboard. The dashboard warns on growth rate only:

Disk use comes from du -sk (core/box-growth/bytes.ts), which reads the same on Linux and macOS. Content is every top-level entry except .git, .beebox and node_modules; .beebox (indexes, caches, logs) is measured on its own so engine data never reads as content growth. Git's own size is in the history sample. A measurement taken before bytes were recorded, or whose du failed, has no disk-use finding.

Rate checks require two complete samples 30–120 minutes apart. A partial scan, first measurement, long scheduler outage, or longer measurement gap does not infer an hourly rate.

The filesystem path is the authoritative source attribution. Git history is a supporting signal only: older commits do not consistently carry a Created-By trailer, while connector-owned paths remain identifiable regardless of the commit message or trailer coverage. The check reports anomalies but never deletes, prunes, or moves box content. If Git cannot be sampled but filesystem growth is healthy, the check remains healthy and reports that history detail as unavailable; Git failure is appended to a real growth warning when both occur.

The dashboard offers two different owner decisions:

Neither action disables monitoring: a rate above the stored thresholds warns again.

The directory count that degrades something is the file watcher's limit (MAX_WATCHED_DIRS, 1,024). The watcher skips dot-directories and high-churn trees, so the growth scan cannot predict it; the watcher records when it runs out instead, and the box-watch-limit check names the path below which live updates are off. Only the server runs a watcher, so bbx health never shows this check.

If the warning is unexpected, inspect the named subtree before accepting it:

find content -type d | wc -l
find content -type f | wc -l
git count-objects -v

Then identify the producing connector, import, capture, or procedure and stop the source of unintended growth. Do not remove content merely to clear the warning. An incomplete scan, a failed scan, or a measurement older than 26 hours is itself a warning; inspect .beebox/scheduler.jsonl for box-growth-scan errors. A local box that has never run the scheduler reports monitoring as not yet run without degrading health. Do not delete the state file to dismiss a warning: use one of the explicit dashboard actions. If the state file is corrupt, monitoring leaves it untouched and reports the validation error so an operator can inspect or remove it deliberately. If state disappears while the scheduler is running, the in-process hourly backoff prevents a rescan loop and the replacement baseline is surfaced as a warning until an owner accepts it.

template-updates (a fix that never reached the box)

bbx health's box-checks section carries a template-updates check reporting upstream template changes that were parked rather than written: the box's copy of a shipped procedure, guide, or schedule card had diverged, so the new version went to _config/_template-updates/<path> for review instead of clobbering the local edit. The message lists the parked paths and ends with the same resolution sentence bbx status prints — copy the parked file over the live one, or discard the parked copy.

Severity is normally warning: a parked update is a pending choice, not a defect, and must not fail a deploy. It escalates to error when a parked path is the procedure card or task card behind a scheduled task that currently reads failing or inconclusive. That combination is the trap this check exists for — the task's own fix is already on disk and unread. The message says which of the two it is (…belongs to a task that is failing: X / …belongs to a task whose last check reached no verdict: X): both escalate, but a failing task is broken and an inconclusive one is only unjudged. The scheduled-task list prints the same association under the affected task:

  ✗ refresh-maps          failing ×5     last attempt 7h ago, never succeeded
      error: step refresh failed
      parked update: _config/_template-updates/_config/procedures/refresh-maps.procedure.card — the fix may already be on disk

The association also rides on TaskHealth.parkedTemplateUpdates, so bbx health --json, the dashboard, the session-start snapshot, and proactive health alerts all carry it. Escalation needs the schedule evaluation, which only bbx health loads — the dashboard and /api/health report the un-escalated warning.

google-auth (is the Google grant still alive?)

bbx health's box-checks section and the dashboard's health warnings both carry a google-auth check. It reports one of three things: nothing at all (Google isn't configured for this deployment, or this box never connected it), Google authorization is live (last verified …), or a warning that the authorization expired or was revoked with a /<box>/admin?reconnect=google link. It's a warning, not an error: Google features pause, the rest of the box works, and bbx health's exit code stays 0.

The check is a pure reader — it never makes a network call, so it's safe to poll. What keeps it honest is a forced token refresh (probeGoogleAuthIfStale) that the scheduler daemon runs at most about once a day; the freshness stamp lives in the shared token record, so on a multi-box server the first box to tick each day probes and the rest skip. Ordinary Google usage refreshes the stamp for free. A box with no scheduler running will show a stale last verified rather than a wrong verdict.

On a flip to broken, the boxholder gets one notification per breakage over the iPhone app and Web Push. Reconnecting (admin page, Google Services section) clears the state and re-arms the alert for a future relapse. Operator-facing detail is in google-setup.md; design notes in implemented-plans/google-auth-reauth-health.md.

connector-activity (a connector that went quiet or keeps failing)

A connector can stop producing while every sync reports success: a filter that no longer matches, a permission revoked upstream, a cursor past everything. The connector activity record (see connectors.md) lets the box notice. connectors/activity-verdict.ts holds the rule:

Each quiet or failing stretch is an episode, stored in the activity record. The scheduler daemon records each new episode once in its log, beside the task-health and Google-auth passes (core/schedule/box-alerts.ts). A new failing episode also sends one quiet notification naming each failing connector and the day it began; it says to reconnect Google only when a Google connector is failing and the grant is missing or dead. A quiet episode sends nothing. The dashboard shows a connector-activity:<name> warning with a This is expected button until new items arrive, an ok sync ends the failing stretch, or the owner dismisses it. A dismissed episode stays dismissed; the next episode is new and alerts again. An open quiet episode does not end when its baseline ages out of the record. A record that fails its schema shows as a connector-activity warning and is never reset, which would discard dismissals.

google-docs-api (Drive syncing Google Docs without the Docs API)

When a Doc pull's getDocument fails for a reason the boxholder can fix — the Docs API disabled on the Cloud project (403 SERVICE_DISABLED, with the enable link) or a grant without documents.readonly — the Doc still syncs, without revision tracking or the lossy: check. The Drive connector folds each pass's outcomes into one diagnosis in _bookkeeping/connectors/google-docs-api.state.json (connectors/google-drive/docs-api-status.ts): a fixable failure opens it; a complete pass whose every Doc pull reached the API (or that found no Doc cards) clears it; a pass with failures, a folder cap, or any transient failure leaves it as it was. On-demand syncs (bbx drive mount, bbx drive add, Settings "Sync now") do not write it; the next connector pass does.

The check is one google-docs-api warning with the fix. There is no dismissal: it lasts until the API answers or Drive stops syncing. Every reader hides it at once when Drive is disabled for the box or Google is disconnected. While the grant needs reauthorizing, Drive is paused and the warning stays, worded as its last sync. The same diagnosis is the Drive line in the Google settings section, and on the first send of each chat session it rides in the <chat-app health> attribute so the agent mentions it. A file that fails its schema shows as the warning's text and is never reset.

gmail-drafts (drafts the connector stopped retrying)

A Gmail draft upload that fails because of the card itself — a missing field, an in-reply-to ref that resolves to nothing, a message Gmail rejects with HTTP 400 — is stamped gmail-draft-error: on the email-outbound card and not retried (connectors/gmail-draft-card.ts). The gmail-drafts warning lists those cards until each is fixed and its gmail-draft-error line deleted, or the card is deleted. Failures that are not the card's fault (an expired grant, a network or server error) are retried every sync, and the connector-activity failing alert covers them. They are not retried forever: the first one stamps gmail-draft-failing-since:, and a draft still failing 7 days later is stranded with gmail-draft-error: like a card problem. A successful upload clears the stamp.

claude-update (nightly Claude Code self-update)

Why this exists

The server runs claude update nightly via claude-update.timer → claude-update.service → deploy/claude-update.sh (wrapper). If the timer or wrapper ever breaks silently, the server's Claude Code could drift far behind upstream. The system has three evidence trails (wrapper log, syslog tag claude-update, and per-unit journalctl) — this check verifies all three are alive and consistent. See server-operations.md for the underlying design.

Run this

deploy/prod-ssh 'bash -s' <<'EOF'
  echo "=== wrapper log (last 40 lines) ==="
  tail -n 40 /var/log/beebox/claude-update.log
  echo
  echo "=== timer ==="
  systemctl list-timers claude-update.timer --no-pager
  echo
  echo "=== service status ==="
  systemctl status claude-update.service --no-pager | head -20
  echo
  echo "=== syslog (tag=claude-update, last 8 days) ==="
  journalctl -t claude-update --since "-8 days" --no-pager
  echo
  echo "=== installed version ==="
  sudo -u beebox /home/beebox/.local/bin/claude --version
EOF

Success criteria

All of these must be true:

  1. The wrapper log's most recent ok: line has a UTC timestamp within the last 48 hours.
  2. No FAIL: lines in the wrapper log within the last 8 days.
  3. No entries in journalctl -t claude-update within the last 8 days (both the wrapper and the OnFailure unit only emit syslog on failure, so any entry = a failure).
  4. systemctl list-timers shows "Last" within the last 48h and "Next" within the next 24h.
  5. systemctl status claude-update.service shows the last run as status=0/SUCCESS (the service is Type=oneshot, so "inactive (dead)" with status 0 is the expected idle state).
  6. claude --version returns a plausibly recent version (≥ 2.1.119 as of the time this doc was written; just sanity-check it isn't wildly stale).

Report format

One line:

Then a short paragraph with the supporting evidence if unhealthy.

Recovery actions

If the check is unhealthy, try these in order:

  1. Force a run: ssh root@<ip> systemctl start claude-update.service — then re-inspect the wrapper log and journalctl -u claude-update.service -n 50.
  2. Look for auth failure: Claude Code credentials can expire if the user logged out locally (see server-operations.md → "Claude Code credentials"). If claude update itself is failing, re-transfer credentials from macOS keychain.
  3. Look for a moved/missing wrapper: ls -la /opt/beebox/beebox/deploy/claude-update.sh — should be executable and owned so root can execute it. deploy.sh handles this but a bad deploy could leave it wrong.
  4. Look for a disabled timer: systemctl is-enabled claude-update.timer — should be enabled. If someone disabled it, systemctl enable --now claude-update.timer.
  5. Check disk space on /var/log/beebox — a full disk would prevent the log from being written, which would silently mask failures.