Server operations

Working with the already-provisioned server: reaching it, running commands and scripts on it, its diagnostics, and the nightly Claude Code update.

What it is

Reference for the running beebox server (production at box.example.com). Provisioning is its own page; this one covers the already-provisioned system.

The production tools

# Interactive shell
./deploy/prod-ssh

# Run a command
./deploy/prod-ssh systemctl status beebox-hub

Uses agent forwarding (-A) so your local SSH key works for GitHub operations on the server. In a worktree, the command falls back to the main checkout's gitignored deploy/target.env; a non-empty local copy takes precedence. This fallback is for diagnostics only: deploy.sh intentionally requires target.env in the invoking checkout.

prod-curl and prod-browse

prod-curl and prod-browse authenticate as the configured owner (BBX_OWNER_EMAIL) to inspect production behind the OAuth wall. This grants no new access: both commands require the boxholder's existing root SSH key, and must never be modified to mint a session for another identity without the boxholder's express, in-the-moment permission.

# Fetch HTML or an API response; extra arguments pass through to remote curl.
./deploy/prod-curl /test1/ -sI

# Open the rendered app in bin/browse's clean browser profile.
./deploy/prod-browse /test1/
bin/browse screenshot --slug prod

prod-curl keeps the signed session cookie on the server. prod-browse puts it in the local isolated browser profile and also requires BBX_DEPLOY_PUBLIC_URL in deploy/target.env; from a worktree it falls back to the main checkout's copy like everything else there. Never print or persist either cookie or URL in tracked files.

Connecting for debugging and inspection

For ad-hoc inspection of the running server (reading logs, checking box state, running bbx commands, etc.) — not initial setup, which is provisioning.

Always SSH to the IP, never the hostname. box.example.com resolves to Cloudflare (the web proxy in front of the box), so ssh [email protected] fails with "No route to host." Use the pinned IP:

deploy/prod-ssh
# or for data work as the service user:
ssh beebox@$(beebox/deploy/deploy-target.sh get BBX_DEPLOY_HOST)

Do not pass -o StrictHostKeyChecking=no. That flag belongs in first-contact provisioning scripts (add-box.sh, hetzner/provision.sh) where the host hasn't been seen yet. For ad-hoc work the host is already in ~/.ssh/known_hosts and disabling the check just removes a real safety. If you get a host-key error, investigate it — don't suppress it.

Common inspection targets (run via deploy/prod-ssh '<cmd>'):

Running bbx commands on the server — as the box's own user, so file ownership stays correct and the command sees the box's own secret store:

# after sshing in as root: the box's user, with the units' environment
su - bbx-<box> -c "set -a; . /etc/beebox/box.env; set +a; cd /var/lib/beebox/<box>/box && bbx validate"
# hub-side commands run as the hub user
deploy/prod-ssh "su - beebox -c 'bbx engine hub add-box --help'"

⚠️ Source /etc/beebox/box.env for box commands. The box units load it (EnvironmentFile=), but an ad-hoc SSH invocation does not inherit it. Most visibly, BBX_UNIT_HOST=1 from that file is what points Google at the box user's own token file (~/.config/beebox/google-tokens.json); without it bbx health's connector-presence probe looks only for a legacy google.secret.json and reports a false blocked: missing connectors for google/gmail — even when both are syncing fine. Always source it so an ad-hoc check sees what the services see:

deploy/prod-ssh \
  "su - bbx-<box> -c 'set -a; . /etc/beebox/box.env; set +a; bbx health --box /var/lib/beebox/<box>/box'"

/home/beebox/.env is the hub's own file; the hub derives box.env from it at each start, so edit the former and restart the hub. (This produced a bogus "two boxes have missing connectors" health report on 2026-07-14 — the connectors were healthy; the bare invocation was the bug.)

Memory probe. Every bbx engine serve process records its memory in each served box's .beebox/memory/ (src/webapp/server/memory-probe.ts):

• samples.jsonl: one line a minute with RSS, used and total heap, the heap limit, external and arrayBuffers memory, and the count of live handles (all in MB). It rotates to samples.1.jsonl at 1 MiB, about three days.
• <time>-pid<pid>-<pct>.heapprofile: the V8 sampling heap profile, written the first time in a process the heap passes 50%, 70%, or 85% of its limit. The server log gets one [memory] heap passed … line naming the file. Open it in Chrome DevTools (Memory tab, Load) and read the bottom-up view: it shows the allocation stacks of the live JavaScript heap. The newest six profiles, at most 64 MiB, are kept.

After an OOM, read the last samples (how fast the heap climbed, and whether external or RSS outside the heap grew instead) and the newest profile. A box engine's crash output is in its unit's journal (journalctl -u beebox-box@<box>); a unit that will not start shows in systemctl status beebox-box@<box>. For a full heap snapshot at the moment of an OOM, start the process with NODE_OPTIONS=--heapsnapshot-near-heap-limit=1 and a --diagnostic-dir with free disk about twice the heap size; it is off by default for that cost.

Live memory inspection. kill -USR1 <node-pid> opens the inspector on 127.0.0.1:9229 without a restart. A heap snapshot forces a full GC and needs free disk about the size of the heap, so check df first. Never send SIGUSR2 to a node process not started with --heapsnapshot-signal: its default action terminates the process.

Administrative restart after manual config changes:

Normal code deployment uses the controller in deploy/deploy.sh; it already restarts and verifies the services. A direct systemd restart does not establish the shared admission/drain boundary. Use the following only after work is explicitly quiesced:

deploy/prod-ssh "systemctl restart beebox-hub beebox-scheduler"

hub.json doesn't hot-reload — adding, removing, or re-pointing a box entry needs a beebox-hub restart, not just a config edit. To add a box, run deploy/add-box.sh rather than editing hub.json: it does the clone, both manifest registrations, the restart, and a canary check (boxes). To register a box that is already on disk, bbx hub add-box <slug> <path> is the validated single step.

Writing scripts that run on the server

Ad-hoc shell as above is fine, but scripts (anything in this repo that shells out to the server programmatically) should go through the single chokepoint at feedback-review/run-on-server.ts. runOnServer({ script, asUser, sshTarget }) SSHes in as root (the entry point key auth is set up for), then immediately su - <asUser> before running the script, with the script piped via stdin so multi-line content and quotes work without escaping. Each box is private to its own user, so anything that reads or writes a box runs as asUser: boxUser(slug) (bbx-<slug>). asUser defaults to beebox, the hub, which can read hub.json and no box. Only set asUser: "root" for operations that genuinely require root (systemctl, chown, package installs) and leave a comment saying why.

Why this matters — historical bug: an earlier feedback-resolver script SSHed as root and ran git commit directly. That created objects under the box's .git/objects/<prefix>/ owned by root, and the next box-user commit that happened to hash into one of those prefixes failed with insufficient permission for adding an object to repository database. Same shape for the mv into config/feedback/resolved/ — that directory ended up root-owned too, blocking beebox writes. The failures were intermittent (hash-prefix-dependent) and never pointed back at the original culprit; they just silently blocked the wakeup agent until someone noticed. The chokepoint enforces the "never write as root inside beebox's home" invariant in one place so this doesn't drift back.

If you're writing a new dev tool that needs to talk to the server: either import runOnServer from feedback-review/, or — if it's awkward to depend on that path from where you are — promote the helper to a shared location (suggested: tools/run-on-server.ts at the monorepo root) and update both callers. Don't write a fresh ssh root@... 'command' line.

Diagnostic endpoints behind auth

In production, /api/trpc/debugLog.get and /api/trpc/health.check sit behind the Google OAuth cookie gate — curl without a browser cookie gets rejected.

Bypass for machine access: if BBX_DIAG_API_KEY is set in /home/beebox/.env, GET requests to those two endpoints (and chat.statusAll) are allowed with an Authorization: Bearer <key> header. The hub passes such a request through without identity, and the box checks the key again against its own copy (/etc/beebox/box.env). The debug log returns the tRPC envelope ({"result":{"data":{"entries":[…]}}}):

curl -H "Authorization: Bearer $BBX_DIAG_API_KEY" \
  https://box.example.com/<box>/api/trpc/debugLog.get | python3 -m json.tool

health.check answers from a cached snapshot by default (see health-checks.md). A machine caller that needs the box's state right now — post-deploy verification, "did that fix land?" — must ask for a live run:

curl -H "Authorization: Bearer $BBX_DIAG_API_KEY" \
  'https://box.example.com/<box>/api/trpc/health.check?input=%7B%22fresh%22%3Atrue%7D' | python3 -m json.tool

(The input is the URL-encoded JSON {"fresh":true}. The diag-key bypass keys on the procedure name only, so the query string doesn't affect it.)

On localhost/dev (no GOOGLE_OAUTH_CLIENT_ID set), auth is disabled entirely — curl works without the header.

For SSH-only debugging: ssh root@<server> tail /var/lib/beebox/<box>/box/.beebox/client-debug.log. See client-debug-log.md for the log file format.

Nightly Claude Code updates

Claude Code's built-in auto-updater only fires reliably during long-running interactive sessions — server-side short-lived invocations drift behind. A systemd timer runs claude update as root nightly around 04:00 UTC (with up to 30m jitter).

Units (installed by hetzner/provision.sh; they run as root, since the wrapper re-copies the update to /usr/local/bin/claude):

• claude-update.timer — schedule (OnCalendar=*-*-* 04:00:00, Persistent=true so missed runs catch up at boot).
• claude-update.service — oneshot that invokes the wrapper below.
• claude-update-failure.service — triggered via OnFailure= if the wrapper can't record its own failure (e.g. script missing / disk full); logs to syslog daemon.err.

Wrapper: deploy/claude-update.sh (served from the checked-out tree at /opt/beebox/beebox/deploy/claude-update.sh, so deploy.sh rsync picks up edits). It:

• Appends starting: / ok: or FAIL: lines with UTC timestamps to /var/log/beebox/claude-update.log.
• Also writes failures to syslog via logger -t claude-update -p daemon.err.
• Exits with claude's exit code so systemd marks the unit failed (triggering the belt-and-suspenders OnFailure unit).

Where to look for evidence:

# The wrapper's own log (every run appends a line)
ssh root@<server> tail -50 /var/log/beebox/claude-update.log

# Systemd view (last run, next run, last 20 journal lines)
ssh root@<server> 'systemctl list-timers claude-update.timer --no-pager; \
  systemctl status claude-update.service --no-pager; \
  journalctl -u claude-update.service -n 20 --no-pager'

# Any failure messages tagged by the wrapper or belt-and-suspenders unit
ssh root@<server> 'journalctl -t claude-update --since "-7 days" --no-pager'

Force an immediate run: ssh root@<server> systemctl start claude-update.service.

Periodic health check: see health-checks.md — a weekly remote routine reminds the user to run that runbook.

Git-annex health

The current annex model, checks, and repair commands are in assets.md. Periodic operational checks belong in health-checks.md. The production conversion and its one-time cutover procedure are retained only as a dated historical report; do not use that report as a current runbook or as current production evidence.