Composer

The chat input bar: its coordination machine and a map of its rendered states.

What it is

The composer is the chat input surface: the composerMachine coordination overlay that backs it, and a map of its rendered UI across state combinations (what each button shows, how you reach the state, and where you can go from it). The unbuilt child-actor design is separate in plans/composer-child-actors.md.

The composer is the bottom button bar (+ inline textarea on desktop, + a drop-up textarea row on mobile) plus two header chips (the narration badge and the mute button) that change with the same state. Screenshots are tight viewport crops: desktop at 1100×380 (header + bar), mobile at 390-wide (bar / typing row).

How it works

Ownership

src/frontend/src/machines/composerMachine.ts is a parallel XState machine with three regions:

• voice: idle | speaking | pausedForSpeech
• hq: idle | inFlight
• keyboard: closed | open.{unlocked,locked}

The machine is a coordination overlay. It does not own microphone or playback device lifecycles. realtimeTranscriptionMachine owns microphone states and speechPlaybackMachine owns playback. InteractiveChat/voice.ts mirrors device facts into the composer and executes the device commands it emits.

Voice coordination

recording and transcriptNonEmpty are mirrored context flags. A queued speech segment follows these rules:

When ordinary playback finishes, the machine returns to idle and restarts the mic if turnTaking is set. When playback finishes or is stopped in pausedForSpeech, it returns to idle and releases the hold. Starting dictation while speech plays stops speech first.

A held mic records silence in place of what it hears. The segment stays open, so its text and its one recording continue, and the HQ pass covers the words from before the speech. A manual Replay (SPEECH_EXTERNAL) holds the mic even when the transcript has text; queued speech never plays over text.

The rendered voicePaused flag is composerSnapshot.matches({ voice: "pausedForSpeech" }). Recording appearance and isTranscribing still come from the transcription machine.

HQ coordination

The hq region tracks zero or more pending HQ transcription requests. The first START_HQ enters inFlight; later requests append to pendingHq. HQ_STATUS updates one row, HQ_SEND_LIVE emits the fallback send command, and HQ_DONE removes one row. The region returns to idle after the last pending request completes.

HQ request work stays in InteractiveChat/voice-keyword-send.ts; the machine owns visible coordination state and emitted commands.

Every voice send with a recording requests the HQ pass (docs/implemented-plans/hq-always.md); there is no setting. The box decides availability: with no usable HQ key the job fails at once and the live text is sent, marked <speech stt="live">. A box pass costs about $0.006 per dictated minute on whisper or whisper-llm and $0.003 on whisper-llm-mini (OpenAI list prices).

When live text goes down mid-recording, the live transcript shows […] where words are missing (docs/implemented-plans/live-gap-marker.md). A reconnect that replays the whole outage removes it once new live text arrives; a longer outage keeps it. If the live text is what gets sent, the marker goes to the agent as <unsure>[…]</unsure>. Spoken commands never match across it.

Mobile keyboard coordination

The machine defines this region:

keyboard (initial: closed)
  closed: OPEN_KEYBOARD → open.unlocked
  open:
    CLOSE_KEYBOARD → closed
    unlocked: TOGGLE_LOCK → locked; MESSAGE_SENT → closed
    locked: TOGGLE_LOCK → unlocked; MESSAGE_SENT stays open

The current React view still owns typingMode and typingLocked and renders the mobile typing row when typingMode || isTranscribing. The keyboard region therefore records the intended transitions and has test coverage, but the view has not yet been wired to read it.

Tests

src/frontend/test/machines/composerMachine.composer-machine.test.ts covers voice, HQ, and keyboard transitions. The React wiring lives in src/frontend/src/components/chat/everywhere/InteractiveChat/voice.ts, and the remaining component-owned keyboard state lives in InteractiveChat/shell.tsx and InteractiveChat-layout/view/ChatComposerSection.tsx.

States

State dimensions

The composer's appearance is a function of these, most read straight off the machines that own them:

The two state-dependent controls

Mic / voice toggle — a strict priority cascade:

Stop buttons (can show together, left of the mic): speechPlaying → "Stop speaking" (speaker-x — audio semantics, rhymes with the header mute icon); isStreaming → "Stop agent" (stop-in-circle). Deliberately different glyphs: they used to be identical, so when one showed alone you couldn't tell which action you were about to take.

Named states, desktop

Idle, empty

The resting state. + (Add menu), the textarea placeholder "Type or paste an image…", a disabled Send (faint up-arrow), and the plain mic ("Voice input"). Header: speaker (unmuted), no narration badge.

→ Typing (type text) · Recording (tap mic) · toggle narration / muted from the header/debug menu.

Typing (Send enabled)

Any non-empty input flips Send to the solid accent up-arrow; the mic is unchanged (you can still dictate instead). Enter sends; Shift+Enter / Ctrl+J newline.

→ Idle (clear / send) · send fires MESSAGE_SENT (ends any voice turn).

Narration mode

Header gains the 🎙️ narration × badge; the mic swaps to the mic-with-speech-bubble icon ("Voice input (narration mode)"). Narration means silent/structured replies. (Every voice send gets the HQ pass, narration or not.)

→ while an HQ pass is in flight after a voice send, the voice chip gains a · transcribing… sub-label (composerMachine.hq === inFlight) — capture pending, see below.

Muted

The header speaker becomes the muted icon (speaker-×, pressed). Queued TTS is never played (markPlayed), so the "Stop speaking" button and the speaking/paused states can't occur while muted.

Named states, mobile

Idle, empty

Below sm, the textarea is hidden; the bar is icon-only: +, a spacer, the keyboard button, and the mic. (Stop-speaking / Stop-agent appear between the spacer and keyboard when active.)

→ Keyboard (tap ⌨ → typing row) · Recording (tap mic).

Keyboard open (typing)

The keyboard button opens a drop-up textarea row with its own Send, plus a lock and close affordance above it (the button bar is hidden). Unlocked, the row closes after send; locked (typingLocked), it stays open.

Locked, the lock button fills in (primary background) so the state reads at a glance — the open-vs-closed padlock outline alone was illegible at that size:

→ lock keeps it open across sends · close returns to the icon bar.

Voice states, desktop

These are the states the machine work was really about. They normally need a live mic / TTS, so the shots below are from the dev gallery (/dev/composer-states, dev-only) which renders the real presentational composer with fabricated props. Behaviour is documented from the code.

Recording (isTranscribing)

Mic button becomes a red stop square ("Stop recording"). The textarea goes read-only showing the live transcript, with Cancel (✕) / Edit (✎) / Send controls. All mic status lives in the MicOverlay floating above the voice button: live volume bars (flat = no audio reaching the recorder, e.g. wrong mic selected) over a rotating voice-keyword hint, filtered to actions valid for the current text — only "mic off" until words arrive. In reconnecting (network or mic recovery in flight) the bars turn amber and a "recovering…" chip replaces the hint: moving amber bars = network blip (mic still hears you), flat amber bars = the mic itself is gone.

Before any words arrive the placeholder reads "Listening…" and Send is disabled — there's nothing to commit yet (pressing it used to silently cancel the dictation):

→ Cancel/Esc (STOP_DICTATION) · keyword send (KEYWORD_SEND → commit → keep recording) · speech queued while recording → pausedForSpeech.

Paused for speech (voicePaused)

Mic button is a pulsing mic with a pause badge ("Resume recording (stops speech)") — the mic is the action (tap to get it back), the badge is the state — and the Stop speaking speaker-x shows. Reached when the agent speaks while you were recording — the mic is held (it records silence) and TTS plays. The transcript and its Cancel/Edit/Send controls stay.

→ playback ends → release the hold (resumeMic) · Resume tap / Stop speaking → release the hold. This state replaced the old voicePaused boolean; it should not flicker between the agent's sentences.

Speaking, not paused (speaking)

Stop speaking shows; the mic stays the plain/narration mic (it wasn't recording). Reached when the agent speaks and you weren't mid-utterance (a replay-while-idle, or a spoken reply to a typed message).

→ playback ends → idle (or reopen mic if a voice turn is active) · Stop speaking → idle.

Agent streaming (isStreaming)

A Stop agent danger-circle (independent of the mic), shown while the agent replies:

It overlays the voice states too — here alongside Stop-speaking during mid-stream speech (speaker-x + stop circle, one per concern):

HQ in flight (hq === inFlight)

The voice chip gains a transcribing… sub-label while the HQ round-trip runs after a voice send; the mic has usually already reopened.

Voice states, mobile

The mobile bar is icon-only; transcription shows the drop-up row beneath it, and the Stop-speaking / Stop-agent circles slot in between the spacer and the keyboard button.

Recording — red stop-square mic + the drop-up transcript row:

Paused for speech — Stop-speaking speaker-x + pulsing mic with pause badge:

Speaking — Stop-speaking speaker-x + plain mic:

Realizable combinations

R = recording, Sp = speaking, Pa = pausedForSpeech, St = streaming, N = narration, M = muted, T = typed text present.

× narration (badge + mic-bubble icon) and × muted (mute icon; forces TTS off) overlay the rows above, except where excluded below.

Impossible combinations

• recording + speaking — speech that starts with the mic open holds the mic and enters pausedForSpeech, and entering recording from speaking stops playback first. The mic only records alongside speech while held, and a held mic records silence.
• muted + (speaking | pausedForSpeech) — muted makes every queued segment markPlayed (never played), so no TTS state is entered, and muting in-flight speech sends STOP_SPEECH.
• pausedForSpeech + not playing — pausedForSpeech is only held while playback runs; it resumes the mic the instant playback ends. (The old voicePaused && speechPlaying AND existed to paper over a brief window where this could desync — gone now.)
• voice speaking/pausedForSpeech without speechPlaying — same: those states exist only to coordinate active playback.
• typed-input-enabled Send while recording — during transcription the textarea is read-only and shows the transcript, not editable input; Send for typed text is replaced by the Cancel/Edit/Send transcription controls.
• mobile inline textarea while idle — below sm the textarea only mounts in typingMode or while transcribing; otherwise the bar is icon-only.

Regenerating the screenshots

The idle / typing / narration / muted shots come from a live chat session; the voice states come from the dev gallery at /dev/composer-states (dev-only, omitted from production builds — see router.tsx and pages/dev/components/ComposerStatesHarness.tsx). The gallery renders the real presentational composer with fabricated props, so it stays prop-typed against the components and breaks the typecheck if their contracts drift.

The gallery doesn't hand-pick states — it declares the axes (button bar: primary mode × narration × muted × streaming; mobile keyboard: lock × input; plus the recording-before-words card), takes the cross-product, filters the impossible combos (listing them with reasons), and renders all 37 realizable ones. Each card is keyed by its active axes (idle, idle-streaming, recording-narration-muted, recording-empty, typing-keyboard-locked, …); isolate one with ?state=<key> for a clean capture. To re-shoot, point bin/browse at /dev/composer-states?state=<key> and screenshot. The named-state shots above are the readable subset; the gallery is the exhaustive grid.