Agent-First Terminal Runtime Plan

Status: PTY sessions, structured screen snapshots, multiplexed runtime events, actor-aware input with TTL leases and human preemption, secret input mode, and the feature-gated OpenAPI server, authoritative DOM-rendered local CLI UI, and UI attachment to an already-running server are implemented. Transcript persistence is deferred: no consumer in this runtime or its callers reads one yet, and it would be the only part of the system that writes terminal content to disk. Revive it when a reader exists that can state a retention period and a read scope.

Summary

agent-first-terminal should be a generic local terminal runtime, not an agent-specific automation layer or application UI data model.

It turns a PTY-backed process into an embeddable, observable, controllable, and replayable UI-producing surface. The stock DOM renderer, other renderers, test harnesses, and higher-level automation controllers all consume the same runtime.

Current Implementation

The runtime currently owns these pieces:

Design Principle

Keep terminal runtime semantics separate from task semantics.

The runtime should know about sessions, screens, input actors, events, and transcripts. It should not know whether a process is Codex, Claude, opencode, psql, k9s, an installer, or a deployment script. It should not decide that a user task is complete.

Higher-level controllers can implement agent-specific task logic on top of the runtime when they need it.

Goals

Non-Goals

Primary Use Cases

Architecture

agent-first-terminal
  PTY process/session ownership
  raw output fan-out
  VT/ANSI screen tracker
  event stream
  input actor coordination
  secret input mode
  (deferred) transcript and replay policy

afterminal ui
  isolated local app window
  multi-session DOM rendering from authoritative styled snapshots
  human input and process controls
  short-lived UI capability
  new-runtime or existing-API attachment

agent/controller layer
  optional task ids
  prompt markers
  agent-specific waiting logic
  high-level automation policy

Proposed Crate Shape

.
├── src/lib.rs
├── src/runtime.rs
├── src/session.rs
├── src/screen.rs
├── src/events.rs
├── src/input.rs
├── src/lease.rs
├── src/server.rs
└── docs/terminal-runtime-plan.md

server.rs should be optional or feature-gated. The core runtime should be a Rust API that embedded hosts can use directly.

Core Concepts

Session

A session owns one PTY-backed process plus host-local state.

pub struct TerminalOpenSpec {
    pub program: String,
    pub args: Vec<String>,
    pub cwd: Option<PathBuf>,
    pub env: BTreeMap<String, String>,
    pub rows: u16,
    pub cols: u16,
    pub title: Option<String>,
}

The runtime may provide a shell helper, but the generic primitive should be program + args. The CLI UI defaults to the user’s shell when it opens an initial session without an explicit program.

Screen

The screen is a host-maintained virtual terminal snapshot derived from raw PTY bytes.

pub struct ScreenSnapshot {
    pub seq: u64,
    pub rows: u16,
    pub cols: u16,
    pub title: Option<String>,
    pub cursor: CursorState,
    pub lines: Vec<ScreenLine>,
    pub activity: ActivityState,
}

The screen model is for observation and rendering. It is not a transcript and should not be treated as the only audit source.

Events

Events are the stable observation stream for hosts and controllers.

pub enum TerminalEvent {
    SessionOpened { session_id: String },
    ScreenChanged { session_id: String, seq: u64 },
    OutputChunk { session_id: String },
    InputAccepted { session_id: String, actor: ActorId },
    InputRejected { session_id: String, actor: ActorId, reason: String },
    InputPreempted { session_id: String, previous: ActorId, by: ActorId },
    InputLeaseAcquired { session_id: String, lease_id: String, actor: ActorId },
    InputLeaseReleased { session_id: String, lease_id: String, reason: String },
    SecretInputStarted { session_id: String, reason: String },
    SecretInputEnded { session_id: String },
    ProcessExited { session_id: String, code: Option<i32> },
}

OutputChunk should not imply that every subscriber can read raw bytes. Raw stream access is a separate capability.

Input Actors

Every input source should identify itself as an actor:

The runtime records actor metadata for accepted input. It should not record secret bytes.

Input Leases

Input leases coordinate concurrent actors without allowing request bytes to interleave. Non-human actors must hold a lease:

{
  "actor": "controller",
  "mode": "exclusive",
  "ttl_ms": 2000,
  "preemptible_by": ["human"]
}

Default rule: human input preempts non-human exclusive leases immediately. Multiple shared holders may submit complete chunks; manager serialization defines their order. An exclusive lease reserves non-human input for its holder.

Transcript (deferred)

Transcript policy would be explicit per session (off / redacted / raw), with redacted the default for anything persisted and raw requiring explicit host authorization. Nothing implements this: there is no reader. When one exists, a transcript must write nothing at all during a secret input window — which the current implementation already satisfies, since that window does not even reach the replay ring.

Secret Input Mode

Secret input mode is a runtime state a session is put into while a person types something that must not be observed — a password, a recovery phrase, a key. While it is on:

Any actor may start a window — raising the shield is always the safe direction, so a best-effort prompt detector should be able to. Only a human actor may end one: an agent that could end it could end it and then read the screen.

Ending waits for the session to go quiet (SECRET_INPUT_SETTLE_MS), because resuming publication would otherwise release the echo of what was just typed that the reader thread has not drained yet. Until then, ending is refused with secret_input_settling and the caller retries; the failure direction is “stays private”.

What this does not cover: output the program leaves on screen after the window closes is visible again, exactly as it is to anyone looking at the window. The guarantee is that nothing derived from the session’s bytes leaves the runtime while the window is open.

Host API

The machine-facing HTTP endpoints are terminal runtime endpoints, not UI transport. Browser state and actions use AFUI’s retained state and typed calls; there is no terminal-owned WebSocket or SSE protocol:

GET    /health
GET    /openapi.json
GET    /schemas/index.json
GET    /schemas/<schema_file>
POST   /v1/sessions
GET    /v1/sessions
GET    /v1/sessions/<id>
GET    /v1/sessions/<id>/screen
POST   /v1/sessions/<id>/input
POST   /v1/sessions/<id>/resize
POST   /v1/sessions/<id>/signal
GET    /v1/sessions/<id>/secret-input
POST   /v1/sessions/<id>/secret-input/actions
GET    /v1/sessions/<id>/leases
POST   /v1/sessions/<id>/leases
DELETE /v1/sessions/<id>/leases/<lease_id>
DELETE /v1/sessions/<id>
GET    /v1/events

Domain responses use the AFDATA envelope (kind: result with a trace, or kind: error with code/message/retryable/hint), and every response is passed through AFDATA redaction at the serialization boundary. The discovery documents are served bare under their own media types. The event stream’s items are described by the operation’s itemSchema rather than wrapped in a finite result envelope, and Last-Event-ID resumes after a given global sequence while it remains in the bounded backlog.

Transcript and a public raw-stream endpoint are not part of this surface; raw bytes remain a host-library interface and never become the browser protocol.

Suggested access levels:

These levels are a design intent, not an implemented check: the server has one bearer credential today. api serve --mode local binds loopback and is the default; --mode lan binds all interfaces for a trusted IPv4 network and publishes this machine’s LAN address in the ready event.

Relationship To The Local UI

afterminal ui is a thin trusted renderer over the same TerminalSessionManager exposed by the OpenAPI server. It can either own a new local server or attach to an existing one with --api-url URL [--session-id SESSION_ID] [--mode window|link|session]:

Promise: the page can rebuild everything it displays from its own URL alone. The server-side screen is the only VT state authority; the browser keeps no parsed terminal state of its own between loads. On first load and reconnect, AFUI delivers the newest complete retained state before declaring the runtime live. There is no browser-owned sequence, SSE gap repair, or private screen refetch protocol. Losing the tab, losing the process, or just hitting reload all land on the same screen through the same URL, with no raw-byte replay and no client-held history required.

The browser UI and capability-management routes are private implementation routes and are not part of the OpenAPI contract. The machine API remains the integration surface for external controllers.

Relationship To Agent Automation

Agent automation is an upper layer.

For example, a Codex/Claude/opencode controller can:

None of those task states belong in agent-first-terminal.

Security Model

Terminal runtime access is sensitive. It can observe command output and may observe secrets if configured poorly.

Required defaults:

Prior Art To Learn From

The intended runtime is smaller than the platform products and more embeddable than the sharing tools.

Migration Plan

Phase 0: Planning Skeleton

Phase 1: Extract Existing PTY Runtime

Phase 2: Add Screen Tracking

Phase 3: Add Runtime Events

The globally multiplexed screen/output/input/lease/resize/signal/process lifecycle stream is done. Title events remain coupled to later title work.

Phase 4: Add Input Actors And Leases

Done. Non-human API actors require a shared or exclusive lease, multiple shared actors can submit atomic chunks, and human input preempts a non-human exclusive lease.

Phase 5: Add Secret Input Mode

Done. Transcript policy and storage hooks are deferred with it — see the status note at the top of this document.

Phase 6: Add A Local Human UI

Done. The CLI supports AFUI Window, Link, and Session delivery, one or many terminal sessions, attachment midway to a separate running server, snapshot replacement through the typed AFUI runtime, and resize, interrupt, terminate, kill, and close controls.

Phase 7: Optional Standalone Host

The initial standalone host is done for the current session/screen/actor-input/lease/resize/signal/close/event surface. It exports and serves the same OpenAPI 3.2 document.

Open Questions

Acceptance Criteria For MVP