Testing Strategy

This project optimizes for functional correctness and protocol stability, not stress/performance testing.

tests/test.sh is the one entrypoint: CI, the monorepo driver, and the release gate all call it, so a mode cannot mean one thing locally and another in CI. It is written against the agent-first-data Bash kit, so afdata has to be on PATH (or named by AFDATA_BIN) to drive the harness — the container it drives needs none of that. Everything cargo touches runs inside that container; nothing in this suite is built on the host.

Default Gates

Run on every PR.

  1. Static checks

    • bash tests/test.sh static (runs ShellCheck at the pinned version on the host, then cargo fmt, cargo build, and cargo clippy inside the selected test container)

  2. Unit + component tests

    • bash tests/test.sh unit (runs lib/bin tests plus tests/check_regressions.sh inside the selected test container)
    • Focus: argument parsing, endpoint URL handling, fetch builder shape, CDP message framing, artifact path resolution, observation/network schema serialization, health/capabilities clients, SDK error mapping.
    • No browser process required.

  3. Regression list

    • Covered by bash tests/test.sh unit; it runs tests/check_regressions.sh inside the selected test container.
    • Every production bug fix adds or updates a regression entry.

Browser Integration Suite

Runs in CI when a browser binary is available, and locally on demand. Separated from the default gates because it requires Chromium installed and is slower.

What it covers

Observation artifact tests

Fixture pages cover:

The tests compare normalized JSON snapshots. Browser-version-specific geometry tolerance is allowed only for pixel-level bounding-box drift.

Network artifact tests

Fixture pages cover:

The network tests assert structure and linkage rather than exact event order when CDP does not guarantee ordering across resource types.

Health, capabilities, and profile tests

Unit tests cover JSON shapes without a browser. Integration tests cover:

Browser discovery

The suite respects, in order:

  1. AFHTTP_TEST_BROWSER_BIN environment variable (explicit path).
  2. which chromium, which chrome, which google-chrome-stable.
  3. Standard install paths per platform (/Applications/Google Chrome.app/..., /usr/bin/chromium, C:\Program Files\Google\Chrome\...).

When none is found, the suite is skipped, not failed, with a clear log line so the CI matrix can either provide a browser or accept the skip.

Running locally

Native arm64 test images do not need Rosetta. If Apple container reports that Rosetta is unavailable while creating its builder, set the user configuration at ~/.config/container/config.toml before running the fixed gate:

[build]
rosetta = false

# Default gates (Apple container on local macOS)
bash tests/test.sh

# Full integration suite (Apple container on local macOS)
bash tests/test.sh integration

# The integration mode runs real test files including:
# tests/browser_fetch.rs, tests/fetch_http_only.rs, tests/health_capabilities.rs,
# tests/cdp_proxy.rs, tests/cookie_jar_isolation.rs, tests/env_isolation.rs,
# tests/display_takeover.rs, tests/network_artifact.rs, and tests/tabs_management.rs

CI (.github/workflows/ci.yml)

Linux is covered by the integration-docker job, which runs the full integration suite through the Docker harness (real chromium + every backend + KasmVNC, AFHTTP_NO_SANDBOX=1 so the in-container sandbox is off). There is no ubuntu native leg: the ubuntu runner’s chromium is a confined snap that can’t complete download-to-disk tests.

The native integration matrix validates the binaries actually shipped, and runs with Chromium’s sandbox on (no AFHTTP_NO_SANDBOX) since these are normal desktops:

OSBrowserSource
macos-latestChromepreinstalled by GitHub-hosted runner (Homebrew binary)
windows-latestChromepreinstalled by GitHub-hosted runner (Scoop binary)

The matrix sets RUST_MIN_STACK=16 MiB so the deep fetch/host future chain doesn’t overflow Windows’ small default thread stack.

The fixed harness selects Apple container on local macOS and Docker in CI. AFHTTP_TEST_CONTAINER_RUNTIME=apple|docker is available for an explicit backend check; ordinary local runs need no override.

The flaky-by-design display takeover suite runs in its own non-blocking nightly workflow (.github/workflows/takeover-panel.yml), not in the gating ci.yml. The Lightpanda backend is exercised inside integration-docker when its binary is present.

Display Takeover Tests

Real-display takeover is exercised through the display_takeover.rs integration suite: it boots an afhttp host --takeover-provider kasmvnc, asserts the host brings up the KasmVNC display provider, serves /takeover/panel through the authenticated listener, and reports display_takeover: true in /capabilities. These run in the nightly takeover workflow because the provider startup is timing-sensitive and version-dependent.

tests/takeover-in-workbench.sh is the other half, and is deliberately not a gate. It takes a real takeover — Xvnc, a headful browser on its display, a real afhttp ui takeover --mode session — and puts it behind agent-first-ui’s afui session serve, then drives the result with a real Chromium over CDP. It reports what it observed (HTTP statuses, the first RFB frame off the WebSocket, what each browser document says about its own origin and storage) rather than only whether it passed, and it writes screenshots to target/. It needs a checkout of agent-first-ui beside this repository and refuses to run without one, which is why it stays out of tests/test.sh.

What is not tested