Testing Strategy

This project optimizes for functional correctness and protocol stability, not stress/performance testing.

tests/test.sh is the fixed local/container entrypoint used by the monorepo driver and release gate. Native CI legs run the corresponding Cargo checks directly. It is written against the agent-first-data Bash kit, so afdata has to be on PATH (or named by AFDATA_BIN) to drive the harness — the container it drives needs none of that. All Cargo work driven by this entrypoint runs inside the container.

Default Gates

Run on every PR.

  1. Static checks

    • bash tests/test.sh static (runs ShellCheck at the pinned version on the host, then cargo fmt, cargo build, and cargo clippy inside the selected test container)

  2. Unit + component tests

    • bash tests/test.sh unit (runs lib/bin tests plus tests/check_regressions.sh inside the selected test container)
    • Focus: argument parsing, endpoint URL handling, fetch builder shape, CDP message framing, artifact path resolution, observation/network schema serialization, health/capabilities clients, SDK error mapping.
    • No browser process required.
    • Deterministic broadcast fixtures verify recovery after event loss, capture limits, idle-based network quiet after loss, and collector cancellation on drop. A local CDP responder counts body requests to verify known oversized responses are skipped before reading or decoding their body.

  3. Regression list

    • Covered by bash tests/test.sh unit; it runs tests/check_regressions.sh inside the selected test container.
    • Every production bug fix adds or updates a regression entry.

Browser Integration Suite

Runs in CI when a browser binary is available, and locally on demand. Separated from the default gates because it requires Chromium installed and is slower.

What it covers

Observation artifact tests

Fixture pages cover:

The tests compare normalized JSON snapshots. Browser-version-specific geometry tolerance is allowed only for pixel-level bounding-box drift.

Network artifact tests

Fixture pages cover:

The network tests assert structure and linkage rather than exact event order when CDP does not guarantee ordering across resource types.

Health, capabilities, and profile tests

Unit tests cover JSON shapes without a browser. Integration tests cover:

Browser discovery

The suite resolves Chromium in this order:

  1. AFHTTP_TEST_BROWSER_BIN (explicit path). An explicit path is authoritative: if it does not name a file, the browser counts as missing and discovery does not fall back to the standard paths.
  2. Without that variable, the standard install paths (/usr/bin/chromium, /usr/bin/chromium-browser, /usr/bin/google-chrome, /usr/bin/google-chrome-stable, /Applications/Google Chrome.app/...).

The other backends follow the same rule through their own variables (AFHTTP_TEST_LIGHTPANDA_BIN, AFHTTP_TEST_FINGERPRINT_CHROMIUM_BIN, AFHTTP_TEST_FOXBRIDGE_BIN, AFHTTP_TEST_CAMOUFOX_BIN).

What a missing backend means is decided by AFHTTP_TEST_REQUIRE_BACKENDS, a comma-separated list of backend names (chromium, lightpanda, fingerprint-chromium, camoufox):

The test container sets the list to every backend its image installs (tests/container-entrypoint.sh): chromium and lightpanda always, fingerprint-chromium on x86_64, camoufox when the image was built with AFHTTP_TEST_INSTALL_CAMOUFOX=1. The native CI legs require chromium. tests/check_backend_requirements.sh, part of the unit mode, proves both halves against real test binaries: a required backend pointed at a nonexistent path fails, an optional one skips.

Running locally

Native arm64 test images do not need Rosetta. If Apple container reports that Rosetta is unavailable while creating its builder, set the user configuration at ~/.config/container/config.toml before running the fixed gate:

[build]
rosetta = false

# Default gates (Apple container on local macOS)
bash tests/test.sh

# Full integration suite (Apple container on local macOS)
bash tests/test.sh integration

# Only the browser download-to-disk tests (browser_navigation_download* in
# tests/browser_fetch.rs), for iterating on download handling
bash tests/test.sh download

# The integration mode runs real test files including:
# tests/browser_fetch.rs, tests/fetch_http_only.rs, tests/health_capabilities.rs,
# tests/cdp_proxy.rs, tests/cookie_jar_isolation.rs, tests/env_isolation.rs,
# tests/display_takeover.rs, tests/network_artifact.rs, and tests/tabs_management.rs

CI (.github/workflows/ci.yml)

Linux is covered by the integration-docker job, which runs the full integration suite through the Docker harness (real Chromium, available optional backends, and KasmVNC, AFHTTP_NO_SANDBOX=1 so the in-container sandbox is off). There is no ubuntu native leg: the ubuntu runner’s chromium is a confined snap that can’t complete download-to-disk tests.

Camoufox/foxbridge installation is opt-in with AFHTTP_TEST_INSTALL_CAMOUFOX=1. Fingerprint Chromium is unavailable on arm64. Those two are the only backends left out of AFHTTP_TEST_REQUIRE_BACKENDS in the container, so they are the only ones whose tests may return early; a passing gate does not demonstrate that they were exercised unless the image carries them.

The native integration matrix validates the binaries actually shipped, and runs with Chromium’s sandbox on (no AFHTTP_NO_SANDBOX) since these are normal desktops:

OSBrowserSource
macos-latestChromepreinstalled by GitHub-hosted runner (Homebrew binary)
windows-latestChromepreinstalled by GitHub-hosted runner (Scoop binary)

The matrix sets RUST_MIN_STACK=16 MiB so the deep fetch/host future chain doesn’t overflow Windows’ small default thread stack.

The fixed harness selects Apple container on local macOS and Docker in CI. AFHTTP_TEST_CONTAINER_RUNTIME=apple|docker is available for an explicit backend check; ordinary local runs need no override.

The flaky-by-design display takeover suite runs in its own non-blocking nightly workflow (.github/workflows/takeover-panel.yml), not in the gating ci.yml. The Lightpanda backend is exercised inside integration-docker when its binary is present.

Display Takeover Tests

Real-display takeover is exercised through the display_takeover.rs integration suite: it boots an afhttp host --takeover-provider kasmvnc, asserts the host brings up the KasmVNC display provider, serves /takeover/panel through the authenticated listener, and reports display_takeover: true in /capabilities. These run in the nightly takeover workflow because the provider startup is timing-sensitive and version-dependent.

tests/takeover-in-workbench.sh is the other half, and is deliberately not a gate. It takes a real takeover — Xvnc, a headful browser on its display, a real afhttp ui takeover --mode session — and puts it behind agent-first-ui’s afui session serve, then drives the result with a real Chromium over CDP. It reports what it observed (HTTP statuses, the first RFB frame off the WebSocket, what each browser document says about its own origin and storage) rather than only whether it passed, and it writes screenshots to target/. It needs a checkout of agent-first-ui beside this repository and refuses to run without one, which is why it stays out of tests/test.sh.

What is not tested