Design

This document records why the engine was chosen, the result model afpdf targets, the extraction rules the article contract implies, and how changes are regression-tested. The principles in the README take precedence; this document makes them concrete.

1. Engine choice

Corpus

This is a historical manual comparison used to choose the initial engine, not a benchmark run by the current test suite. The full external corpus and mutated files are not checked in; timings and failures below describe only that comparison and should not be generalized to current upstream releases.

The comparison used real documents from the first consumer: Chinese legislation and regulatory PDFs downloaded from government sites. The set covered:

long text with running headers and page numbers, including full-width page numbers such as — ５６ —;
ruled tables and whitespace-aligned tables;
formulas set in math fonts;
embedded figures, seals, and barcodes;
fully scanned documents and scanned pages with a thin text layer;
a document whose fonts are not embedded and have no ToUnicode map;
a truncated file;
99 mutated copies (truncation and random byte flips) for robustness.

Results

Concern	PDFium (baseline)	pdf_oxide 0.3	LiteParse 2.16 (PDFium fork)
Reading order	Correct line order	Default output moves line ends and centered headings to the page end	Plain text correct; Markdown merges headings into body and joins provisions
Paragraph joins	Mostly correct; headings glued to body; indented article markers missed	Inserts spaces inside Chinese words at joins	Inserts spaces inside Chinese words; no cross-page continuation
Page furniture	Removed in this set	Page numbers kept; header option quadratic and ineffective	Half-width numbers removed; full-width `— ５６ —` kept; Markdown only
Tables	Flattened to lines	2 of 8 tables found, columns lost; ruled 6-row table split into 14 rows	Ruled tables good, with cell boxes; unruled table body put in one cell; spaces inserted inside numbers
Images	Bitmaps and boxes available, not placed	Boxes available; all images emitted at page end	Placed in reading order; decorative and repeated images filtered
Hidden text	Extracts text clipped out of view	—	Drops text hidden by clipping paths
Page status	Whole-document only	Per page, correct on this set	Per page; “too little text” reason misfires on sparse legal pages
Unmapped fonts	Lost 22 characters (`第 123 号` became `123`)	No loss	Same loss as PDFium
Other text loss	—	—	Lost a whole line beside a tall math-font text box
Robustness	Malformed files fail fast with an error	No crash, but one mutated file ran past 60 s; empty text on a truncated file without error	One mutated file aborted the process (≈1e18-byte allocation); one hung past 60 s
Speed	56 pages in ≈56 ms	≈3× slower	4–155 ms per file

Decision

Use PDFium through pdfium-render. Implement the article layer here. The alternatives’ strengths become requirements (section 3), not dependencies. A second engine may be used only as an isolated cross-check, for example to recover characters PDFium cannot map, and never as a silent fallback.

2. Two result layers

Extraction has schema_version: 1, basic, and an optional article. The SDK returns typed Rust structures; the CLI frames them as AFDATA results. No engine-specific objects cross the SDK or worker boundary.

BasicDocument contains metadata, total page count, selected pages, optional encoded images, and extraction diagnostics (pages processed and failed, the page cap, clipped and unmapped glyph counts, image failures). Each page retains:

the engine’s original raw_text, before furniture removal or reflow;
glyph entries: native Unicode values (including UTF-16 surrogates), font name, font size in points, optional RGBA color, bounds and origin, generated flag, and clipping flag;
visual lines, geometry availability, image-object metadata and page statistics;
both clipped-filtered and unfiltered lines for repeatable organization;
page processing failures with stable codes.

The basic layer represents PDFium observations, not a complete PDF object dump. Native extraction cannot recover wording from missing font maps or scans. Unmapped counts and original glyph values make that limitation visible.

organize(&basic, &ProcessingOptions) performs no I/O. It derives page text-layer status, reading order, heading/paragraph/provision/list/table/figure/footnote blocks, source spans, and removed furniture, plus its own diagnostics (pages reflowed without geometry, repetition detection skipped). It carries nothing the basic layer already holds: a page’s engine text is raw_text on the basic page only, and extraction diagnostics are not copied. Article::text() flattens text blocks in reading order; it invents no OCR text or image placeholder language. Figures reference image IDs; caller protocols can define their own markers.

Image objects are enumerated at the page’s top level. Bitmaps drawn inside form XObjects are neither listed as image metadata nor extracted.

All coordinates use PDF points and a bottom-left origin. Glyphs and lines preserve style/geometry evidence; article blocks contain source regions, not an exact glyph-to-block alignment map. Cross-page paragraphs carry multiple page spans. Headers and footers remain separately inspectable.

3. Configurable organization

ExtractOptions.processing = None selects basic extraction. Otherwise the same pure organizer is invoked by the native SDK, CLI worker, and isolated SDK. Defaults enable these independent options:

Option	Effect when disabled
`remove_page_furniture`	Keep page numbers and repeated running headers/footers
`merge_paragraphs`	Keep visual line boundaries and disable cross-page joins
`merge_across_pages`	Keep paragraph fragments separate at page breaks
`drop_clipped_text`	Include text outside recognized object clipping bounds
`place_images`	Omit figure blocks from reading order; retain image metadata

expected_language defaults to auto. An explicit chinese hint enables the legacy Han-ratio heuristic for a broken text layer over a page-sized scan. English text is never rejected merely because it has few Han characters.

Lines use baselines and font size so tall math glyphs do not absorb neighbours. Paragraph joining uses indentation, text edges, spacing, and numbering. Chinese provisions remain boundaries, and CJK joins do not acquire spaces. Furniture uses repetition and margin position; short page selections have less evidence and diagnostics report when repetition detection is skipped. Tables use text geometry and alignment; exact table cells, equation semantics, caption association, and arbitrary-layout reconstruction are not guaranteed. Clipping recognition is conservative: top-level text objects with understood rectangular path bounds; nested form-local paths are not interpreted.

4. Native runtime and limits

PDFium is accessed through pdfium-render 0.9.4. The engine feature is optional; clients using only isolation do not load native PDFium in the parent process. The worker executable must be installed by the caller. The shared library is resolved from EngineOptions, which is process-level configuration rather than a per-document option: an explicit library_path, else the library in the install directory when present, else the platform’s dynamic-library search for pdfium. afpdf engine install fills the install directory (per-user data directory by default) with the pinned build 8035 from bblanchon/pdfium-binaries, verified against a checksum table compiled into the executable, and writes a manifest beside it; afpdf engine status reports the resolved library, whether it binds, and the platform’s limit support without changing anything. In-process extraction serializes complete calls and fixes the library after its first successful initialization, matching PDFium’s global binding lifetime.

IsolatedParser creates one afpdf worker process per PDF. Stdin contains a versioned typed request with PDF bytes encoded as base64. The worker responds with one AFDATA result on stdout or error on stderr. No persistent worker state is reused. Native aborts become worker_failed; malformed replies become worker_protocol; deadline and output bounds produce worker_timeout and worker_output_limit. A worker that rejects the call before reading its document (a mismatched executable) is reported by its own stderr event, not by the parent’s failed write. Raw native stderr is never copied into a caller error.

Defaults: 500 pages, 120 MiB input (also the hard maximum), 40 million pixels per image/render, 300000 ms wall-clock time, 256 MiB worker result, 64 KiB worker stderr, and a 2 GiB address space on Linux (zero, meaning no limit, elsewhere). The worker request frame is capped at 192 MiB. The page cap is explicit in diagnostics rather than silently pretending the full PDF was processed. Image extraction failures record IDs and codes; failed pages retain status. Input and output bounds also protect parent-side buffering. Base64 and glyph records add overhead; callers should lower limits to suit service capacity.

On Linux, the worker uses RLIMIT_AS before opening PDFium. A native allocation failure can abort rather than produce a graceful Rust error; the parent still reports a failed document. A nonzero memory limit on another platform fails every worker explicitly; the default is zero there, so the limit is never silently skipped and never silently invented. In-process SDK calls have no hard timeout or native crash containment. Processes provide containment, not a security sandbox. Timeouts and failed exchanges kill and reap the child. Async cancellation kills it via Tokio’s kill_on_drop process handling. Services retain their own bounded queues and concurrency controls. One worker handles one document.

5. Consumer boundary

The extraction tool owns PDF decoding, geometry, native image encoding/rendering, repeatable organization, and the versioned worker protocol. A service owns acquisition, authentication, transport/protobuf, encryption, caching, quotas, queue scheduling, storage, indexing, OCR/model orchestration, and domain rules. Legacy text/image placeholders belong in the consumer’s compatibility adapter. The initial service migrates with a version-only Cargo dependency plus its own local Cargo patch until this crate is published. Its deployment stages afpdf next to the PDFium shared library and names that library in EngineOptions.

6. Verification

Use scripts/test.sh for formatting, Clippy, pure layout/CLI tests, generated reference consistency, and native checks. scripts/test.sh engine runs the four small government PDF fixtures in a container against a system-installed PDFium. Automatic runtime selection tries Apple container on macOS, Podman, then Docker. The suite exercises the CLI worker and runs afpdf engine install into a temporary directory to check the download, checksum, manifest, and precedence over the system library. The Windows CI job runs the same install test through a real pdfium.dll. Fixtures document their sources in tests/fixtures/pdf/README.md.

The initial consumer migration saves the previous parser’s output before switching, then compares four fixtures across readable/raw/marker/embedded-image requests. Text, furniture, page status, image geometry and encoded-image hashes are compared. These fixtures constrain migration behavior; they are not proof of accuracy for arbitrary papers, languages or scanned PDFs.

Isolation tests exercise aborts, hangs, output floods, malformed replies, and recovery on a subsequent request. Broader stored corpora and mutated PDFs should remain release gates when available; do not claim that this small checked-in fixture suite covers an unavailable external corpus.

7. CLI configuration and AFDATA boundaries

afpdf extract INPUT returns document facts, organized article structure, and text. The configuration’s output.basic can include native evidence independently. There is no basic/readable/both mode, nor one flag for each processing rule. --set KEY=VALUE applies typed, invocation-only overrides. A new settings field needs its typed model/default/validation, not a new CLI argument or dispatcher.

Precedence: built-in defaults < JSON file < repeated invocation overrides. .afpdf/config.json in the current directory is the default location; an explicit --config PATH selects another JSON file. No ancestor search, global file merge, or SDK auto-loading is implied. Ordinary reads and extraction do not create files. Explicitly selected missing files fail extraction.

config show/get inspect defaults-filled settings. config set saves a sparse override, while config unset removes one. Both use AFDATA Document and DocumentFile: dot-path traversal, type-directed coercion, source-preserving staging, complete typed/domain validation, and atomic commit. A rejected edit leaves the file unchanged; a rejected first edit creates no file or directory. Config reads are capped at 1 MiB. Mutations inherit AFDATA’s refusal to overwrite unsafe link targets. Concurrent edits use the document layer’s last-writer-wins save semantics; atomic installation does not imply multi-writer merging.

extract settings reuse ExtractOptions directly; execution holds the engine location and isolation settings; output selects basic evidence, article blocks, and text independently. The CLI result always carries the document facts (metadata, total_pages, diagnostics) so an incomplete extraction is visible whatever content was selected. article and text are on by default; basic adds the per-page evidence (raw_text, glyphs, lines, image metadata) and nothing the top level already states. Requested image payloads appear once in the images array, independently of output.basic. The SDK’s full BasicDocument and internal worker protocol keep their typed image payloads unchanged. Pure basic output can disable organization with extract.processing=null after selecting basic-only output. See configuration.md for runnable configuration examples.

The CLI registry declares legal command shapes and AFDATA owns argument parsing, help, event construction, redaction and output routing. The internal worker always uses JSON and split result/error routing, and receives resolved settings. Native library diagnostics do not become public error prose. Coordinates carry PDF units (_pt, _px), byte limits/durations use _bytes/_ms, and binary payloads use base64 strings. PDF dates remain native PDF date strings.

Protocol and naming checks run against real CLI events and serialized extraction results. Bash orchestration forwards the AFDATA child marker into the container so the outer command owns a single terminal result.