Agent-First PDF

Agent-First PDF (afpdf) is a small extraction tool and Rust library for article-like documents: research papers, legislation, judgments, and reports. Its purpose is to turn a page-oriented layout into content that a caller can read, convert to long text, or pass to its own indexing pipeline.

The SDK and CLI expose two layers: basic extraction preserves native text, glyph styles, geometry, and image references; organization derives readable article blocks without overwriting that evidence. PDFium performs decoding. The default CLI and the service SDK isolate native parsing in a child process.

Ask your agent: “Read this regulation PDF and give me its articles as clean text, with the page each one came from.”

Install the CLI

# prebuilt binary
brew install agentfirstkit/tap/afpdf   # macOS / Linux
scoop bucket add agentfirstkit https://github.com/agentfirstkit/scoop-bucket && scoop install afpdf   # Windows

# or from crates.io
cargo install agent-first-pdf --locked

# then fetch the pinned PDFium build this release was tested against
afpdf engine install

Prebuilt archives are also available from GitHub Releases. Depending on the crate as a library wants default-features = false plus the features you need; see Rust SDK.

Agent Skill

Use skills/agent-first-pdf/SKILL.md to teach a coding agent when to extract with afpdf, how to read its results and diagnostics, and when a document needs OCR or a vision model instead.

Design principles

1. Focus on articles

Optimize for documents whose main content is an ordered article with headings, paragraphs, and supporting figures. Research papers and legal documents are the primary use cases. General-purpose PDF editing, arbitrary format conversion, and faithful reconstruction of every visual layout are outside the scope.

2. Preserve basic extraction; make organization configurable

Basic extraction records the engine’s native text, glyph Unicode values, font names, font sizes, RGBA colors, bounds, and image metadata. “Basic” means the engine’s observations, not a lossless representation of every PDF operator. Clipped and unmapped glyphs remain available for inspection.

Organization consumes this serializable evidence. Callers can disable furniture removal, paragraph merging, cross-page joining, clipping removal, or image placement, and rerun organization without reopening the PDF. Both layers retain source page numbers and PDF-point coordinates. Heuristic results are interpretations; removed furniture remains available separately.

3. Recover structure before flattening text

Identify reading order, headings, paragraphs, and their source regions before producing continuous text. A two-column page must follow the article’s reading order. Chapter, section, and legal article boundaries must remain distinct.

The structured article is the source for text exports. The current SDK exposes continuous text through Article::text(). Callers can build other renderers from the same blocks; a Markdown renderer is not currently provided.

4. One natural paragraph has no layout-induced line breaks

Join lines wrapped by page layout within the same paragraph. Join a paragraph that continues across a page boundary when the available evidence supports it. Keep real paragraph boundaries, lists, headings, and separate legal provisions.

A sentence ending is not necessarily a paragraph ending. Punctuation alone cannot determine paragraph boundaries; use geometry and typography when available. Chinese text must not acquire artificial spaces when lines join.

Preserve the source wording. Reflow may normalize layout-induced whitespace; it must not summarize, paraphrase, or silently correct the document’s text.

5. Keep page furniture out of the article body

Exclude page numbers and repeated running headers and footers from body text. Use position and repetition as evidence. Preserve substantive headings, citations, footnotes, and numbered legal clauses even when they resemble page furniture. A request for a subset of pages may provide too little evidence to classify repetition confidently.

6. Preserve source locations and supporting content

Retain page numbers and source regions for extracted content, including paragraphs that span pages. Cleaning text must not erase the caller’s ability to check it against the source.

Keep supporting images in reading order where geometry supports it. Embedded bitmap extraction and full-page rendering serve different purposes: a figure may combine text, vector graphics, and bitmaps. Region rendering is not currently exposed. Keep recognized tables and footnotes distinguishable from prose. Table cells, caption associations, and formula semantics are not guaranteed; recognition can improve within the article extraction scope.

7. Reuse mature engines

Delegate PDF decoding, text geometry, bitmap extraction, and rendering to an established engine. Reuse upstream layout processing where it meets the article contract. Own the article-oriented behavior, clear results, and integration boundary; avoid maintaining another low-level PDF parser.

Choose the initial engine using representative article PDFs and regression cases. Keep engine-specific types behind the library boundary. Start with one engine rather than a framework for interchangeable backends.

The initial engine is PDFium through pdfium-render. A historical manual comparison used the same Chinese legislation and regulatory PDFs with pdf_oxide (Rust) and LiteParse (Rust over its own PDFium fork). Their layout tradeoffs and malformed-input failures did not justify changing the engine. Their strengths became requirements for this project: image placement in reading order, dropping text hidden by clipping, per-page text-layer classification, and recovering characters from fonts without a usable Unicode map. Evidence and rules are in docs/design.md.

8. One implementation, a Rust SDK and an agent-facing CLI

The library owns extraction and article results. It must be usable with PDF bytes and in-memory results, including image bytes, without requiring a CLI process or intermediate files. A service can embed that same library and own its authentication, transport, storage, and scheduling.

The CLI owns argument handling, file input, configuration, AFDATA output, and fetching the pinned PDFium build for the machine it runs on. Encoded images are returned in the result rather than written as separate files. The SDK and CLI must share extraction behavior. Library-only builds must not pull in CLI administration or output dependencies.

PDF engines are native code, and malformed or hostile PDFs can make them abort the process or never return. The SDK therefore offers two execution modes with the same request and result types:

• in-process: an explicit option for trusted input and tests;
• isolated: the SDK runs the afpdf executable as a child process per document, enforces a wall-clock timeout, bounded output, and page limit, with an address-space limit on Linux, and decodes its result. An abort, hang, or limit hit becomes a typed extraction failure for that document instead of taking the caller down.

A service that ingests PDFs from the open web must use the isolated mode. There is no long-running afpdf host: unlike a browser, PDF extraction keeps no session state, so a process per document is the isolation boundary.

9. Make incomplete extraction explicit

Report processed pages, requested limits, and extraction failures. Distinguish a blank page, a page without a usable text layer, and a page that failed to process. A successful command must not conceal missing text or images.

Start with text-bearing PDFs. A model or external service must not be silently required. For scanned pages, report the missing text layer and make rendering available so the caller can choose OCR or a vision model. Any future OCR must be explicit and distinguishable from native text extraction.

10. Leave indexing and interpretation to callers

Return content and provenance that another program can index. Corpus management, search, embeddings, image descriptions, summarization, and question answering belong to downstream applications. The tool does not need a database, background service, or knowledge-base lifecycle.

Service integration

afpdf is the PDF counterpart of afhttp inside a fetching service. The service keeps what is service-specific: authentication, transport (for example its gRPC PdfService), caching, quotas, and storage. It hands PDF bytes to the afpdf SDK in isolated mode and maps the article result onto its own protocol. The afpdf executable ships in the service image next to the PDFium library, the same way browser hosts ship for afhttp.

Responsibilities stay split as in principles 9 and 10:

Migration from an existing in-process parser runs both on a stored corpus of the service’s own PDFs and diffs every output. Each class of change must be explained before the service switches; see the regression section of docs/design.md.

Interface

afpdf engine install
afpdf engine status
afpdf extract article.pdf
afpdf extract article.pdf --set output.basic=true
afpdf extract article.pdf --set extract.image_mode=embedded
afpdf config show
afpdf --docs

engine install downloads the pinned PDFium build for this platform into a per-user directory, verifies its checksum, and probes it. engine status reports which library extraction would load (an explicit execution.engine.library_path, the installed library, or the system search), whether it loads, and whether this platform enforces the worker memory limit. Nothing else fetches PDFium implicitly.

Extraction returns document facts (metadata, page count, diagnostics), organized article blocks, and continuous text by default. Set output.basic=true to also return native evidence: per-page raw text, glyph styles, geometry, and image metadata. Processing options and returned content are independent configuration fields; there is no --mode. extract.image_mode=embedded|render_pages|both requests encoded image bytes in the result’s images array (base64 in JSON), independently of output.basic. Image bytes appear once. No OCR or external model is required.

Configuration uses AFDATA dot paths and validated atomic persistence. config show/get inspect effective settings, config set/unset explicitly save overrides, and --set KEY=VALUE lasts for one invocation. The default file is .afpdf/config.json in the current directory. See configuration for precedence, examples, output selection, and platform limits; CLI reference lists every command.

Rust SDK

The SDK exposes extract_in_process, organize, and IsolatedParser::parse.

# #[cfg(feature = "isolation")]
# mod isolated_example {
use agent_first_pdf::{EngineOptions, ExtractOptions, IsolatedParser, IsolationLimits};
# async fn example() -> Result<(), Box<dyn std::error::Error>> {
let parser = IsolatedParser::new("afpdf", EngineOptions::default(), IsolationLimits::default())?;
let bytes = std::fs::read("article.pdf")?;
let result = parser.parse(&bytes, &ExtractOptions::default()).await?;
// The default options already organized the article; the evidence it was
// derived from is kept alongside, so organization can be rerun with other
// options without reopening the PDF.
let text = result.article.as_ref().map(|article| article.text());
let lines = agent_first_pdf::organize(
    &result.basic,
    &agent_first_pdf::ProcessingOptions { merge_paragraphs: false, ..Default::default() },
);
# let _ = (text, lines);
# Ok(())
# }
# }

The engine feature enables native extraction. isolation enables the process client without linking PDFium. cli enables the executable and its runtime features. With default-features = false, only typed models and pure organization are built. The executable must be available to isolated SDK callers.

EngineOptions names the PDFium library and is process-level: a worker binds it before reading its document, and an in-process engine binds one library for the life of the process. Every path resolves it the same way: an explicit path, then the directory afpdf engine install fills, then the system search.

Hard wall-clock enforcement requires isolation. Workers enforce the address-space limit on Linux only, so IsolationLimits::default() sets it to 2 GiB on Linux and zero elsewhere; a nonzero limit on another platform fails every worker explicitly instead of running unlimited. A child process contains native crashes; it is not a filesystem or network sandbox. Service concurrency and queue limits remain the caller’s job.

Development

Run ./scripts/test.sh for static, unit, CLI, and native container checks. See CONTRIBUTING.md for prerequisites, focused checks, and fixture requirements. The design notes record engine choices and extraction limits; the bundled skill guides agent usage and development.

Licensed under MIT.