Agent-First Data v0.35.0: When Four Answers Agree

This crate ships the same rules four times: in Rust, Go, Python, and TypeScript. A shared set of fixtures — input on one side, expected output on the other — runs through all four, and the thing it checks is agreement. If the four SDKs render a value differently, one of them is wrong, and the fixture says so.

Most of this release is making them agree on questions nobody had noticed they answered differently. One of those questions is a secret-redaction rule, and there the first agreement reached was the wrong one. That is the part worth reading.

Four answers to “is this whitespace?”

Three places in the formatter ask whether a character is whitespace: trimming a _url value before redacting it, deciding whether a Plain value needs quotes, and deciding whether a YAML key would read back as a number. Each SDK answered with its own language’s built-in predicate, and the four predicates are not the same predicate.

Python’s str.isspace counts the ASCII separator controls U+001C–U+001F. JavaScript’s \s and trim() count the byte-order mark. Rust’s char::is_whitespace counts neither. So the same _url value trimmed differently depending on which SDK rendered it, and the same Plain record quoted differently.

YAML keys were the same story with numbers. A key that looks like a number has to be quoted, or a reader turns "123": x into the integer 123. Each SDK asked its own number parser, and the parsers disagree at the edges:

Python’s float accepts underscores, JavaScript’s Number accepts hex, and Number("1e9999") is Infinity, which fails the finite check. None of that is YAML. Each SDK now uses one explicit grammar — a decimal with an optional exponent, plus the inf/nan spellings — and the whitespace set is an explicit list written into the spec rather than whatever the host language happens to mean by the word.

The agreement that leaked

A _url field gets surgical redaction: the URL keeps its shape and only the secret parts go. When a value cannot be parsed as a clean URL, there is a fallback, and the fallback fails closed in two cases — the value contains whitespace, or it contains an @. Otherwise it passes through as it is, on the theory that something like /cb?page=2 is not a URL with a secret in it.

Making the whitespace set explicit meant choosing, for the trimming step, what each SDK would stop counting. Python stopped stripping U+001C, and TypeScript stopped stripping the BOM. Both then behaved like Rust and Go — and Rust and Go had never handled either:

$ cat callback.json
{"callback_url":"https://h/cb?code_secret=hunter2"}
$ afdata render callback.json        # 0.34.0; the leading U+FEFF is invisible
{"callback_url":"https://h/cb?code_secret=hunter2"}

A BOM in front of a URL cannot be trimmed as whitespace, so the URL does not parse. The value has no internal whitespace and no @, so the fallback lets it through, secret included. The fixture added in the same change recorded this as expected output. All four SDKs passed it, and the suite was green.

This is the trap in testing for agreement. The suite can tell that four answers differ. It cannot tell which one is right, and when you make four implementations agree, the cheapest move is toward the one that does the least. Here that meant the one that leaked.

The leak was also wider than the BOM. A zero-width space, a control character, a label (链接https://…), or an angle bracket in front of a URL all did the same thing, because the fallback never asked the obvious question: does this value contain a URL? It does now. A value that is not a clean URL but contains :// is redacted whole, in all four SDKs:

$ afdata render callback.json
{"callback_url":"***"}

The two fixtures that recorded the leak now expect ***. New ones cover the zero-width space, the label, and the bracket, and one checks that /cb?page=2 still passes through untouched — so the new rule cannot quietly turn into “redact anything odd”.

Bytes you did not ask to change

Two document formats rewrote parts of a file that the edit never touched, but only when the file used Windows line endings.

The dotenv editor worked out where the key started on a line by subtracting two lengths. One of them included the line’s \r and the other did not. On a CRLF file the result was one character off, and that character got written twice:

$ printf 'KEY=old\r\nexport OTHER=x\r\n' > .env
$ afdata set .env KEY new && afdata set .env OTHER y    # 0.34.0
$ cat -v .env
KKEY=new^M
export OOTHER=y^M

The command reported success, and the key it had just written was now spelled differently. A key added at the end of the file also got a bare \n, so the file ended up with mixed line endings.

TOML went wrong differently. The editor re-renders the document through toml_edit, which writes \n, so a single set on a CRLF file converted every line in it to LF. That was the whole file, not just the line that was edited. Both formats now keep the line ending the file already uses. TOML needed one more step: a string value containing a newline used to be written as a multi-line literal, with its newlines as real line breaks in the file. A line-ending pass cannot tell those from the document’s own line breaks, and would have turned a \n inside the value into \r\n. Such strings are now written as single-line escaped strings, so the value read back is the value that was set.

A budget you can state

The CLI’s document commands now refuse files over 16 MiB. Before this they read any file whole, so a mistyped path pointing at a log file or a disk image went straight into memory. A refusal with no way around it would just replace one problem with another, though, so the budget is a flag:

$ afdata get big.json
{"error":{"code":"document_too_large",
  "hint":"raise the limit with --max-file-bytes BYTES, or read the document from stdin with -",
  "message":"`big.json` exceeds the 16777216-byte read limit", ...}}
$ afdata set big.json app.port 8080 --value-type number --max-file-bytes 67108864

Every command that reads a document file accepts the flag: get, value, values, paths, keys, set, unset, add, remove, and lint. That includes lint’s JSON path, which reads text rather than a document and had no cap at all. Offering the flag on a command where it applies to some inputs but not others would be worse than not offering it. Stdin is not capped: the caller already chose to send that stream, and there is no path that could have been mistyped.

Smaller promises

render ignored --output-to and always wrote its output to stdout, so --output-to stderr printed the value to stdout. It follows the flag now, like every other command.

A reader that closes the pipe early — afdata value f k | head -c0, or a consumer that has read all it needs — is the normal way a pipeline shuts down, not an error. afdata value exited 1 there. It exits 0 now, and the Python and TypeScript emitters treat EPIPE the same way. Python also points the closed descriptor at /dev/null, so interpreter shutdown does not try to flush it again and print a traceback.

What this release was about

A cross-language fixture suite can only check that the four SDKs agree. It cannot check that the answer they agree on is the right one, and while you are fixing disagreements, the easiest change to make is the one that makes each SDK do less.

So the general lesson is about which way a fix moves. When an alignment change switches one implementation from catching something to letting it through, that is a change in behavior, and it needs a reason that holds up on its own, not just “the others already do it”. Here the right move was the opposite: find the case all four had let through, and make all four catch it.