The tests were green and the feature was dead

Visura’s job is a negative claim. Not here are the differences — every tool since the 1990s does that — but nothing else moved, and here is the count. That claim is the entire product, and it has a property most claims don’t: being confidently wrong about it is worse than not making it. A tool that shows you a false difference wastes a minute. A tool that says a clause is unchanged when it isn’t makes you skip reading the clause that moved.

So the interesting question was never whether the feature existed. It was whether it was true.

The number I had not looked at

The claim rested on chunk reuse: split the document into sections, hash each one, and anything whose hash matches the previous version is carried forward untouched. This had been in the codebase for months. It had a schema, metrics columns, a dashboard readout. What it did not have was a measurement.

I wrote one. Take a 90-clause agreement, insert a single sentence into clause 8, and ask how much of the new version is carried forward from the old. The answer should be “almost all of it.”

That number is a confession. Reuse depended on where the edit landed, which means it was not really working anywhere — it just failed invisibly when the edit happened to be near the end. The cause was one line: chunk boundaries were decided by a running character count. Every boundary therefore depended on everything before it, so inserting a sentence at the front shifted every subsequent boundary by a few characters and changed every subsequent hash. Nothing matched. The feature the whole product rested on worked properly only for edits near the end of a file.

The fix is an old idea: let the content decide. A boundary opens when a hash of that unit’s own text passes a gate, not when a counter fills up; a size ceiling only cuts a chunk that no gate has closed. Because the size accumulator resets at every boundary, an edit can only disturb the chunk it lands in and any that follow before the next boundary the content decides, and alignment re-syncs from there.

One detail is worth keeping, because it is the one I got wrong twice. There is deliberately no minimum chunk size. A minimum feels obviously sensible — nobody wants a 40-character chunk — but “close the boundary if the chunk is already long enough” puts accumulated length back into the decision, and a perturbation carries into the chunks after it before it re-syncs. Dropping the minimum is what bought the perfect re-sync in that last column.

Then I made it hard to be wrong

Knowing how bad the first measurement was, I went at the tests properly. The danger is not a missed difference; it is a fabricated one. Two exports of the same contract from two different word processors differ in line wrapping, hyphenation, quote glyphs, ligatures, running headers and page numbers, without a single word having changed. If any of that surfaces as a change, the coverage number is noise. That is why the text is normalised before anything is hashed. The normaliser also drops any line that reads like a page number, including a line of nothing but digits, so a change inside such a line can go unreported.

So: a golden corpus. One agreement in 13 export variants — the file unchanged, curly quotes and em dashes, ligatures, different column widths, hyphenation at a line end, hyphenation landing on a page break, different running headers and page numbers, non-breaking spaces, doubled blank lines, CRLF, and finally several of them at once. Every variant must report zero changes, in both directions. Then tests that a real edit is still found buried in the noisiest variant, that a renumbered list is cosmetic, that net 30 → net 45 is not.

Every test was green. I believed the feature worked, and I had receipts.

The PDFs

The last thing left was end-to-end validation with real documents, which I had been treating as a formality. I generated two PDFs with a real PDF writer: the same agreement, once at A4 and once at US Letter. Different page size means different line breaks, different pagination, running headers in different places — and not one word different. Then I drove the actual page in a real browser and dropped them in.

SAME contract, A4 vs Letter — reported as changed, ranked material.

That is the precise failure the product exists to prevent, printed by the product, on the easiest case there is.

The cause took a while to find because it was nowhere near the code I had been testing. The PDF extractor built each page’s text by taking the text items pdf.js returns and joining them with a space. Not a newline — a space. Every PDF therefore arrived downstream as a single unbroken run of characters with no line structure and no page boundaries in it.

And the pass that removes running headers and page numbers works on lines. With no lines, it matched nothing and removed nothing. It had been dead code on every real PDF since the day it was written. The A4 file and the Letter file each carried Acme Corp / Confidential at the top of both pages, at different character offsets, and the comparison correctly reported that some text had moved.

The tests passed because every fixture in them was a string I had typed, and I had typed newlines in it. The furniture pass worked beautifully on my hand-written documents. It had simply never once been given a document the program itself had produced.

A test that supplies its own input can never catch the input being wrong.

That is the whole lesson and it generalises past this bug. A fixture encodes an assumption about the shape of the data, and if that assumption is the thing that is broken, the test agrees with the bug. Every one of mine agreed with it and told me I was fine.

What fixing it uncovered

The repair was to keep the structure: rebuild lines from the flag pdf.js already sets at the end of each rendered line, separate pages with a form feed, and make the furniture pass page-aware — a short line that reads like a label, not a sentence, is a running header if its digit-folded template appears at the same page edge on two or more pages. That catches Acme Corp / Confidential 2, which is never byte-identical twice. I also lost time to a detail I would not have guessed: trim() counts a form feed as whitespace, so the page markers were being quietly erased one step before the pass that needed them.

With repagination no longer drowning everything, two more real bugs surfaced immediately — both of them the under-reporting kind, which is the kind that matters.

Two edits in adjacent passages were being merged into a single card. That merge exists for a good reason: an edit that crosses a passage boundary should not be reported three times. But two unrelated edits that happen to land next to each other are two changes, and a reviewer counting cards was being told there was one. The split now happens at long runs of untouched text, and a single edit crossing a boundary still arrives as one card.

And thirty days → ninety days was ranked merely substantive, because the materiality classifier looked for digits. A notice period tripling is material whether the contract writes it as 30 or as thirty. It had been hidden until now because the money edit sitting next to it had been dragging the merged card up to material.

Where it ended up

Three pairs, through the production build, in a real browser:

A new set of tests holds the fix down, and every fixture in them carries the page markers the extractor actually emits. Several of them assert the pass does not fire: a repeated complete sentence survives, a heading that appears at the edge of only one page survives, a short phrase that appears twice mid-body survives. Those are the tests that matter, because deleting real text is how you manufacture a false “unchanged.”

The engine that produces the claim still contains no model of any kind. It is hashing, alignment and a diff — deterministic and checkable by hand. That was a deliberate choice and this episode is the argument for it: when the answer came out wrong, there was something to measure and a specific line to blame. You cannot bisect a vibe.