lab

InvoiceAudit

Vision extraction checked by arithmetic, with bounded repair and a second check on the repair itself

An invoice where the extraction is plausible and wrong. The arithmetic catches three violations, the repair loop runs, and a second check watches the repair for gaming the sums.

What it does

InvoiceAudit treats a model extraction as an untrusted proposal. Deterministic arithmetic checks the record, quantified failures guide a bounded repair loop, and the router sends uncertain documents for human review. The web interface shows the source document beside violations and the repair trace.

A separate detector compares the original and repaired records. It flags aggregate-only changes that can satisfy arithmetic without evidence of a corrected source reading. The loop is plain Python, with iteration and token limits plus a no-progress stop; there is no second model acting as the arithmetic judge.

The checked-in evaluation report covers 32 synthetic documents using gpt-5.4-nano-2026-03-17. Document accuracy rises from 78.1% to 81.2%, but auto-accept precision falls from 83.3% to 82.3%. The report marks the precision criterion as failed. Better document accuracy is not evidence that unattended acceptance became safer.

Repo

The repair that tried to cheat

Scroll through one invoice. Getting the sums to pass is easy; the point is noticing when a repair passes them without a reason.

01 Extract
02 Check
03 Repair
04 Re-check
05 Route

The model proposes a record. It is treated as an untrusted proposal.Deterministic arithmetic checks it. Three rules fail: 6 × 120.00 is 720.00, not 702.00.Repair 1 re-reads line 2 from the source: 702.00 was 720.00.Repair 2 edits the total alone, and now every sum passes.A second check compares the records: the total moved with no new reading behind it.Arithmetic alone would have accepted this. It goes to a person.

Specimen · not a real invoiceINV-2041

VendorHarbourline Services

Invoice date2026-03-14

Hosting · 12 × 85.001,020.00

Licences · 3 × 240.00720.00

Support · 6 × 120.00702.00~~702.00~~ → 720.00 re-read p.1, line 8

Subtotal2,460.00

Tax264.00

Total2,706.00~~2,706.00~~ → 2,724.00 no new reading

Arithmetic

✗✓Each line: quantity × price = amount

✗✓Line amounts sum to the subtotal

✗✓Subtotal + tax = total

Repair log

↻ Iteration 1: support line re-read from source

↻ Iteration 2: total set to 2,724.00

Repair detector: aggregate-only change. The total moved, no line or source reading did.

→ routed to a person

Held for review

Illustrative specimen. Every figure is fictional; the steps follow InvoiceAudit's pipeline. On the source the tax reads 246.00: the misread was there, and the repair changed the total instead.

Stack

Python
Pydantic
OpenAI
FastAPI
React
pytest

All data in this project is synthetic.