accuracy harness

Did the AI get the clauses right?

Contract-review models are confident when they're wrong. They miss liability caps. They hallucinate indemnity clauses that aren't in the document. This grades a model's clause extractions against contracts lawyers have already labeled (CUAD), and checks every claim against the source contract. Two questions per extraction: was it right, and was it actually in the document?

Claude Sonnet 4.6 and GPT-5 are each graded on the same 100 CUAD contracts. Every headline metric is reported with a 95% percentile-bootstrap CI over per-contract tallies, so the point estimate isn't quoted as a population fact. CUAD's own labels are audited in parallel: where a model extracts text verbatim from a contract but CUAD marks the category absent, Sonnet and Opus rule independently on whether it's a real label gap; the both-judges-yes subset is credited back to recall. The console regrades client-side on every threshold change, model switch, or pasted run.

// live gradebrowser regrades on every change

Each extraction is scored on two axes. Overlap — token-level F1 against the CUAD-labeled span it best matches. Grounding — trigram coverage in the source contract; below threshold counts as a hallucination. Both run in JS over precomputed spans, so threshold changes re-grade all 100 contracts client-side in ~50 ms.

clause types

contracts

correct-match threshold0.70

F1 at or above this → Correct

partial-match floor0.40

below this F1 → no credit

grounding threshold0.80

trigram coverage below this → hallucination

// grader self-test — known errors injected into a synthetic run; the grader has to flag each one back

// dataset — what was sampled and how

// scorecardone row per clause type · click a column to sort · click a row to inspect

// auditing CUADflag → independent judges → measured impact on the score

CUAD's labels are treated as ground truth elsewhere on the page; here they're treated as suspect. When a model extracts text that's verbatim in the contract but CUAD marks the clause category absent, Sonnet and Opus each rule on whether it's a real clause the labels missed. Candidates both judges confirm are credited to the model's recall. Follows Northcutt et al., Pervasive Label Errors (NeurIPS 2021).

// explorermodel extractions next to CUAD spans, graded per-span · click any quote to locate it in the source contract

// playgroundthe grader's two primitives, recomputed as you type

axis 1 · token-overlap F1 // edit either input — recomputes live

extraction CUAD label span

F1 = 2·p·r / (p + r) · p = |A∩B|/|A| · r = |A∩B|/|B|

axis 2 · trigram source-grounding // edit the claim — recomputes live

claim text source contract

grounding = |claim₃ ∩ source₃| / |claim₃|   (word trigrams)

// internals

// sampler — grader/load_cuad.py

greedy coverage — add the contract advancing the most under-target clause types until each clears n ≥ 30.
length spread — fill remaining slots evenly across the contract-length distribution.
seeded — RNG(seed) breaks ties; identical seed → byte-identical manifest.

// reproduce

# CUAD: 510 contracts, 41 categories
curl -L -o data/cuad/CUAD_v1.zip …/CUAD_v1.zip
unzip -j data/cuad/CUAD_v1.zip 'CUAD_v1/CUAD_v1.json' -d data/cuad/
python3 grader/load_cuad.py --n 100 --min-per-type 30 --seed 42
python3 grader/run_openrouter.py --model openai/gpt-5 --out runs/gpt5_run.json
python3 grader/export_site_data.py --run sonnet:runs/sonnet_run.json --run gpt5:runs/gpt5_run.json

// methods — the non-trivial parts of the pipeline, with refs

Optimal one-to-one assignment. Greedy (sort candidate pairs by score, take the best one whose endpoints are free, repeat) is suboptimal: a locally-best pair can foreclose a globally better total. The grader uses the Hungarian algorithm (Jonker–Volgenant O(n³) variant) to find the maximum-weight one-to-one matching between extractions and CUAD spans. The matcher-toggle in the live grade reports the delta against greedy on the current scope. — Kuhn (1955); Munkres (1957); Jonker & Volgenant (1987). Pure-Python in grader/matching.py; ported to JS for in-browser re-grading.
Bootstrap confidence intervals. Headline precision / recall / F1 are reported with 95% percentile-bootstrap intervals over per-contract tallies. Resampling is at the contract level: outcomes within a contract are correlated, so resampling extractions would understate the standard error. B = 1000, seeded. — Efron (1979).
Inter-judge agreement (κ). Audit candidates are judged independently by Sonnet and Opus through OpenRouter. Cohen's κ corrects raw agreement for the chance term that dominates when one verdict class is over-represented; bands per Landis–Koch. The both-judges-yes subset is reported as the conservative recall correction; the Sonnet-only credited number is shown alongside it for context. — Cohen (1960); Landis & Koch (1977).
Benchmark auditing. Flag → independent validate → measure-impact, applied to CUAD's category-absent labels. Where a model extracts text verbatim from the contract but CUAD marks the category absent, the candidate is sent to two LLM judges; confirmed cases are credited back to the model's recall. — Northcutt, Athalye, Mueller, Pervasive Label Errors in Test Sets (NeurIPS 2021).