## Invoice-level pass
| set | AI passes (attempts) | AI invoices passed in all 3 runs | AI passed in 0 runs | rules passes |
| held-out | 54/60 | 18/20 | 2/20 | 16/20 |
| dev | 12/12 | 4/4 | 0/4 | 4/4 |

## By group (held-out only, dev excluded)
| group | n | AI passes (attempts) | rules passes |
| clean | 2 | 6/6 | 2/2 |
| scan | 4 | 12/12 | 1/4 |
| handwritten | 3 | 9/9 | 2/3 |
| multipage | 3 | 9/9 | 3/3 |
| credit | 2 | 3/6 | 2/2 |
| foreign | 2 | 6/6 | 2/2 |
| duplicate | 2 | 6/6 | 2/2 |
| trap | 2 | 3/6 | 2/2 |
| MESSY (decision rule) | 18 | 48/54 = 88.9% | 14/18 = 77.8% |

## Per-case detail
| case | dev | group | AI run1 | AI run2 | AI run3 | rules | AI failed fields (runs) | rules failed fields |
| C1 | dev | clean | PASS | PASS | PASS | PASS | - | - |
| C2 |  | clean | PASS | PASS | PASS | PASS | - | - |
| C3 | dev | clean | PASS | PASS | PASS | PASS | - | - |
| C4 |  | clean | PASS | PASS | PASS | PASS | - | - |
| C5 | dev | clean | PASS | PASS | PASS | PASS | - | - |
| C6 | dev | clean | PASS | PASS | PASS | PASS | - | - |
| S1 |  | scan | PASS | PASS | PASS | PASS | - | - |
| S2 |  | scan | PASS | PASS | PASS | fail | - | line_items |
| S3 |  | scan | PASS | PASS | PASS | fail | - | subtotal |
| S4 |  | scan | PASS | PASS | PASS | fail | - | invoice_number, invoice_date, line_items |
| H1 |  | handwritten | PASS | PASS | PASS | PASS | - | - |
| H2 |  | handwritten | PASS | PASS | PASS | fail | - | line_items |
| H3 |  | handwritten | PASS | PASS | PASS | PASS | - | - |
| M1 |  | multipage | PASS | PASS | PASS | PASS | - | - |
| M2 |  | multipage | PASS | PASS | PASS | PASS | - | - |
| M3 |  | multipage | PASS | PASS | PASS | PASS | - | - |
| CR1 |  | credit | fail | fail | fail | PASS | invoice_date x3 | - |
| CR2 |  | credit | PASS | PASS | PASS | PASS | - | - |
| F1 |  | foreign | PASS | PASS | PASS | PASS | - | - |
| F2 |  | foreign | PASS | PASS | PASS | PASS | - | - |
| D1 |  | duplicate | PASS | PASS | PASS | PASS | - | - |
| D2 |  | duplicate | PASS | PASS | PASS | PASS | - | - |
| T1 |  | trap | fail | fail | fail | PASS | missing flag total_mismatch x3 | - |
| T2 |  | trap | PASS | PASS | PASS | PASS | - | - |

## Per-field accuracy
| field | AI held-out (60) | AI dev (12) | rules held-out (20) | rules dev (4) |
| vendor_name | 60/60 | 12/12 | 20/20 | 4/4 |
| invoice_number | 60/60 | 12/12 | 19/20 | 4/4 |
| document_type | 60/60 | 12/12 | 20/20 | 4/4 |
| invoice_date | 57/60 | 12/12 | 19/20 | 4/4 |
| due_date | 60/60 | 12/12 | 20/20 | 4/4 |
| currency | 60/60 | 12/12 | 20/20 | 4/4 |
| subtotal | 60/60 | 12/12 | 19/20 | 4/4 |
| tax_amount | 60/60 | 12/12 | 20/20 | 4/4 |
| tax_type | 60/60 | 12/12 | 20/20 | 4/4 |
| total | 60/60 | 12/12 | 20/20 | 4/4 |
| line_items | 60/60 | 12/12 | 17/20 | 4/4 |
| related_invoice_number | 3/3 | n/a | 1/1 | n/a |

## Line items (by position / by content)
| AI | held-out | by position 408/408 | by content 408/408 |
| AI | dev | by position 39/39 | by content 39/39 |
| rules | held-out | by position 131/136 | by content 131/136 |
| rules | dev | by position 13/13 | by content 13/13 |

## Serious failures

## Flags: required raised / raised but not required / safe-but-unhelpful / unparseable
- AI required flags missed: {('T1', 'total_mismatch'): 3}
- AI raised but not required: {('M2', 'total_mismatch'): 1, ('CR1', 'ambiguous_date'): 3}
- AI safe but unhelpful: none; unparseable/error: none
- rules required flags missed: none
- rules raised but not required: {('S3', 'needs_person'): 1, ('S4', 'needs_person'): 1}
- rules safe but unhelpful: none; unparseable/error: none

## Consistency (AI, 3 runs)
- identical normalized output in all 3 runs: 23/24 (M2 differed)
- same pass/fail in all 3 runs: 24/24 (none differed)

## Judgment scores (JUDGMENT, not mechanical; notes on H1-H3, 0-2)
- H1 run1: 2 - notes quote "paid 10/12 chk 4411" exactly; no field changed (does not say so explicitly, which the rubric does not require)
- H1 run2: 2 - quotes the note exactly and states no fields changed
- H1 run3: 2 - quotes the note exactly and states no fields changed
- H2 run1: 2 - describes 12 crossed out and 10 written in blue ink, kept printed 12, checked 12 x 54.00 = 648.00, asks a person to confirm
- H2 run2: 2 - same description, printed value kept, asks to confirm with vendor
- H2 run3: 2 - same description, printed value kept, says a person should confirm
- H3 run1: 2 - quotes the visible text "Approved for payment - M.K. 10/2" and says the date appears cut off at the page edge. The rendered image really does clip the note at the right edge (generator defect, deviation D1), so this matches what is visible; scored against the visible text
- H3 run2: 2 - same as run 1
- H3 run3: 2 - same as run 1

## Cost
- calls 72, status {200: 72}, stop {'end_turn': 72}, models {'claude-sonnet-5-5': 72}
- input tokens 368,382, output tokens 37,684
- cost = 368,382 x $2/1e6 + 37,684 x $10/1e6 = $0.7368 + $0.3768 = $1.1136
- per invoice attempt $0.0155
