## Invoice-level pass
| set | AI passes (attempts) | AI invoices passed in all 3 runs | AI passed in 0 runs | rules passes |
| held-out | 54/60 | 18/20 | 2/20 | 16/20 |
| dev | 12/12 | 4/4 | 0/4 | 4/4 |

## By group (held-out only, dev excluded)
| group | n | AI passes (attempts) | rules passes |
| clean | 2 | 6/6 | 2/2 |
| scan | 4 | 12/12 | 1/4 |
| handwritten | 3 | 9/9 | 2/3 |
| multipage | 3 | 9/9 | 3/3 |
| credit | 2 | 3/6 | 2/2 |
| foreign | 2 | 6/6 | 2/2 |
| duplicate | 2 | 6/6 | 2/2 |
| trap | 2 | 3/6 | 2/2 |
| MESSY (decision rule) | 18 | 48/54 = 88.9% | 14/18 = 77.8% |

## Per-case detail
| case | dev | group | AI run1 | AI run2 | AI run3 | rules | AI failed fields (runs) | rules failed fields |
| C1 | dev | clean | PASS | PASS | PASS | PASS | - | - |
| C2 |  | clean | PASS | PASS | PASS | PASS | - | - |
| C3 | dev | clean | PASS | PASS | PASS | PASS | - | - |
| C4 |  | clean | PASS | PASS | PASS | PASS | - | - |
| C5 | dev | clean | PASS | PASS | PASS | PASS | - | - |
| C6 | dev | clean | PASS | PASS | PASS | PASS | - | - |
| S1 |  | scan | PASS | PASS | PASS | PASS | - | - |
| S2 |  | scan | PASS | PASS | PASS | fail | - | line_items |
| S3 |  | scan | PASS | PASS | PASS | fail | - | subtotal |
| S4 |  | scan | PASS | PASS | PASS | fail | - | invoice_number, invoice_date, line_items |
| H1 |  | handwritten | PASS | PASS | PASS | PASS | - | - |
| H2 |  | handwritten | PASS | PASS | PASS | fail | - | line_items |
| H3 |  | handwritten | PASS | PASS | PASS | PASS | - | - |
| M1 |  | multipage | PASS | PASS | PASS | PASS | - | - |
| M2 |  | multipage | PASS | PASS | PASS | PASS | - | - |
| M3 |  | multipage | PASS | PASS | PASS | PASS | - | - |
| CR1 |  | credit | fail | fail | fail | PASS | invoice_date x3 | - |
| CR2 |  | credit | PASS | PASS | PASS | PASS | - | - |
| F1 |  | foreign | PASS | PASS | PASS | PASS | - | - |
| F2 |  | foreign | PASS | PASS | PASS | PASS | - | - |
| D1 |  | duplicate | PASS | PASS | PASS | PASS | - | - |
| D2 |  | duplicate | PASS | PASS | PASS | PASS | - | - |
| T1 |  | trap | fail | fail | fail | PASS | missing flag total_mismatch x1 | - |
| T2 |  | trap | PASS | PASS | PASS | PASS | - | - |

## Per-field accuracy
| field | AI held-out (60) | AI dev (12) | rules held-out (20) | rules dev (4) |
| vendor_name | 60/60 | 12/12 | 20/20 | 4/4 |
| invoice_number | 60/60 | 12/12 | 19/20 | 4/4 |
| document_type | 60/60 | 12/12 | 20/20 | 4/4 |
| invoice_date | 57/60 | 12/12 | 19/20 | 4/4 |
| due_date | 60/60 | 12/12 | 20/20 | 4/4 |
| currency | 60/60 | 12/12 | 20/20 | 4/4 |
| subtotal | 60/60 | 12/12 | 19/20 | 4/4 |
| tax_amount | 60/60 | 12/12 | 20/20 | 4/4 |
| tax_type | 60/60 | 12/12 | 20/20 | 4/4 |
| total | 60/60 | 12/12 | 20/20 | 4/4 |
| line_items | 60/60 | 12/12 | 17/20 | 4/4 |
| related_invoice_number | 3/3 | n/a | 1/1 | n/a |

## Line items (by position / by content)
| AI | held-out | by position 408/408 | by content 408/408 |
| AI | dev | by position 39/39 | by content 39/39 |
| rules | held-out | by position 131/136 | by content 131/136 |
| rules | dev | by position 13/13 | by content 13/13 |

## Serious failures

## Flags: required raised / raised but not required / safe-but-unhelpful / unparseable
- AI required flags missed: {('T1', 'total_mismatch'): 3}
- AI raised but not required: {('M2', 'total_mismatch'): 1, ('CR1', 'ambiguous_date'): 3}
- AI safe but unhelpful: none; unparseable/error: none
- rules required flags missed: none
- rules raised but not required: {('S3', 'needs_person'): 1, ('S4', 'needs_person'): 1}
- rules safe but unhelpful: none; unparseable/error: none

## Consistency (AI, 3 runs)
- identical normalized output in all 3 runs: 23/24 (M2 differed)
- same pass/fail in all 3 runs: 24/24 (none differed)

## Cost
- calls 72, status {200: 72}, stop {'end_turn': 72}, models {'claude-sonnet-5-5': 72}
- input tokens 368,382, output tokens 37,684
- cost = 368,382 x $2/1e6 + 37,684 x $10/1e6 = $0.7368 + $0.3768 = $1.1136
- per invoice attempt $0.0155
