Our experiments start with a specific question: can a model copy invoice fields, follow customer-contact rules, or do something a simpler script cannot? The published report explains the inputs, scoring, failures and limits. Read those alongside the headline result before using it to make a decision.
What counts as a test?
We distinguish an actual tool run from a researched comparison, an illustrative workflow and a desk test of a checklist. Our chatbot comparison is researched, not a side-by-side test. Our follow-up blueprint is an illustrative design. The readiness experiment applies a scorecard to invented workflow descriptions; it does not run AI tools on those workflows.
Inputs and expected answers
The four experiments below use invented invoices, inquiries, reminders or workflow cards. They are not customer case studies. The inputs deliberately include awkward cases, so their proportions do not estimate how often those problems occur in a real business.
The reports describe expected answers or scoring rules set before evaluation. Where a script and model are compared, they receive the same cases. The invoice test separates four development invoices from 20 held-out invoices; counting all 24 as unseen test cases would give the wrong impression.
How results are scored
Mechanical checks and judgment are different kinds of evidence. Scripts can compare amounts, dates and required fields. Judging whether a reply is useful or makes an unapproved promise needs interpretation. Each report explains which checks were scripted and which were scored by the site's AI writing assistant.
The AI assistant helped design and score these experiments. The published methods identify Ross C as the editor who reviewed and approved the original reports. The lead-follow-up report distinguishes that original approval from later corrections. Editorial review is not the same as an independent, blinded regrading of every output. The reports disclose those limits, including cases where the same assistant created the inputs and expected answers.
Read the failures and corrections
A passing score does not mean an output is safe to send or use without review. The reminder test's corrected mechanical score still coexists with a customer-note channel mistake. The invoice model copied a bad subtotal accurately without flagging that it did not add up.
When we correct a result, the report should make the change and its reason visible. The lead-follow-up report preserves its original and corrected grades, and the reminder report explains the scoring-script error. Evidence downloads let readers inspect the saved cases, outputs and scoring files where provided. Rechecking saved outputs is different from running a new test.
Explore the experiments
- Invoice extraction: 24 invented invoices, including four development cases; one model run three times per invoice against a rules script. See the held-out results, repeated errors and downloadable evidence.
- Appointment and payment reminders: 30 invented records, one rules run and three model runs per record. No messages were sent. Read both the mechanical checks and the separately labeled judgment scores.
- Lead follow-up drafts: 30 invented inquiries and written contact rules. The report includes a correction, a recheck of the saved outputs and the evidence files.
- Readiness checklist: 12 invented workflows, expected answers and item-level scores. This is a desk test, not evidence that the proposed workflows work.
What the numbers cannot tell you
These are small tests of particular inputs, instructions and models or rules on stated dates. Repeated runs of the same cases are not new independent cases. Synthetic scenarios cannot establish live-business accuracy, compliance or savings. A cost figure does not prove time saved, and an experiment that did not time the work cannot support that claim.
Use the findings to design your own narrow, supervised pilot with representative, approved data. Our pilot evaluation guide covers safeguards, quality, review time and net value. For how we choose topics, use AI in production and handle commercial relationships, see our editorial policy. To flag a problem in a report, contact the editor.