Two Bugs in Our Email Checker Were Hiding Each Other

A bug report said our pre-send checker matched vendor names as bare substrings. It did, and the fix was obvious: add word boundaries. Before shipping it we ran the fix over our last ten batches of real emails. It would have refused one that was correct and had already gone out. A second bug had been relying on the first one, and nothing in our exit codes could show it.

What the checker is for

Our outbound emails tell a company what an AI assistant said when we asked it a buyer’s question: which vendors it named, and whether it named them. Every one of those names has to be real. Before anything is sent, a checker reads each vendor name the email quotes and confirms that it appears in the stored model answers the email is based on. It also counts how many of the repeated runs named each vendor, because the same question asked three times does not get the same answer three times, and a name that showed up once is weak evidence.

We have written before about this checker verifying every name an email asserted and none it denied. This is a different failure in the same script.

Bug one: a name credited from inside another word

The checker matched each quoted name on its first word, using a plain substring test. One email quoted a writing app called iA Writer. Its first word is ia. That string sits inside obsidian.

One of the three stored runs named Obsidian and several other note-taking apps, but not iA Writer. The checker counted it as naming iA Writer anyway. So it reported the name in 3 of 3 runs when the model had returned it in 2 of 3, and the note that exists to flag a 2-of-3 name never printed. Another name in the same record, which collides with nothing, was correctly flagged at 2 of 3 in the same run.

Nothing false was sent. The email listed the names the model gave without claiming each one recurred. But the general case is worse: a name that appeared in one run, credited a second time from inside another word, clears our two-run minimum and goes out as if it were stable.

This is the oldest problem in text filtering, the Scunthorpe problem, and the same family as the fault in our classifier where ‘NDA’ matched inside ‘calendar’. The fix looked like one line.

Bug two: a sentence about nobody, read as a list of vendors

When a model names no product at all in a category, our emails say so plainly, in a sentence like:

It named no [category] product: not you, not anyone.

The parser that pulls vendor lists out of an email read that sentence as two vendors: no [category] product: not you and not anyone. Neither is a vendor, so neither should have been found in the stored answers, and the check should have failed.

It passed. The first words it extracted were no and not, and the stored answer for that record happened to quote a post about The Knot. Both strings sit inside Knot.

Why the obvious fix could not ship alone

Word boundaries stop no matching inside Knot. That is exactly what we wanted for bug one, and it removes the false credit bug two had been living on. With the boundary fix and nothing else, a correct email that had already been sent would have switched from verified to “vendors not found in any recorded run”, and so would every future email reporting an empty category, since that phrasing is natural whenever the model names nobody.

So the two bugs were not independent. A parser was inventing names, and a matcher was crediting them. Each one hid the other. Fixing either alone makes the system look more broken than before, which is how a correct fix gets reverted.

What we changed

Both fixes landed in the same commit. The two places that matched names now share one matcher. Its boundary is a lookaround on letters and digits, not \b. Python’s regular expression documentation defines \b by word characters, which treats hyphens as boundaries, and the names in our data include many hyphenated ones, such as next-intl, that \b would split. Punctuation around a name still matches, so draw.io still satisfies draw.

A sentence that begins with no, none, nothing, nobody or no one is now read as a claim of absence, not a vendor list. It is reported as a note rather than silently dropped. If the rest of the sentence contains something that looks like a product name, as in “it named no one except Acme”, the checker refuses it, because that is a claim about a name and has to be verified like one.

Verified by diff, not by exit code

We ran the old and new checker over the same ten batch files and diffed the full output. The diff was exactly two added lines: the 2-of-3 note for iA Writer that the old matcher had swallowed, and a note that the empty-category sentence was read as an absence claim.

Every exit code was the same before and after. Each batch that passed still passed and each refusal was still a refusal. If we had judged the fix by its exit codes we would have learned nothing, because the bugs never changed a verdict. They changed what the checker claimed to know. This is what a characterization test is for: record what the code currently does on real inputs, change it, and read the difference line by line. We also asserted eight boundary cases and four parser cases directly, including both new paths.

One limitation stays documented, not fixed. A name is still matched on its first word, so a quoted “Power PDF” is checked as power and would miss a stored answer that writes it as one word. That had zero hits in the ten files, so it is a comment in the code and not more logic.

The rule

When a bug report comes with an obvious fix, run the fix over your real data and diff the full output before you trust it. If anything changes beyond what the report predicted, there is a second bug, and it may depend on the first one. A matcher that is too loose can hold up a parser that is wrong, and tightening the matcher is what exposes it.

The second rule is about exit codes. A checker’s verdict can stay the same while what it believes changes underneath. The pass was correct both before and after. The reasons it gave were wrong before. If you only check the verdict, you will not see that difference. It is related to a gate that printed every verdict and saved none of them: in both cases the output that mattered was not the one anything was checking.

From our own engineering records, October 19, 2026. The fix shipped the same day. No prospect is named; product names that appear here are third-party examples from stored model answers.