We Typed Our Experiment Arms by Hand. 3 of 4 Were Wrong.

We set up our outbound email test so that nobody could choose a company’s category after seeing whether it replied. A frozen classifier decided the category and stored it on the record. Then the experiment arm, the field the analysis would actually group by, was typed by hand. When we finally checked it against the classifier, three of the four hand-written arms were wrong.

The design

The question was whether cold emails do better in some industries than others. On September 28 we wrote down two arms. Arm A: security and compliance. Arm B: fintech, legal tech, health tech and marketing tech. Developer tools and infrastructure were left out on purpose, and the date of the read was fixed in advance. That is ordinary pre-registration: write the analysis down before the data can tempt you, because otherwise every choice made after seeing results is a fork in what Gelman and Loken called the garden of forking paths.

The guard against fooling ourselves was the classifier. Each prospect is classified when it is qualified, by keyword rules pinned to a fixed version, and the verdict is stored on the record as its vertical. The rules cannot change mid-test, so a category cannot be picked once the outcome is known.

Where the arm actually came from

The mapping from vertical to arm existed in one place: a paragraph in the decision document. The vertical_arm field on each record was written by hand. No script set it, derived it, or compared it to anything. A search of our scripts for the field name returned a single hit, and it was inside a comment.

When we derived the arm mechanically from each record’s own stored vertical:

The typed field was wrong in both directions. Three tags put records into an arm their own classifier verdict did not support, including an infrastructure company in Arm A after infrastructure had been removed from the hypothesis. Four records that did belong to an arm had no tag at all. One record out of eight was right.

The frozen classifier did its job. The hand-written field next to it undid that work.

Why it went unnoticed

Nothing read the field until the analysis would. A pre-registered test is quiet by design: you do not look at the grouped results until the read date, which is the point. But that also means a broken grouping column produces no symptom until the one moment it matters. A field that only the final analysis consumes is a field nobody tests.

The design had been careful about the classifier and said nothing about the step after it. Freezing the input to a decision does not help if the decision itself is made somewhere else, by hand.

What we built, and what we refused to do

The arm is now derived in code from the stored vertical, using the arm definitions copied verbatim from the decision document. A checker reports the counts the read will actually use and exits 2 if any record that could still be emailed carries an arm that contradicts its vertical. The mailer runs that check before every send and refuses while it fails.

For one queued record, the wrong tag was cleared before any email went out, with the reason written on the record. A correction made before the send is just fixing an input.

For records already emailed, we changed nothing. Moving a contacted company from one arm to another, after it has had the chance to reply, is the post-hoc re-slicing the frozen design exists to prevent, even when the move is a correction. So the tag stays as it was typed, and the read ignores it: a record whose stored vertical is not in an arm is not in that arm, whatever its tag says. The checker deliberately writes nothing.

The stored vertical has a problem of its own, which surfaced the same day: the classifier matches keywords as bare substrings, so ‘NDA’ fires inside ‘calendar’. That is a separate fault with a separate fix. Deriving the arm from the vertical removes the hand-typed layer; it does not make the vertical correct.

The rule

If a field can be computed from evidence you already store, compute it when you read it. A stored copy of a derivable value is a second source of truth, and the two will drift the first time someone fills it in from memory. Database designers have a name for this risk, the update anomaly, and normalisation exists largely to remove it.

If you must store it, for speed or for history, check the stored value against the derivation in code that has an exit status, and make the action that depends on it refuse when they disagree. A mapping that lives in a paragraph is a note. It needs a check with an exit code to become a rule.

For experiments in particular: freezing your classifier, metric or cohort definition is half the job. Audit every field between that frozen input and the grouped result, and ask of each one whether a person can type into it. Anywhere they can is where the design will leak.

It is a relative of two failures we have written about before: a checker that verified every claim an email made and none it denied, and a gate that printed every verdict and saved none. In each case the careful part of the system was real, and the step right after it was trusted without a check.

From our own engineering records, October 18, 2026. The checker and the send gate shipped the same day. No prospect is named. The test’s results are not reported here; its read date has not arrived.