We Ran a Control Group by Accident. Popularity Predicted AI Mentions at p = 0.0008.

We screen companies for whether an AI assistant names them in category answers. On 6 August we tried a new way of finding companies and got the result backwards: the cohort we sourced was named at 80%, against 8.1% for every cohort before it. The sourcing method was wrong. The comparison it produced is the cleanest evidence we have.

What we were doing

Our records hold 67 company-level visibility checks: a real buying question put to one model, at least three runs each, with runs attempted, runs returning a usable answer, and runs naming the company recorded separately.

We had exhausted our usual way of finding companies and needed another. We tried GitHub: search repositories with recent activity above a star threshold, take the homepage domain, screen for a live blog and a published contact address. It is a reasonable-sounding way to find small technical companies.

Five companies survived every filter. We checked them, expecting the usual result.

The 2×2

Fisher’s exact test, two-tailed: p = 0.00077. Fisher is the right test here rather than a chi-square precisely because one arm is tiny; it makes no large-sample approximation and the p-value is exact.

Two of the four named companies were named in every run, not one in three.

Why this is a control group we could not have built

We check a company because we are considering contacting it, and we contact companies we expect to be absent — absence is the thing our service addresses. So our normal cohort is selected on something close to the outcome, and its 8.1% is manufactured by our own method. We have said so before and we will keep saying it.

A proper comparison needs a cohort screened for the opposite property. We had no legitimate way to assemble one: it would have meant researching companies we had no intention of contacting, purely to make our numbers look better. That is a real constraint on measuring your own business honestly, and it is rarely discussed.

The error did it for us. Stars are accumulated third-party attention. By screening on stars we screened for the property whose absence defines our prospects — and the model’s answers flipped accordingly.

What we think the mechanism is

Not stars. Stars are a marker, not a cause; no model reads your star count when answering “what are the best tools for X.” What a starred project has is the material around it — posts, comparisons, newsletter mentions, forum threads — written by people who are not the company. That is the text a model trained on public writing would have ingested.

This lines up with the other finding we did not go looking for: companies with extensive comparison pages, in one case nine of them targeting named competitors, were still absent from the exact queries those pages were written for. What you write about yourself appears to make you eligible. What others write about you appears to track whether you get named.

We hold this as the best available explanation, not a demonstrated mechanism. This design cannot separate stars from everything correlated with them, and we cannot see inside the model.

What we are not claiming

• Not a population rate. Neither 80% nor 8.1% is what would happen to a randomly chosen company. Both cohorts are screened.
• Not an effect size. Five companies pin down the direction, not the magnitude. The true rate for a popularity-screened cohort could plausibly sit anywhere from around 40% to nearly 100%.
• Not a claim about AI generally. One model, answering from training knowledge, without live web search, in July and August 2026. A different assistant ingested different material; the same assistant with web grounding behaves differently again.
• Not causal. Nothing here was randomised. Two differently screened groups differ in every way their screens differ.

What it changed for us

We rejected GitHub as a source the same day we built it. That is the second source we have rejected for the same reason, and naming the shared error was worth more than either rejection: we kept screening on popularity when the thing we screen for is its absence. Hacker News points, then GitHub stars. Two different platforms, one mistake, twice.

Update, same day: the replacement failed too, and our reasoning was wrong

We built the replacement a few hours after writing the paragraph above. It sourced companies from “alternatives to X” roundups on a software directory — which we argued would mark a company as commercial and in-category while saying nothing about how much attention it has. 849 domains, 91 passing our filters, ten checked against the model.

Roundups versus our normal cohort: p = 0.000052. Roundups versus the GitHub cohort we had just rejected: p = 1.0. The fix and the thing it was fixing are statistically indistinguishable.

The claim in the paragraph we replaced was simply false, and it is worth being precise about how. A directory roundup is not a neutral category marker. It is someone else publishing about you — which is the exact quantity we had just finished arguing was the thing that matters. We described the new source as selecting on a different variable while it was selecting on a weaker version of the same one.

So that is three sources rejected, and the lesson we drew after the second one — “stop screening on popularity” — was not sharp enough to prevent the third. The sharper version: every list of companies is a list somebody made, and being on a list is third-party attention. Any source that is a curated enumeration of noteworthy companies will select for visibility, however indirect the curation looks.

What that implies is uncomfortable and probably right: the source we need is not a list of interesting companies but an enumeration of all companies — something like certificate transparency logs, where you appear because you provisioned a TLS certificate and for no other reason. We have not built that yet and we do not know whether it works.

The general lesson

If you measure your own business, you will find that the cohort you most need for comparison is the one you have no reason to assemble. Watch your mistakes for it. A sourcing error that produces the wrong population is a bad day for the pipeline and can be a very good day for the evidence — but only if you check the result before you throw the batch away.

We nearly did throw it away. The five companies were failures by the only criterion that mattered on the day: none of them were prospects.

And the same-day update above is the other half of that lesson. Publishing the prediction before the test meant we had to publish the refutation a few hours later, in the same piece, against our own reasoning. That is the cost of writing predictions down. It is lower than the cost of not noticing you were wrong.

Frequently asked questions