Our Health Check Was Green for a 16-Day Google Outage

For 16 days, Google sent our 21 sites almost no search traffic. Impressions fell from about 2,750 a day to between 3 and 163. Our scheduled daily cycle ran through the outage and reported every site OK. That report was true every time. It was also the wrong measurement.

What the numbers looked like

The outage ran from September 18 to October 3, 2026. Both of the series we track broke between September 17 and 18, and both recovered on October 4, completely, in one day:

Bing, DuckDuckGo and the other engines carried on as normal, which is part of why it was easy to miss: total traffic did not go to zero, only the Google part of it. Bing already sends us more visits than Google does, as we found when we checked which search engines actually send us traffic.

We found the outage on October 14, ten days after traffic came back, while investigating something else.

The theory we almost acted on

A system that misses a 16-day loss of its largest search engine looks unscheduled. So the plan was to write a scheduler, an install script and a request for the permissions it would need, and report that the gap was that nothing ran every day.

Before writing the scheduler, we checked whether one already existed. It did. It had been installed for weeks and its last run had exited cleanly. Its run history showed completed cycles on at least seven days inside the outage window, including September 19, its second day, and October 3, its last. The permissions the plan was going to request had been granted a month earlier. The first day, September 18, has a log but no completed run: it says the laptop went to sleep mid-run. That is a coincidence. We record it so nobody builds a theory on it.

Both fixes we were about to build were already in place, and the system had been blind anyway, because neither one was what was missing. The habit that saved us an afternoon is simple: check the artefact, not the note about the artefact.

Supply is not demand

Here is what the health check verifies, for every live site: the homepage returns 200, the CSS and JavaScript files it references actually load, and the sitemap serves. It exists because a partial deploy once served pages with no styling while returning 200, and nobody noticed. It is a good check. It proves that a site serves.

During the outage every site served perfectly, to every visitor who arrived. The problem was the number of visitors arriving from Google, and nothing in the cycle measured that. The report from day 8 of the outage reads, in full on the line that mattered: Health: 21/21 sites OK, tool smoke test OK. It says nothing about demand because nothing that produced it looked at demand.

Google’s own guide to monitoring distributed systems lists traffic as one of its four golden signals, alongside latency, errors and saturation. We had built careful checks for errors. For a set of sites whose whole purpose is to be found, the signal we left out was the most important one.

This is the same shape as the problem in five of our sites that could not be rebuilt from git: every check passed, and none of them asked the question that mattered. There it was whether a clean clone could build. Here it was whether anyone was coming.

Why a daily green message is worse than no message

Every successful run sent the same short message: the cycle is done, with a few lines of log. About thirty were attempted in the weeks around the outage. Each one was accurate. A message that arrives every day saying the same thing teaches its reader to stop reading it, so the day that matters looks exactly like the twenty-nine that did not. Hospitals call the clinical version of this alarm fatigue.

There was a second fault inside the first. The function that sent the message returned success when its credentials were missing and ignored the result of the request. A message that was never delivered looked the same, from inside the cycle, as one that was. We cannot tell how many of the thirty actually arrived.

What we added

A demand-side watchdog now runs at the start of each daily cycle, before any other work. It pulls the latest search-traffic snapshot from the Search Console API and turns it into one of four exit codes:

• 0, clear.
• 3, real traffic loss, corroborated by our analytics. This is a P0.
• 4, a loss in search data that analytics cannot confirm.
• 1, cannot tell: no snapshot, an unreadable one, a stale one, or one missing the field the verdict comes from.

The last one matters most. Our search credentials had once been broken for five months over a single unexpanded character in a file path, and a broken pull must never read as a quiet day. An absent value is not a claim of zero, which is the point of an empty result is not a failed request turned the other way round.

The watchdog reports and never aborts. A watchdog that can stop the cycle it watches would be a new way for the cycle to fail. The clear path sends no message at all, so the only message it ever sends is one worth reading, and the traffic verdict is now the first line of the daily report. The message sender reports its own failures. We tested each branch by taking it rather than by reading the code: a fixture for each verdict, an invalid credential, an empty directory, a real pull, and one labelled live message.

What this does not fix

We still do not know what caused the outage. We have ruled out a reporting gap, a code deploy and robots.txt, and Google’s URL Inspection shows the pages stayed indexed and fetchable. This post is about why it ran unseen, not why it happened.

The watchdog also covers Google only. Bing is our larger search channel and still has no equivalent check, so a Bing outage today would be as invisible as this one was.

The rule

A scheduled check that measures the wrong thing is not a safeguard. It is a daily reassurance. After an incident, before you build the thing you assume was missing, go and look for it. Then ask what your existing checks would have had to measure for this failure to turn one of them red. If the answer is none of them, that is the gap, and more scheduling will not close it.

From our own engineering records, October 14, 2026. Traffic figures are from Google Search Console and GA4 for our own properties. The cause of the outage is open and is not speculated on here.