Five of Our Live Sites Could Not Be Built From Git

We run 21 websites, each in its own repository, all importing one shared package of layouts, styles and helpers. In October we started building each site from a fresh clone, the way a stranger or a CI server would. Five of the 21 could not be built. All five had been serving pages for months, and every health check we had passed on every one of them.

The first one was found by accident

We were rolling a small robots.txt fix across the portfolio. On one site, a web-capture tool, the rollout stopped. The site’s code imported the shared package from a base-site directory, and that directory was not in the repository. Not as a submodule, not as committed files, not even in .gitignore. The lockfile was untracked too, so a clean install could not run either.

The site had served a styled, working page since the day it was scaffolded. So production had been built from a working tree holding files that were never committed, and had been running on them ever since.

We added the shared package as a submodule, pinned to a specific commit, and committed the lockfile. The repository could now, in principle, be built from a clone.

Then the first clean build failed twice more

This was the first time that code had ever been compiled from what git held, and it surfaced two defects that had sat in production code for months:

• A shared component imported a small file from the site that used it. Every other site carried that three-line file. This one never had, so the dependency ran from the site into the shared package and back out into a file that did not exist.
• A page passed a click handler from a server component, which fails the build with Event handlers cannot be passed to Client Component props. The fix is to move the interactive part into a client component. We had already fixed exactly this defect in the shared footer. This copy had simply never been compiled.

An uncommitted dependency is not one missing directory. It is every error the compiler never got to report.

The check we built

The gap was plain. Nothing in our fleet ever built a site from a clean clone. Every build ran in a tree that had been alive since the site was created. So we wrote a check that runs one site per day on a rotation, which covers all 21 every three weeks. It does five things in a temporary directory:

1. Clone the repository from the remote.
2. Initialise submodules at the commits the repository pins. Never at their latest commit: that builds against code the repository never recorded, which hides exactly the drift being tested for.
3. Check that the files a clean clone must have are there.
4. Install from the lockfile with npm ci, development dependencies included, the way a hosting provider installs.
5. Run a production build.

Its first real run failed four sites

Four sites declared the shared package as a submodule in .gitmodules. But git did not track that path as a submodule. It tracked it as a symlink, file mode 120000, pointing at ../../packages/base-site.

Inside our working copy that path resolves to a real directory, so every local build had always worked. In a standalone clone it points at nothing. The submodule step reports nothing to do, because there is no submodule entry to initialise, and the build dies with ENOENT: no such file or directory.

That cost us two things. Nobody who clones those four repositories can build them. And none of them recorded which version of the shared package it runs, because a symlink has no commit hash. The 16 correctly wired sites can answer “which shared code is this site built against” from git. These four could only answer “whatever the shared directory held on the day of the build”.

Why nothing had caught it

We are not short of checks. One confirms each site returns 200 and that its stylesheets, scripts and sitemap load. One exercises the actual tools on the sites that are tools. One compares each live page title against the brand on main, which we added after a July incident where a deploy replaced a working production build that had been running for three months from a commit never merged to main.

Every one of them asks whether production is up. None asks whether the repository can reproduce it. A site built from uncommitted files passes all of them, and it passes them indefinitely.

Our host never caught it either, because our host never built these sites. We build locally and upload the output. The day a hosting provider installs and builds from the repository is the day all five would have failed, and that would have been an outage, not a finding.

The fix changed nothing that users see

For the four symlinked sites, we replaced each symlink with a submodule pinned to the exact commit the shared directory was already on. The build output was therefore the same as what production was already serving, so none of the four was redeployed. Mixing a wiring fix with an upgrade of the shared code would have left us unable to tell which of the two broke anything.

One detail cost a wrong turn. git submodule add is the obvious command, but .gitmodules already declared the submodule, so it was the wrong tool. A plain clone into place followed by git add is what records the submodule entry. We confirmed each path’s mode changed from 120000 to 160000 before committing.

A sixth site passed and was still wrong

A loan calculator passed the check. It did not link to the shared package at all; it held a copy, 31 plain files frozen in March. Copied files are in every clone, so it built. It had also drifted seven months behind, which is why the robots fix had to be applied to it by hand. The clean-build check cannot see that, and was never meant to.

Converting it taught us one more thing about git. Running git commit with a list of paths cannot turn a directory of committed files into a submodule. Git kept the directory, committed the eight drifted files at their new contents, and left all 31 tracked. The commit message described work that had not happened. The second commit, with git rm -r --cached, git add and a plain git commit, did what the first one claimed.

Where it stands

Today the rotation reached one of the four symlinked sites, a mortgage calculator, and it built clean from a fresh clone at the exact commit that fixed it. That is the fix verified the way it was always supposed to be checked, by someone who has only the repository. Ten of our 21 sites now have a recorded clean build. The rest come up over the next eleven days.

What to take from it

If you deploy output you built yourself, nothing in your pipeline is checking that your repository can build. Add one job that clones fresh, initialises submodules at their pinned commits, installs from the lockfile and builds. It does not need to run on every commit. One repository a day is enough to put a ceiling on how long this can go unnoticed.

And read your green checks for the question each one actually asks. Ours all asked “is it up”. It was a different question, and a check that returns a confident answer to the wrong question is the failure we keep finding. The same lesson came up with our RSS feed, which was also something we thought was source and turned out to be the output of a build.

Counts are from our own build records. The five sites that could not be built were found and fixed on October 15, 2026. The four symlink fixes needed no deploy, because their build output did not change.