Search visibility

Everything else on this site is verified before it ships. A script refuses the deploy if a page is missing a description, if two pages claim the same title, if a heading level is skipped, if an image has no alternative text, or if a page exists but is absent from the sitemap. That is the right place for those checks, because they are properties of the code and the code is what changes.

One class of fault escapes it entirely. The file that tells crawlers what they may read can be altered by the network in front of this Worker, after the Worker has finished responding. A check that reads the source sees a correct file. A crawler receives a different one. The only way to tell the difference is to fetch the site from outside and read what comes back, which is what the nightly job does.

This is not hypothetical. It is how the discrepancy recorded below was found, and it was found by looking rather than by reasoning: the first attempted repair looked obviously correct in the source, and measuring proved it changed nothing at all. That fault has since been fixed at its actual cause, which was a setting rather than a file. The check remains, because the same setting can be switched back on without any trace appearing in the code.

The nightly job has not yet recorded a run. It is scheduled for 05:00 UTC daily, and the first results will appear here. Until then this page states its method and nothing more, because an empty table of green rows would be a claim without a measurement behind it.

For a period, the robots.txt this site served was not the one this site wrote. The Worker's own file permits every crawler and then names thirteen AI agents explicitly, so that the machine-readable layer written for them is unambiguously open. The network in front of it was prepending a block that forbade nine agents outright, six of which were agents the site's own file invites.

The consequence was precise: for those six, the invitation was cancelled by the file that governs it. A parser resolves such a contradiction by choosing the most specific matching group and, on a tie, the one that appears first, and the prepended block appeared first by construction. Reordering the site's own groups could not have changed that, which was confirmed by simulating the served file and re-running a standards parser against it: not one verdict changed.

The cause was a setting on the network account rather than anything in the code, which is why the file in version control looked correct throughout. It has since been turned off. The served file is now byte-identical to the one this site writes, and a standards parser confirms that every agent named as welcome can in fact fetch the site. The nightly check that found it remains in place, because the setting could be re-enabled and the failure would otherwise be silent again.

It is recorded here, in the past tense rather than deleted, because a site that asks to be checked should be willing to show what it got wrong and how it knew.

Twelve checks run nightly against the live site: that robots.txt and the sitemap can be fetched at all; that neither blocks the search crawlers or the invited AI agents once served; that robots.txt declares where the sitemap is; that every address the sitemap lists resolves without redirecting; that the five machine-readable files still respond; that the plain-text copies of the doctrine documents point back at the pages they mirror; and that the www address still redirects to the canonical one.

Results are kept for a year. That is longer than the retention promised for contact submissions, and deliberately so: this table holds no personal data, and the reason for keeping history is that a fault introduced by something upstream has no commit to correlate it with, so the only way to say when it started is to have been looking every night.

No third-party service is involved in any of this. There is no analytics script, no verification tag and no external monitor. The checks are made by this site, against this site, and the results are stored in its own database.