Audit the built HTML for SEO invariants and the live site once per crawler #228

New Issue

Closed

opened 2026-09-10 04:13:34 -05:00 by null · 1 comment

null commented

2026-09-10 04:13:34 -05:00

Owner

What is true now. Nothing checks the built HTML:

react-helmet-async 3 on React 19 does not merge, so a second <SEO> on a page ships two titles and two canonicals;
a missing title ships a page with no <title> at all;
nothing checks internal links, anchors, or that the sitemap matches the route list;
after a deploy, scripts/deploy.sh:430-458 checks only /api/health.

What to do. scripts/lib/html-audit.js plus scripts/audit-html.js.

Build mode, run as scripts/verify.d/15-built-html (mode 755, after 10-build): exactly one title, one description, one canonical equal to the site origin plus the route, and one h1 per page; JSON-LD parses and has no FAQPage; no U+2014; the preload matches the hero; internal links and fragments resolve; no run-on descriptions; 404.html is noindex and not in the sitemap; the sitemap equals the route list. Exit 2 when dist/ is stale or missing.
URL mode (--url): every sitemap URL fetched once per crawler (OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot, bingbot). HTTP 200, identical bytes across agents, the same page checks, and lastmod present. A by-hand check after a deploy, not a gate.

Traps. The page body is one line and entities are escaped: parse, never grep -c. Strip comments first; the index.html template contains a commented <link rel="preload">. A spoofed agent string is not the real crawler.

Verify: bash scripts/verify.sh shows guard 15-built-html passing; each proof mutation (a doubled title, a wrong canonical, a second h1, broken JSON-LD, a FAQPage, an em dash in JSX, a link to /servicez, #contact-forms) fails it with exactly one finding; and node scripts/audit-html.js --url https://queuenorth.com exits 0 after the release.

**What is true now.** Nothing checks the built HTML: - react-helmet-async 3 on React 19 does not merge, so a second `<SEO>` on a page ships two titles and two canonicals; - a missing title ships a page with no `<title>` at all; - nothing checks internal links, anchors, or that the sitemap matches the route list; - after a deploy, `scripts/deploy.sh:430-458` checks only `/api/health`. **What to do.** `scripts/lib/html-audit.js` plus `scripts/audit-html.js`. - **Build mode**, run as `scripts/verify.d/15-built-html` (mode 755, after `10-build`): exactly one title, one description, one canonical equal to the site origin plus the route, and one h1 per page; JSON-LD parses and has no FAQPage; no U+2014; the preload matches the hero; internal links and fragments resolve; no run-on descriptions; `404.html` is noindex and not in the sitemap; the sitemap equals the route list. Exit 2 when `dist/` is stale or missing. - **URL mode** (`--url`): every sitemap URL fetched once per crawler (OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot, bingbot). HTTP 200, identical bytes across agents, the same page checks, and `lastmod` present. A by-hand check after a deploy, not a gate. **Traps.** The page body is one line and entities are escaped: parse, never `grep -c`. Strip comments first; the `index.html` template contains a commented `<link rel="preload">`. A spoofed agent string is not the real crawler. Verify: `bash scripts/verify.sh` shows guard 15-built-html passing; each proof mutation (a doubled title, a wrong canonical, a second h1, broken JSON-LD, a FAQPage, an em dash in JSX, a link to `/servicez`, `#contact-forms`) fails it with exactly one finding; and `node scripts/audit-html.js --url https://queuenorth.com` exits 0 after the release.

null added this to the Batch 18: Guards and landmines found building Batch 17 milestone 2026-09-10 04:13:34 -05:00

null added the

infra

seo

enhancement

labels 2026-09-10 04:13:34 -05:00

null referenced this issue from a commit

2026-09-10 04:51:13 -05:00

feat(build): audit what the site actually serves, in the build and on the wire

null closed this issue

2026-09-10 04:51:13 -05:00

null commented

2026-09-10 04:51:25 -05:00

Author

Owner

Done in ee5186c. scripts/lib/html-audit.js plus scripts/audit-html.js, wired as guard 15-built-html after 10-build. Build mode requires per page exactly one title, description, canonical (site origin plus route) and h1, JSON-LD that parses with no FAQPage, no em dash or U+FFFD, a preload naming the actual hero, and no run-on description; across pages every internal link and fragment must resolve, the sitemap must equal the route list, and 404.html must carry noindex and no canonical. It exits 2 when dist/ is missing or stale. URL mode fetches every sitemap page once per crawler (OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot, bingbot), requiring 200, identical bytes and a lastmod. Proven by nine mutations, each caught: wrong canonical (7 pages), second h1 (4), FAQPage, em dash in copy, link to a missing route (18), fragment not on target (7), preload keyed on the old attribute (19), template title left in place (19), industry routes dropped (56). Clean build audits clean; URL mode passes against localhost as all five crawlers. Harness: scratchpad/prove-audit.sh.

Done in ee5186c. scripts/lib/html-audit.js plus scripts/audit-html.js, wired as guard 15-built-html after 10-build. Build mode requires per page exactly one title, description, canonical (site origin plus route) and h1, JSON-LD that parses with no FAQPage, no em dash or U+FFFD, a preload naming the actual hero, and no run-on description; across pages every internal link and fragment must resolve, the sitemap must equal the route list, and 404.html must carry noindex and no canonical. It exits 2 when dist/ is missing or stale. URL mode fetches every sitemap page once per crawler (OAI-SearchBot, PerplexityBot, ClaudeBot, Googlebot, bingbot), requiring 200, identical bytes and a lastmod. Proven by nine mutations, each caught: wrong canonical (7 pages), second h1 (4), FAQPage, em dash in copy, link to a missing route (18), fragment not on target (7), preload keyed on the old attribute (19), template title left in place (19), industry routes dropped (56). Clean build audits clean; URL mode passes against localhost as all five crawlers. Harness: scratchpad/prove-audit.sh.

null referenced this issue from a commit

2026-09-10 05:18:07 -05:00

fix(build): tell a rewritten response from a page served by agent