303 lines
14 KiB
Bash
Executable File
303 lines
14 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
#
|
||
# Prove a guard fails before you believe it passes.
|
||
#
|
||
# ## The failure this catches
|
||
#
|
||
# A guard that cannot fail is worse than no guard, because it is trusted.
|
||
# `docs/architecture/GUARDS.md` opens with that sentence and its first rule is
|
||
# this procedure, written out as a manual recipe: back the file up, break exactly
|
||
# the thing the guard protects, run the guard, expect one failure, restore.
|
||
#
|
||
# The recipe is thirty seconds and it is skipped anyway, for two reasons this
|
||
# script removes:
|
||
#
|
||
# - **Restoring is a step you can forget**, and forgetting is silent. The tests
|
||
# pass again once the mutation is undone in your head but not on disk, so the
|
||
# reverted code ships looking green. Here the restore is a `trap`, which runs
|
||
# on success, on failure, and on Ctrl-C.
|
||
# - **Counting the failures is the part people skip.** GUARDS.md §1: "If
|
||
# breaking the guard's target fails three tests, two of them are coincidental
|
||
# and will mask a real regression later." A human doing this by hand sees red
|
||
# and stops reading.
|
||
#
|
||
# ## Usage
|
||
#
|
||
# bash scripts/prove-guard.sh <file> <find> <replace> <test command…>
|
||
#
|
||
# bash scripts/prove-guard.sh src/lib/thing.ts \
|
||
# 'if (body.error)' 'if (false)' \
|
||
# npx vitest run tests/thing.test.ts
|
||
#
|
||
# Everything after the third argument is the command that runs the guard, so any
|
||
# runner works. `$PROVE_GUARD_CMD` is used when no command is given.
|
||
#
|
||
# ## It edits the working tree, so mind what else reads it
|
||
#
|
||
# The mutation is written to the real file and restored by a `trap`. On a host
|
||
# where something else renders or deploys FROM this checkout on a schedule —
|
||
# Gridiron Central renders every fifteen minutes — a mutated template is briefly
|
||
# the one that would be published. Measured 2026-09-10: two agents ran proofs
|
||
# concurrently in one checkout and a render fell between them; nothing shipped,
|
||
# but only because the windows missed. Two proofs at once in one tree is not
|
||
# safe, and a long proof beside a short cron is a race worth knowing about.
|
||
#
|
||
# ## Counting the failures
|
||
#
|
||
# "Exactly one" is a claim about test *cases*, and counting matching log lines
|
||
# does not measure that: Gradle reports a single failing test on six lines — the
|
||
# task, the test, its assertion, the summary, and twice more for the build — and
|
||
# a naive count calls that six coincidental failures. Tried that first; it fired
|
||
# on the very first run against a guard that was behaving perfectly.
|
||
#
|
||
# So the summary line is preferred, because almost every runner prints one and it
|
||
# is the runner's own count. Runners disagree about which side of the word the
|
||
# number goes on, so both orders are read: `1 failed` from vitest, `1 failed, 5
|
||
# passed` from pytest, `6 tests completed, 1 failed` from Gradle — and `fail 1`
|
||
# from node --test, `Failures: 2` from Maven and JUnit, `failures=2` from python
|
||
# unittest, `# fail 1` from TAP. The **last** such line wins, and only if none is
|
||
# found does it fall back to counting lines matching `$PROVE_GUARD_FAIL_PATTERN`
|
||
# — saying so, because an approximate count presented as an exact one is the kind
|
||
# of thing this script exists to object to.
|
||
#
|
||
# ## Exit codes
|
||
#
|
||
# 0 the guard caught it, and nothing else did — the outcome you want
|
||
# 1 the guard stayed GREEN with its target broken. It is not testing what you
|
||
# think it is, and you have just learned that for the price of one edit
|
||
# 2 nothing was proven: bad arguments, missing file, or a find-string that is
|
||
# absent or ambiguous. **Two is not a pass**
|
||
# 3 the guard caught it, but so did something else. Red for more than one
|
||
# reason hides the next regression behind a failure you have learned to
|
||
# expect — narrow the guard, or the mutation
|
||
#
|
||
# The file is restored in every one of those cases.
|
||
set -euo pipefail
|
||
|
||
FAIL_PATTERN="${PROVE_GUARD_FAIL_PATTERN:-(FAIL|✗|[0-9]+ (tests? )?failed|FAILED|AssertionError)}"
|
||
|
||
if [ "$#" -lt 3 ]; then
|
||
sed -n '2,30p' "$0" >&2
|
||
exit 2
|
||
fi
|
||
|
||
FILE="$1"; FIND="$2"; REPLACE="$3"; shift 3
|
||
|
||
if [ "$#" -gt 0 ]; then
|
||
CMD=("$@")
|
||
elif [ -n "${PROVE_GUARD_CMD:-}" ]; then
|
||
# shellcheck disable=SC2206
|
||
CMD=($PROVE_GUARD_CMD)
|
||
else
|
||
echo "prove-guard: no test command given and PROVE_GUARD_CMD is unset." >&2
|
||
echo "Nothing was proven, which is not the same as nothing being wrong." >&2
|
||
exit 2
|
||
fi
|
||
|
||
[ -f "$FILE" ] || { echo "prove-guard: no such file: $FILE" >&2; exit 2; }
|
||
|
||
BACKUP="$(mktemp)"
|
||
cp "$FILE" "$BACKUP"
|
||
|
||
# ── PURGE THE BYTECODE, OR THE RESTORE IS A LIE ─────────────────────────────
|
||
#
|
||
# **A SAME-LENGTH MUTATION SURVIVES THE RESTORE, IN `__pycache__`.** CPython
|
||
# decides a `.pyc` is still valid by comparing the source's mtime — stored as
|
||
# WHOLE SECONDS — and its size. Mutating `"auto"` to `"wiki"` changes neither:
|
||
# same four bytes, and both writes land in the same second. So the interpreter
|
||
# keeps serving the MUTATED bytecode after this script has faithfully put the
|
||
# original source back, and every later run in that second asserts against code
|
||
# that is not on disk.
|
||
#
|
||
# Found 2026-09-08, and it had already happened: `tests/run_all.py` reported
|
||
# `test_digest.py` failing on a guard that passed standalone minutes earlier,
|
||
# because `scripts/__pycache__/gridironlib.cpython-312.pyc` still held `wiki`
|
||
# while `scripts/gridironlib.py` said `auto`. That is the exact failure this
|
||
# whole script exists to prevent — a mutation left in place, looking green —
|
||
# arriving through the door nobody was watching.
|
||
#
|
||
# Purged BEFORE as well as after: a stale `.pyc` from an earlier run would
|
||
# otherwise make the mutation appear to do nothing, and the guard would be
|
||
# reported as failing to catch it.
|
||
purge_pycache() {
|
||
find "$(dirname "$FILE")" -name '__pycache__' -type d -prune -exec rm -rf {} + 2>/dev/null || true
|
||
}
|
||
purge_pycache
|
||
|
||
restore() {
|
||
# Idempotent, because the success path now restores explicitly and the trap
|
||
# still fires afterwards.
|
||
[ -f "$BACKUP" ] || return 0
|
||
cp "$BACKUP" "$FILE"
|
||
rm -f "$BACKUP"
|
||
purge_pycache
|
||
echo "prove-guard: restored $FILE"
|
||
}
|
||
trap restore EXIT INT TERM
|
||
|
||
# Exact-string replacement, and it must be unique. A mutation that lands in two
|
||
# places proves nothing about either, and a regex here would make the mutation
|
||
# itself the thing to debug.
|
||
#
|
||
# Both refusals here exit 2, like every other "nothing was proven" path
|
||
# above. `sys.exit("message")` prints it and exits **1** — the code this
|
||
# script reserves for "the guard stayed GREEN with its target broken", which
|
||
# is a diagnosis about the guard, not a refusal to run. So a mistyped
|
||
# find-string accused the guard under test of being broken. `TOOLS.md`
|
||
# teaches callers to tell 1 from 2 and that "two is never a pass"; that
|
||
# distinction has to survive this block.
|
||
python3 - "$FILE" "$FIND" "$REPLACE" <<'PY'
|
||
import sys
|
||
|
||
|
||
def refuse(message: str) -> None:
|
||
print(message, file=sys.stderr)
|
||
raise SystemExit(2)
|
||
|
||
|
||
path, find, replace = sys.argv[1], sys.argv[2], sys.argv[3]
|
||
text = open(path, encoding="utf-8").read()
|
||
count = text.count(find)
|
||
if count == 0:
|
||
refuse(f"prove-guard: the string to break is not in {path}")
|
||
if count > 1:
|
||
refuse(
|
||
f"prove-guard: {count} occurrences of that string; a mutation in "
|
||
"two places proves neither. Pick a longer, unique one."
|
||
)
|
||
open(path, "w", encoding="utf-8").write(text.replace(find, replace))
|
||
PY
|
||
|
||
LOG="$(mktemp)"
|
||
trap 'restore; rm -f "$LOG"' EXIT INT TERM
|
||
|
||
echo "prove-guard: broke $FILE — expecting '${CMD[*]}' to go red"
|
||
echo
|
||
|
||
if "${CMD[@]}" >"$LOG" 2>&1; then
|
||
echo "prove-guard: FAILED — the guard stayed GREEN with its target broken." >&2
|
||
echo >&2
|
||
echo "It is not checking what you think. Either the assertion does not reach" >&2
|
||
echo "the mutated code, or it would pass without it. Log: $LOG" >&2
|
||
tail -20 "$LOG" >&2
|
||
exit 1
|
||
fi
|
||
|
||
echo "--- what failed ---"
|
||
grep -E "$FAIL_PATTERN" "$LOG" | head -12 || true
|
||
echo
|
||
|
||
# The runner's own count, from the last summary line that states one. Preferred
|
||
# over counting log lines for the reason in the header: one failing test is
|
||
# routinely reported on half a dozen lines.
|
||
# Two orders, because runners disagree about which side the number goes on.
|
||
# The first pattern reads `1 failed` (vitest, pytest, Gradle); the second reads
|
||
# the number on the right: `ℹ fail 1` (node --test), `Failures: 2` (Maven,
|
||
# JUnit), `failures=2` (python unittest), `# fail 1` (TAP).
|
||
#
|
||
# Matching only the first order made every node run fall through to the
|
||
# approximate line count, and that fallback is not conservative. A guard over a
|
||
# status enum — mutating the string `'FAILED'` — matches FAIL_PATTERN three
|
||
# times inside one AssertionError diff, so a single correct guard exited 3 with
|
||
# "narrow the guard, or narrow the mutation". The header says that exact false
|
||
# fire was tried once and rejected; it was still reachable through the fallback.
|
||
# The message compounded it, reporting "this runner printed no summary" about a
|
||
# runner that printed one this script could not read.
|
||
# The third order, and it is this household's OWN harness: `tests/harness.py`
|
||
# prints `<suite>: 1 of 196 checks FAILED` and then one ` X ` line per failed
|
||
# check. Neither pattern above reads it — the first needs the number adjacent to
|
||
# the word, the second needs it on the right — so every run fell through to the
|
||
# approximate count, which matched the summary line AND the ✗ line and returned
|
||
# 2 for a single failure. **A correct guard reddening exactly one check
|
||
# therefore always exited 3** with "narrow the guard, or narrow the mutation",
|
||
# and told its reader "this runner printed no summary" about a runner that
|
||
# printed one. That is the same compounding false fire the comment above
|
||
# describes, arriving through a different door: the fallback is not
|
||
# conservative, and any project using this harness got the wrong verdict on
|
||
# every proof it ever ran.
|
||
# THE UNIT IS A TEST, NOT AN ASSERTION, and reading the check count was the
|
||
# third way this script got the verdict wrong on its own household's harness.
|
||
# GUARDS.md §1 says "if breaking the guard's target fails three TESTS, two of
|
||
# them are coincidental", and §7 wants "exactly one failing test". A test
|
||
# function is one claim with a docstring stating it; asserting that claim three
|
||
# ways is style. Counting checks called that three coincidental failures and
|
||
# told the author to narrow a guard that was already narrow — pressure to
|
||
# assert LESS about a claim, which is how a guard becomes decoration.
|
||
#
|
||
# Measured 2026-09-09: one mutation of one call site in a wiki renderer
|
||
# reddened four checks across two tests and was refused with "narrow the guard,
|
||
# or narrow the mutation". Both reds were the same claim.
|
||
#
|
||
# So `tests/harness.py` now states both numbers — `2 of 477 checks FAILED in 1
|
||
# of 34 tests` — and the test figure is read first. The checks pattern stays
|
||
# beneath it for a harness that has not been updated: over-counting is the
|
||
# conservative direction, and a stale harness must not silently become
|
||
# permissive.
|
||
COUNT="$(grep -oiE 'in [0-9]+ of [0-9]+ tests?\b' "$LOG" | tail -1 | grep -oE '[0-9]+' | head -1 || true)"
|
||
if [ -z "$COUNT" ]; then
|
||
COUNT="$(grep -oiE '[0-9]+ of [0-9]+ checks? FAILED' "$LOG" | tail -1 | grep -oE '^[0-9]+' || true)"
|
||
fi
|
||
if [ -z "$COUNT" ]; then
|
||
# The OTHER shape this household prints. `cronwrap`'s own suite ends on
|
||
# `1 check(s) FAILED: ['name']` — no "of N", so the pattern above misses
|
||
# it and the run fell through to counting lines, which matches that
|
||
# summary AND the per-check line. A correct guard therefore came back
|
||
# "2 failures" and was told to narrow itself, while saying "this runner
|
||
# printed no summary" about a runner that printed one. Same false fire as
|
||
# the two above, third spelling.
|
||
COUNT="$(grep -oiE '[0-9]+ check\(s\)? FAILED' "$LOG" | tail -1 | grep -oE '^[0-9]+' || true)"
|
||
fi
|
||
if [ -z "$COUNT" ]; then
|
||
COUNT="$(grep -oiE '[0-9]+ (tests? )?failed' "$LOG" | tail -1 | grep -oE '^[0-9]+' || true)"
|
||
fi
|
||
if [ -z "$COUNT" ]; then
|
||
COUNT="$(grep -oiE '\bfail(ure)?s?[:= ]+[0-9]+' "$LOG" | tail -1 | grep -oE '[0-9]+$' || true)"
|
||
fi
|
||
COUNTED_BY="the runner's summary"
|
||
if [ -z "$COUNT" ]; then
|
||
COUNT="$(grep -cE "$FAIL_PATTERN" "$LOG" || true)"
|
||
COUNTED_BY="matching log lines, approximately — this runner printed no summary"
|
||
fi
|
||
|
||
if [ "$COUNT" -gt 1 ]; then
|
||
echo "prove-guard: the guard caught it — but $COUNT failures, by $COUNTED_BY."
|
||
echo
|
||
echo "GUARDS.md §1: if breaking one thing fails three tests, two are"
|
||
echo "coincidental and will mask a real regression later behind a red you have"
|
||
echo "learned to expect. Narrow the guard, or narrow the mutation."
|
||
exit 3
|
||
fi
|
||
|
||
# ── AND GREEN AGAIN ─────────────────────────────────────────────────────────
|
||
#
|
||
# **THIS SCRIPT NEVER CHECKED THE BASELINE, SO IT CERTIFIED GUARDS THAT DO NOT
|
||
# EXIST.** It mutated, ran the command once, and called any red a catch. A
|
||
# suite already red for an unrelated reason therefore produced
|
||
# "prove-guard: good — the guard caught it, and only it" for an assertion
|
||
# nobody had written. Found 2026-09-10 by an audit that hit exactly that state:
|
||
# the honest answer was exit 2 and the script said exit 0.
|
||
#
|
||
# GUARDS.md §1's own recipe has this step and the script had dropped it —
|
||
# the recipe ends `cp /tmp/thing.bak src/lib/thing.ts` then
|
||
# `npx vitest run ... # expect: green again`. Restoring and re-running proves
|
||
# two things at once: the red came FROM the mutation, and the restore actually
|
||
# worked. A mutation that leaves the file broken is otherwise reported as a
|
||
# successful proof.
|
||
#
|
||
# It costs a second run of the command. That is the price of the difference
|
||
# between a proof and a coincidence.
|
||
restore
|
||
echo
|
||
echo "prove-guard: re-running with $FILE restored — expecting green again"
|
||
if ! "${CMD[@]}" >"$LOG" 2>&1; then
|
||
echo >&2
|
||
echo "prove-guard: NOTHING WAS PROVEN — the command is red with the file" >&2
|
||
echo "restored, so the red above was not caused by the mutation." >&2
|
||
echo >&2
|
||
echo "Fix the unrelated failure first, then prove the guard. Log: $LOG" >&2
|
||
tail -20 "$LOG" >&2
|
||
exit 2
|
||
fi
|
||
|
||
echo "prove-guard: good — the guard caught it, and only it ($COUNTED_BY)."
|