The check that passed because it measured nothing

Somewhere in your build there is probably a check that cannot fail. Not one that rarely fails. One that is structurally incapable of going red, and that has been reporting success the whole time.

Every check below did exactly that. None of them threw, none of them warned, and none of them was measuring the thing it claimed to measure. They came out of one codebase over a few months, and every one was found by accident rather than by the check itself.

Run them. They all reproduce on an ordinary machine in a few seconds, and reproducing one is a faster way to believe this than reading about it.

The fallback after the pipe is dead code

printf '' | head -5 || echo FALLBACK

Nothing prints. The exit status is zero.

The || binds to the whole pipeline, a pipeline’s status is its last stage’s, and head exits zero whether or not it read anything. So a guard written this way can never run its fallback. It looks like a safety net and it is a no-op.

The form that works captures the output and tests the value, because that is the only shape that can tell “nothing came back” apart from “something did”.

out="$(some_command)"
if [ -z "$out" ]; then
  echo FALLBACK
fi

The word boundary your grep does not have

Take a file with the word manifest on three separate lines.

git grep -E '\bmanifest' -- notes.md
git grep -cE 'manifest' -- notes.md
git grep -cP '\bmanifest' -- notes.md

The first prints nothing and exits 1. The second and third both report three lines.

Word boundaries are a GNU extension. Git’s POSIX extended regex does not carry one, so the first command matches nothing, and exit 1 is byte for byte what a genuine absence returns.

Now picture what that costs on a banned-word sweep. Someone writes a check to keep a list of words out of published copy. They write the patterns with \b, because that is the careful way to write them. They ship a check that reports every file clean, for every term, forever. It cannot fail. It was never able to.

Drop the boundary and filter the hits afterwards, or use -P where PCRE is available. Either way, pair the sweep with a control term the file is known to contain, so a run that finds nothing is distinguishable from a run that cannot find anything.

The shell splits where you did not want it to

zsh and bash disagree here, and the disagreement is silent in both directions.

files="a.md b.md"
ls -- $files

In bash that lists two files. In zsh it looks for one file named a.md b.md, because zsh does not word-split an unquoted parameter expansion. A path-filtered check written this way passes zero real arguments, matches nothing, and exits 0.

Then the other direction:

for f in $(cat list.txt); do
  echo "[$f]"
done

zsh does split an unquoted command substitution, on spaces as well as newlines. A file named two three.md arrives as two loop iterations and both of them are wrong.

So the same shell refuses to split where you wanted it and splits where you did not, and neither case raises anything. Read line by line and pass literal arguments:

while IFS= read -r f; do
  printf '[%s]\n' "$f"
done < list.txt

If a script carries a bash shebang it word-splits normally and none of this applies. This bites commands typed inline, which is most of what an agent runs.

The last line of a failure is often blank

python3 -c "raise AssertionError('unlinked\n')" 2>&1 | tail -1

Empty.

The message ends in a newline, so the last line is the empty one. A check that reads a failure through tail -1 compares an empty string against whatever it expected and calls it a pass. The pipeline’s exit status belongs to tail, so the status agrees with the empty string and nothing anywhere disagrees.

Read whole output and exit codes. Never judge a failing case through head, tail, or any pipe.

The grep that respects your ignore file

Modern search tools skip ignored paths by default, and on a lot of machines grep is a shell function wrapping one of them.

type grep

If that prints a function or an alias rather than a path, your recursive search is not searching what you think it is. In a directory with a vendored folder listed in .gitignore, and the same string present inside and outside it:

grep -rn needle .
command grep -rn needle .

The wrapper found one hit. The real binary found both.

This one is worse than it looks, because the ignored directories are usually the vendored code, and vendored code is exactly where you go looking when the question is “does anything still use this”.

The comparison that cannot see a squash merge

git cherry main feature

git cherry compares patch ids. A squash merge mints one commit whose patch id matches none of its inputs, so every commit on a long-since-landed branch comes back marked +, meaning unlanded. At the same moment:

git diff --quiet main feature

reports that the trees are identical. Both of those are true together.

Ask a question the shape of the history can answer. Is the branch head an ancestor of the default branch?

git merge-base --is-ancestor "$(git rev-parse feature)" main

Or read the merged pull request’s own commit list. Either one answers what you actually asked.

What they have in common

None of these is a bug in the tool. head really does exit zero. Git’s extended regex really has no word boundary. zsh really is documented not to split parameter expansions. Every one of them is correct behaviour meeting an assumption nobody wrote down.

What they share is the failure signature. Each reports success, and each reports the same success it would report if the thing being checked were fine. That is what makes them expensive. You cannot find them by reading the output, because the output is what you wanted.

So make it fail

The habit that catches the rest is small. Before you trust a check, break the thing it checks and confirm the check goes red. Once, out loud, and then keep doing it.

We put that in the pipeline for the check we care most about. Kin has a rule that its answer paths never read raw files, and a guard script that scans those paths for filesystem primitives. A guard like that is easy to fool by accident, so a second script exists whose only job is to fool it on purpose and fail the build if it does not notice.

scripts/falsify-zero-file-search.py copies the source tree to a scratch directory, plants a file read into every module the guard claims to cover, at several points in each including the very end of the file, and requires the guard to fail and to name the offending file every time. Three details in it carry the whole lesson.

One probe shows one location. The harness used to plant a single read at the top of one file. That showed the guard worked at that spot and nowhere else, and it passed for as long as a comment-tracking desync left roughly the last third of the largest answer module unscanned. The probe happened to sit inside the region that still worked.

Coverage can be lost from the exemption side, and that direction is quieter. The guard has an allowlist. An entry that grows to cover more than it should produces a summary line and an exit status identical to a clean run. So the harness attacks that directly, by planting a brace inside a comment in an allowlisted function body, which is the exact spelling a brace counter without lexical awareness reads as an unclosed body before excusing everything after it.

The list of covered modules is read out of the guard rather than restated. A module added to the guard’s coverage gets falsified automatically, and the harness cannot quietly fall behind what the guard claims.

One more thing worth stealing. Every probe asserts that the scan reported the file, not merely that the tool exited nonzero, because an allowlist error also exits nonzero and also prints a path. The failure mode you are hunting is the one that masks the scan, so an exit code is not a fine enough instrument to hunt it with.

The job that runs the harness also runs a formatting check first, and a comment in the workflow says why: the harness locates function bodies textually, by indentation and matching braces, so an unformatted tree could move a probe site, leave the harness nothing to probe, and let it report success having shown nothing. The meta-check needs its own precondition checked. That is usually where these end up.

The line worth keeping

A check you have never watched fail is not evidence. It is a check-shaped thing sitting in the place where the evidence goes, and the difference only shows up on the day you needed it.

Both scripts are in scripts/ in the Kin repository, which is Apache-2.0. Kin is early and it is an alpha, and this habit is most of the reason I trust anything in it. If you run a guard of any kind, the question worth asking today is when you last watched it go red.