Four safeguards, eight bugs, not one self-reported

After a book went live with the wrong cover, we spent a day building four checks: a record of provenance, a delivery gate, an archive, and a check that actually looks at the images.

Those four checks contained eight bugs.

Not one of them reported itself. Every single one came from somebody with a different view.

Four sentences that follow

Each stands for a concrete incident; none is general caution.

1. The test must be able to go red

A run that was only ever green proves nothing — it shows that nothing happened, not that anything would be caught. Every safeguard needs a case that must fail and demonstrably does.

The delivery gate was only proven once a single appended byte turned it red.

2. The test reproduces the real call, not an idea of it

One of the safeguards let three bugs through because the test invocations omitted a required field that the real call always carries. It had been tested against a reconstruction that never occurs in practice.

Take a real input, not an imagined one.

3. What ran on one system only is tested on one system only

Three of the eight bugs were Windows peculiarities, structurally invisible on the Linux runner: how a command name is resolved there, which character encoding the console uses, which shell tool actually starts.

“Probably runs” is not a state. It means untested.

4. “It went red” is not a passed test

This is the most uncomfortable of the four, because it nearly slipped through.

A test was meant to prove that a certain build aborts. It went red — but for the wrong reason: the subprocess died before the build even began. The assertion “the build fails” fires on a broken invocation exactly as it does on a correct abort.

What caught it was a second check looking for a specific marker in the output — and that one had been built for a different purpose. It saved us by accident.

➜ Since then, deliberately: every case that expects an abort additionally checks the signature of that particular abort.

The sentence above all four

The second look is not a courtesy step but the most effective check this project has.

Before a delivery, it belongs on the record who besides the author has looked. That can be another editor, the founder, or an outside checking tool — but not nobody.

What we did not turn this into

We did not retroactively demand a test for every script. As of 3 September 2026: eleven of them a delivery depends on; one had a test. (As of 20 September: twelve scripts, seven with a test of their own, two more with a partial one — the rule has worked without us having to catch up on anything.) A test comes into being when a script is new, when it is changed in a load-bearing place, or when a bug has been found in it — then as a regression case.

A rule that cannot be kept will not be kept. Better a smaller one that holds.