August 2026 · PostgreSQL · testing

A green test is not evidence that your rule is why

Locking accounts in ascending ID order prevents deadlocks. My tests passed. That proved nothing until I removed the rule.

Every concurrency test in my transfer service was green. The service locks accounts in ascending ID order — the standard defence against two opposing transfers deadlocking on each other — and no deadlock ever occurred.

Which sounds like proof, and isn’t. There are at least three reasons those tests could have been passing:

  1. The ordering rule works.
  2. The test never actually creates opposing transfers close enough in time to contend.
  3. Something else entirely — connection pool size, transaction boundaries, test sequencing — is serialising the operations before the lock ordering ever matters.

A passing test distinguishes none of these. It tells you the system didn’t deadlock. It doesn’t tell you what stopped it.

Removing the defence

So I broke it on purpose. I replaced the Math.min / Math.max ascending-ID ordering with the obvious naive version — lock the source, then lock the destination — and re-ran the opposing-transfer test unchanged.

SQLSTATE 40P01: deadlock detected
SQLSTATE 40P01: deadlock detected
SQLSTATE 40P01: deadlock detected
... × 6, at roughly one-second intervals

Six deadlocks. Postgres’s deadlock detector firing on its default one-second timer. Then I put the ordering back and the suite went green again.

That is the evidence. The test creates real contention, the contention really does deadlock without ordering, and the ordering really is what prevents it. All three of my alternative explanations are now ruled out, and I know that if someone deletes the rule in six months, this test will notice.

The general form

The question worth asking about any passing test is: what change to the system would make this test fail? If the answer is “I’m not sure,” the test may be passing for reasons unrelated to the property it claims to check.

This is cheap to establish. Delete the mechanism, run the test, watch it fail, restore the mechanism. Ten minutes, and afterwards you know the difference between a test that guards an invariant and a test that decorates one.

It applies well beyond locking. A retry test that passes because the mock never fails, an authorisation test that passes because the endpoint refuses everyone, a validation test that passes because the input was never routed through the validator — all of these look identical to working code from the outside. The only way to tell is to make the thing you are testing stop working, once, on purpose.

← All writing