The test that caught it wasn’t testing for this. It was a conservation check: after a batch of concurrent transfers, the ledger’s credits minus its debits should sum to zero. It came back 2450.0000 against an expected 2000.0000.
Nothing had failed. No exception was thrown, no deadlock was detected, no lock acquisition timed out. Postgres was, from its own point of view, working perfectly, and so was the application. The only thing that knew something was wrong was an assertion I had written mostly out of paranoia.
What the code looked like
The transfer executor did three things in order: resolve the currency of the source account so it could be stamped onto the transfer request, lock both accounts in ascending ID order, then move the money.
// step 1 — innocent-looking
var source = accounts.findById(sourceId).orElseThrow();
request.setCurrency(source.getCurrency());
// step 2 — the actual lock
var locked = accounts.findByIdForUpdate(sourceId).orElseThrow();
Step one is a plain read. It looks like nothing. It is the bug.
Why the lock didn’t help
findById loads an Account entity into Hibernate’s persistence context — the
first-level cache, scoped to the session. When findByIdForUpdate runs a moment
later, Hibernate genuinely issues the SELECT ... FOR UPDATE. The database
genuinely takes the row lock. The fresh column values genuinely come back over
the wire.
And then Hibernate throws them away.
The identity map guarantees that within one session, one row maps to one object instance. An entity with that ID is already managed, so the already-managed instance is what you get back, populated with the values from before the lock was held. This is documented, intentional behaviour and it is usually what you want.
The result is the worst possible shape for a concurrency bug. The lock is real, so no deadlock or timeout surfaces. The read is stale, so the balance you write back is computed from a value another transaction has already changed. A textbook lost update, produced by two mechanisms each behaving correctly.
The fix, and the part that matters more
The fix is small: don’t materialise the entity before the lock.
// scalar projection — nothing enters the persistence context
String currency = accounts.findCurrencyById(sourceId);
findCurrencyById returns a String. No entity, no identity map entry, nothing
for the locked read to collide with.
The part that matters more is that this bug is reintroducible by being helpful.
Anyone who later needs the account’s owner, or its type, or its status, before the
lock will reach for findById, and everything will look fine. So
findByIdForUpdate now carries a comment saying exactly this, and the
conservation assertion runs in roughly twenty tests rather than one.
What I took from it
Concurrency bugs at the ORM layer don’t announce themselves through the ORM’s error channels, because from the ORM’s perspective nothing went wrong. The only thing that catches them is an invariant asserted over the data itself — something that is true of a correct ledger regardless of how the code is written. If your tests only check that operations return what you expect, a bug that makes every operation succeed is invisible to them.