Coordination state belongs in the database

A dropped call is obvious. You notice immediately and you ring back. A message you never received is not obvious at all — it just looks like nobody had anything to say.

Software has the same two kinds of failure, and the second kind is the one that hurts.

The failure that makes no noise

Most things going wrong are loud. A server falls over, someone gets paged. A page fails to load, a customer complains.

But consider a job that split into twelve smaller jobs and went to sleep until they finish. Something has to remember that it is asleep, and what it is waiting for.

Lose that, and the twelve small jobs still run. They still succeed. They still save their results. The main job simply never wakes up.

No error. No alert. Nothing in any log.

A job that failed is red on somebody's screen. A job that is asleep looks exactly like a job being patient. That is why this kind of problem survives for months.

Two completely ordinary ways it happens

You ship an update

The old version is replaced by the new one. Anything the old version was holding in its head goes with it. This is not a rare event — it is what happens every time you improve anything.

You get busier

This one is nastier because nobody changes a single line of code.

With one machine handling everything, keeping notes in its head works perfectly. Add a second machine to cope with more customers, and now the message saying "your twelve jobs are done" arrives at a machine that has never heard of them. It shrugs and does nothing.

Growth breaks the product. Nobody touched it.

Write it down instead

The rule we ended up with is uncomfortable and short: if losing it would leave work asleep forever, it does not get to live in memory. It goes in the database, where a crash, an update and a second machine cannot touch it.

That forces two habits which are good anyway.

Two things finishing at once must only wake the job once

If the last two small jobs finish in the same instant, both of them will notice everything is done and both will try to wake the sleeper. Get it wrong and the main job runs twice — producing two reports from one week's work, and a customer wondering why they got the same thing twice.

Written down properly, only one of them can win. The other checks, sees it has already happened, and leaves it alone.

Know the difference between "not started" and "was running"

When you ship an update, the system gets a warning and then gets shut down — usually within a couple of minutes. A real piece of work takes five or ten. So the shutdown cannot finish politely, and every update interrupts something.

What you do next depends entirely on a distinction most systems never make:

• Was waiting in the queue and never started. Nothing happened. Nothing was sent. Nobody was charged. Put it back in the queue — running it now is the same as it having waited slightly longer.
• Was actually running. It might have sent an email. It might have taken a payment. You cannot tell from the outside. Leave it stopped, explain why, and let a human decide.

Automatically retrying the second kind is how a customer gets the same payment reminder twice from the company that was supposed to be handling it — and nothing catches it, because the second one looks exactly like a first.

The question worth asking

For anything your system is holding onto: if this vanished right now, would anyone notice?

If the answer is "some work would wait forever and nobody would ever find out" — write it down.