Our backups skipped the show for ten days

Mamba Smith is the founder of Queen City Garage, which produces The Grand National Show: a weekly NASCAR panel show on Grand National Today where broadcasters and drivers argue about the last race and pick the winner of the next one.

For ten days in September, the finished files of The Grand National Show had no backup. The nightly backup job reported "ok" every one of those nights. So did the check that was supposed to catch the backup job failing.

I am writing this down because a studio that only publishes its wins is a studio you should not trust.

What happened

When we built the delivery system after Episode 1, the finished deliverables got their own storage bucket. By the time we found the gap it held 100 objects and 4,866 megabytes, including the Episode 2 master.

The nightly backup job copied buckets from a list. The list had eight buckets in it. The new one was not on it.

The watchdog that checks whether every bucket has a recent backup also worked from a list. The same eight. So the job skipped the show's files, the watchdog agreed that nothing was missing, and the nightly board showed a clean line for ten days.

The job's own comments even warned that a new bucket only needed to be added to the list. That entry was exactly what nobody made.

Why this is the interesting kind of failure

It is not that someone forgot. Someone will always forget. It is that the check and the thing it checked shared one blind spot. Two lists, maintained by hand, that happened to be the same list. When the list was wrong, both were wrong together, and the board could not tell.

We had seen this shape before in other parts of the platform, where a hand-typed mapping silently dropped things that were added after the mapping was written. The fix we have learned to reach for is never "be more careful." The fix is: do not keep a list. Ask the system what exists, and back up everything it names.

What changed

No hand-kept list survives anywhere in the backup path. The mirror asks the system what exists and copies all of it, so a bucket created tomorrow is copied tomorrow night. And the watchdog deliberately asks a different question than the job does: not "did the copies work" but "did the job report at all," so it can no longer inherit the job's blind spot.

The whole set is now mirrored every night to a second provider, off-site, on the principle that two copies in one account is not a backup, it is a hope. The first mirror ran the day the gap was found: 11,357 objects, 22.16 gigabytes, zero missing. A restore drill then pulled six files back from the off-site copy and matched every hash, including the 1.25 gigabyte Episode 2 master, streamed in full.

Nothing was lost. That is luck, and we do not want to depend on it again.

Why tell you

Because the same rule runs through everything else in this series. The eight gates work because they check for an artifact rather than trusting a report. The backup failed for ten days because its check trusted the same list the job did. A check that cannot disagree with the thing it checks is not a check.

We have a lot of automation around this show. Every piece of it is only as honest as the thing it compares against. This is the first post in the series where that lesson cost us something, and it will not be the last.

What a person did: found the gap, and changed the rule from "keep the list current" to "there is no list." What the machine did: reported green for ten nights, because it was asked the wrong question.

This is No. 05 in How We Make the Show, the Queen City Garage build log on producing The Grand National Show. Watch the show at Grand National Today.