Checking our own copy proved nothing

Mamba Smith is the founder of Queen City Garage, which produces The Grand National Show: a weekly NASCAR panel show on Grand National Today where broadcasters and drivers argue about the last race and pick the winner of the next one.

At 9:29 this morning I reported that my own clips from this week's Truck Series episode had the sound out of step with the picture. Every automated sync check we run had passed them. The checks were right about what they measured and wrong about what mattered, and the difference is the most useful thing I have learned making this show.

What went wrong

The step that lines up the four separate recordings on one clock before the edit, which we call the conform, did picture and sound in a single pass. For two panelists, that moved the sound by as much as 0.7 seconds relative to the picture. Not much. Enough that lips and words visibly disagree.

Every sync check downstream compared the finished cut against our own conformed copies. Against those copies, the sound was in step, because the copies carried the same shift. The check passed. The second version of the finished show, the master, went live on the episode page at 10 a.m. with the shift in it.

The fix

Picture and sound are now conformed in separate passes, so a correction to one cannot drag the other. And there is a new check that measures lip sync against the original recordings as they came down from the studio platform, not against anything we made.

The corrected clips went out at 10:46. A redone clip for one segment followed at 11:14. The third master was built, checked at eight points across the show with picture and sound within one frame at every one, and swapped onto the live episode page at 11:50. A corrected production package went to the panel at noon, saying plainly that the error was ours. The previous week's Truck episode was checked the same way and is clean.

The lesson, which is bigger than sync

A check is only as good as what it compares against. If the reference shares the fault, the check will pass the fault.

That is the same shape as the backup that reported green for ten days: the job and its watchdog shared a list, so the watchdog could not disagree with the job. Here the conform step and the sync check shared a copy, so the check could not disagree with the conform.

The rule we took from both is that a reference has to be independent of the thing it checks. Backups are now compared against what exists, not against a list. Sync is now compared against the source, not against our own output. If anything else in this pipeline checks itself against itself, we want to find it before it finds us.

Why say so

Because the version with the shift was live for an hour and fifty minutes, and it was found by a person watching his own clip, not by a check. That is the failure. A show that tells its audience how carefully it is made owes them the days when the care was not enough.

What a person did: watched his own clip and said it was wrong when every check said it was right. What the machine did: now checks lip sync against the original recordings, and rebuilt the master to within one frame at eight points.

This is No. 13 in How We Make the Show, the Queen City Garage build log on producing The Grand National Show. Watch the episode: Jake Garcia Breaks Through at Bristol, Kansas Picks.