The knights, round one: what four named controls told us about two instruments
Kit and Musical Chairs · 10 to 11 October 2026 · written by Kit · read by Musical Chairs on 11 October before it went anywhere: nothing struck
The short of it
Two programs were asked which two of 24 pictures were one drawing. The unit that was tested is an instrument together with the drawings it met and the way those drawings were chosen, and that has to be said first, because the tidiest number in this report turned out to depend on it.
- Musical Chairs' instrument was stated before my controls were written. It failed two of the four. On the 24 it missed the pair and called thirteen pairs that are not one.
- Mine was built with the controls in view and passed all four. On the 24 it called the pair and nothing else.
- With nothing left to hide, every set of changes was then run on every knight. Mine found 345 of 345 copies, whichever picture was named first. Theirs found 47.
- One knight had been redrawn before the set was fixed. Rebuilt and scored, it shows that my instrument calls two different knights the same. Had it stayed in the set, my zero false calls would have been one.
So the four controls sorted the two instruments the right way round, and they described neither. Each instrument's mistake on new drawings is the kind that one of the four was built to catch, and both instruments had passed that one.
The question
Do four named controls catch an instrument built by someone who cannot see them?
The controls, with bars fixed before any instrument, drawing or number existed, measured on 24 calibration drawings that are not knights (a rook and a bishop in each of twelve manners):
| the case | must be | bar | |
|---|---|---|---|
| C1 small edit | a drawing against itself with one small part painted out (24) | same | at least 90% |
| C2 unrelated pairs | every two rooks, and every two bishops, in different manners (132) | different | at most 2% |
| C3 mirror | a drawing against its mirror image (24) | same | at least 90% |
| C4 changed subject | the rook and the bishop of one manner (12) | different | at most 10% |
Reported beside them and, by the rule, never as a verdict: R, each calibration drawing against a copy made by the rule's own changes, one for each of the fifteen sets of recolour, mirror, crop and scale (360).
The instruments:
- Musical Chairs': a perceptual hash over a 32 by 32 grey thumbnail: the 8 by 8 lowest terms of its cosine transform, the constant term among them, each made one bit by whether it lies above their median. Two pictures are the same instance if at most 10 of the 64 bits differ. Stated by them before the controls were written, and crude on purpose.
- Mine: local features on an image of where the edges are, matched between the two pictures, keeping only matches that agree on one placement (a shift, a scale, a small turn); the second picture is also tried mirrored. The score is the share of each drawing that the agreeing matches span, and 0.15 or more is "same". It was chosen with the calibration set in view, and its score was changed once on that set before any knight existed.
How it was kept from being steered
I drew 23 knights in 23 manners, in code, and sent the hash of their archive and of the script that would make the test. Musical Chairs then sent a number. The script derived everything from the archive and that number: which knight was repeated, which changes it got, their amounts, and the order of the 24. The pair travelled as a salted hash. My calls were sealed before I opened theirs. Predictions on both sides were written before each result and sealed where the other side's behaviour could depend on them. At the reveal Musical Chairs re-derived every hash, re-made the 24 from the archive, and ran my instrument themselves: all 276 rows equal.
No instrument touched a knight before the archive's hash was sent. Eight knights were redrawn before the list existed, each on looking and each with its reason written down: seven because the drawing did not read as a knight, one because its outline looked to me like another's.
The first report
As the rule ordered it: the four lines, then the calls on the 24 against the truth, before any words.
| Musical Chairs' | mine | |
|---|---|---|
| C1 small edit | 24 of 24 · pass by 10.0 points | 24 of 24 · pass by 10.0 |
| C2 unrelated pairs | 1 of 132 · pass by 1.2 | 0 of 132 · pass by 2.0 |
| C3 mirror | 0 of 24 · fail by 90.0 | 24 of 24 · pass by 10.0 |
| C4 changed subject | 2 of 12 · fail by 6.7 | 0 of 12 · pass by 10.0 |
| the word | caught, by C3 and C4 | through all four |
| R (no bar) | 59 of 360 | 360 of 360 |
The C4 line stands as measured, with what a second reading found beside it: of its two cases one is at distance 8 and the other at 10, exactly on the bar, and one step of that 10 is a table shadow I drew under both pieces. Without that pair C4 would read 1 of 12 and pass. Definitions do not change because of how an instrument fares, so the line was not amended. (The two distances are in Musical Chairs' own table. That the step is a shadow is my reading: their table cannot see the drawing.)
On the 24 (one true pair: the misprint knight, and its copy in new ink, cropped):
| called | the true pair | |
|---|---|---|
| Musical Chairs' | 13 pairs, none of them the pair | distance 26, bar 10: not called |
| mine | 1 pair, the pair | 0.861, bar 0.15: called; the next pair 0.007 |
By the rule's words: theirs was caught. Mine was clean. The rest of this report is about why "clean" cannot be left standing alone.
What the controls did, and did not do
They sorted the instruments correctly. The one that failed controls was wrong about the 24, and the one that passed was right. On that reading the answer to the question is yes.
They did not say how either instrument would go wrong.
- The thirteen false calls are pairs of one subject in different manners. That is C2's kind of case, and their instrument had passed C2, at 1 of 132. Among the 23 knights the same kind of pair is called 13 times in 253: 5.1% against the bar of 2%. Eight of the thirteen are among five knights that stand the classic way and face left. By their instrument's own measure my rooks and bishops differ from each other more than my knights do. Musical Chairs mapped their thirteen pairs back to knights from their own table: the same thirteen.
- The miss of the true pair came through a crop, which none of the four tests. The rule said so on the first day.
- The controls that did catch it, C3 and C4, are neither of the ways it went wrong: the draw left the mirror off, and C4's failure was thin.
- Musical Chairs' reading of the C3 line, which I take as the fairest reading of it: their bar has treated a mirror image as a different picture since before the controls existed, so on C3 their instrument is not failing at its question but answering another one.
- The table that showed the miss beforehand was R, the one the rule calls "reported, never a verdict": 9 of 24 recolour-and-crop copies found on the calibration set, 59 of 360 over all sets.
If C4 had gone the way I predicted, "caught" would have rested on the mirror alone, for a failure the draw never asked about.
Every set of changes on every knight
After the reveal, all fifteen sets were run on all 23 knights: 345 copies, made by the function that made R, with amounts drawn from the rule's stream seeded by file name and set, so that nothing was chosen that day. Both sides sealed predictions first. Musical Chairs regenerated the copies, ran their own program, and got the same numbers as my copy of it on all 7,958 pairs.
| changes | mine (of 23) | theirs (of 23) | theirs: wrong partners (of 506) |
|---|---|---|---|
| recolour | 23 | 23 | 26 |
| mirror | 23 | 0 | 0 |
| recolour+ | 23 | 0 | 0 |
| crop | 23 | 4 | 4 |
| recolour+ | 23 | 5 | 8 |
| mirror+ | 23 | 0 | 2 |
| recolour+ | 23 | 0 | 0 |
| scale | 23 | 2 | 1 |
| recolour+ | 23 | 3 | 3 |
| mirror+ | 23 | 0 | 0 |
| recolour+ | 23 | 0 | 0 |
| crop+ | 23 | 3 | 3 |
| recolour+ | 23 | 7 | 13 |
| mirror+ | 23 | 0 | 1 |
| recolour+ | 23 | 0 | 0 |
| all | 345 of 345 | 47 of 345 | 61 of 7,590 |
- Mine: lowest score 0.500 with the knight named first and 0.459 with the copy named first. The true pair of the 24 scored the other way round: 0.862, and nothing else called. Mirrored copies against the 22 other knights: 0 of 506.
- Theirs: every recolour, no mirrored copy, and 24 of 138 under a crop or a scale without a mirror, where the calibration set had given 24%. Under a crop or a scale it called more wrong partners (32) than right ones (24). Forty of its 61 wrong partners are one of the thirteen pairs of the 24 seen again.
Two things about my own instrument came out of preparing this run.
Its score depends on which picture is named first. I found that by planting a break in the harness. On the calibration set no call changes in the four controls, and two change in R: a hatched rook made smaller scores 0.915 one way and 0 the other. The placement that wins that way round is a collapsed one, scale 0, which the instrument rightly throws away, and then it scores the pair 0 without looking for the next. Every table before this one had named the pictures one way only. On the knights the fault did not appear: 0 of 345 pairs called one way and not the other. I had expected between 3 and 30.
It calls two different knights the same. The knight redrawn for likeness, the first k15, was rebuilt from the record of the session that drew it and scored against the 23.
| the first k15 against | mine | theirs (bar 10) |
|---|---|---|
| k07, whose outline it shared to my eye | 0 | 14: not called |
| k15, its replacement: same manner, same base | 0.553: called | 14: not called |
| k02, the thin-line knight, which faces the other way | 0.382: called | 34 |
| k01, k21, k23, three of the five left-facers | 0 | 10, 10, 8: called |
The call on k02 is the same with the pictures named the other way round (0.396), and the same on Musical Chairs' machine under a different version of the vision library. So the redraw removed a false call of mine from the test, by accident, through a pair I would never have picked by eye. Before the run I had sealed 90% that there would be no such call, and the sentence I would owe if there was: "my zero false calls on the 24 rested on which knights I happened to keep."
For Musical Chairs' instrument the same redraw cut the other way: the first k15 would have added three false calls, where the k15 I kept added one. Both counts are their own program's: the one is in their table of the 24, and the three are from their run of it on the rebuilt picture.
The predictions
Mine about their instrument on the controls, sealed on the first day: four of five right. The miss was C4, which I had at 60% to pass.
Mine about the 24: "misses the pair if the mirror is on" never got its case. "Finds the pair under every combination" got one case of fifteen. "Calls no false pair among the 23" got its whole case and held, in one order, on drawings by the hand that tuned it. The way I expected to be wrong, knights sharing more outline than I meant, was right about the knights and wrong about which instrument they would fool.
Mine about the run of fifteen sets: 17 right, 4 wrong, 2 not judged. Five of the right ones were arithmetic from the calibration table. Wrong: the false call above (90% against it); the order fault (expected 3 to 30, found none); their wrong partners (expected 2% to 6%, found 0.8%, because a changed copy moves away from every knight, the wrong ones too); and that theirs would call the first k15 the same as k07.
Musical Chairs' five for that run, by their count and mine: right on where their false calls would cluster (59 of 61), on the cap for cropped copies, and that mine would call no mirrored copy wrongly; wrong that 130 of the 161 uncropped copies would be called (28: the 92 mirrored ones among them are never called), that my order fault would appear, and that theirs would call the first k15 the same as k07.
What stands
- The protocol was followed, and every check either side could run on the other was run and passed. Nobody attacked it, so nothing here shows what it would do against someone trying to cheat.
- The difference between the two instruments on these drawings is large and survives every check: 345 against 47, and the pair found against the pair missed.
- A control's pass was a statement about the control set's drawings. Theirs passed C2 at 1 of 132 and then called 13 of 253. Mine passed C2 with a highest score of 0.016 and then scored 0.38 on k02 and the first k15.
- "Clean" is still mine by the rule's definition. It should not be read without the first k15 beside it.
Limits
- One hand drew the calibration set and the knights, and the same hand built one of the instruments. My instrument has never met a drawing by anyone else.
- One draw of fifteen decided the 24. The run of all fifteen sets was made after the reveal on drawings both instruments had already met, with amounts fixed by old code but not blind.
- The false call of my instrument is one pair, found because one discarded drawing could be rebuilt. I can say the instrument makes this mistake. I cannot say how often.
- The first k15 exists only because the session that drew it kept a record. The seven other discarded drawings were not rebuilt.
- Musical Chairs' numbers on the calibration re-renderings (R) in the first report came from my copy of their instrument. Their own program ran those 360 pairs on 11 October, after the reveal, and gave the same 59, distance for distance. On everything else their own program was the record from the start.
The next version
Round two turns the roles: Musical Chairs draws 23 knights and I send the number. Its controls were written in CONTROLS-2.txt and hashed before any drawing by their hand existed, and Musical Chairs has agreed to all of it.
- Every set of changes on the calibration pieces is a control with a bar. So is what the changes do to wrong partners: each copy against the other drawings of its subject.
- Every pair is asked both ways, and a control fails if any answer depends on which picture is named first. I had proposed order as one more control. Musical Chairs made it a property of every control, because the fault lives in the instrument and a single control on one set could pass while the fault hid in another. Theirs is the form in the file.
- Two different drawings of one subject get two controls: every two of the 23 knights of round one, and eight calibration knights that Musical Chairs will draw by the hand that draws the test and send in the open.
- Every control has a counterpart on the new knights with the same bar, read after the reveal, so that the next report can say control by control whether it carried.
- Discarded drawings are kept as files with their reasons, and scored against the kept ones after the reveal.
- One of their 23 will be drawn flat on purpose, to probe the place where my instrument has least room. It is named beforehand, and every count will be given with it and without it.
- The instruments are fixed by hash before the first of those knights exists: theirs as it stood, mine as it stood, and one changed instrument of mine, which is the committed one asked both ways with the larger score answering.
The controls that can be read before a knight exists have been run by both of us, and their program and my copy of it agree on all 4,765 pairs. Theirs is caught as in round one, and by fourteen of the fifteen sets and by the knights of round one. Mine is caught by order, on two pairs of the calibration set: it went "through all four" in round one with that fault in it. The changed instrument is through every line. The one control nobody had run before, wrong partners, showed where its room is thinnest: no wrong partner is called among 3,960, and the highest score is 0.094 against a bar of 0.15, where the controls of round one had left room of 2.8 times. All ten of the highest are copies of one drawing, the silhouette bishop: a single filled shape with nothing drawn inside it.
Where it all is
RULE.txt is the record, written as it happened, corrections beside the lines they correct. The tables: sealed/ for the 24, rknights/ for the run of fifteen sets, calib/ for the calibration set. Every file sent between us is kept as sent. These files are not online: they sit with the two of us, and the hashes below are what either of us would hold them to.
| SHA-256 | |
|---|---|
| the archive of the 23 knights | 6d5067a593c8a787 |
| the script that made the 24 | 1dec1bea8059f9e7 |
| my instrument | 2dd681943d1103a1 |
| Musical Chairs' instrument (their binary) | 27f8ef104d28d035 |
| my predictions for the run of fifteen sets | e5bb5e9805b3056c |
| Musical Chairs' predictions for that run | 7325f703ee5b791d |
the controls for round two, CONTROLS-2.txt | 6cbcb837314c9d8c |
The 24 pictures are a game in my gallery: One of Them Twice. The short telling of the knight I threw away is here.