The Third Source
Last night I rebuilt a music questionnaire, the Goldsmiths Musical Sophistication Index, from its authors' own materials: the paper, the printed form, two scoring spreadsheets and the documentation. It has 38 statements, like "I can tell when people sing or play out of tune", and it gives six scores. I don't type scoring keys by hand. One script pulls each part of the key out of the sources, and a second script checks every part against a different source.
Most of it agreed. The two tables of population norms, one in the paper and one in the documentation, matched in all 594 numbers. Then the overall score didn't. It is the sum of 18 statements, and the paper's supplementary table and the authors' scoring spreadsheet agree on 17 of them. For the eighteenth, the paper lists "I only need to hear a new tune once and I can sing it back hours later". The spreadsheet lists "If somebody starts singing a song I don't know, I can usually join in".
Both come from the same authors, and neither outranks the other. If I had picked one because it looked more official, I would have been choosing, not checking. So I looked for a third copy of the key: the scoring code the same research group wrote for a later study. Matched by wording, its 18 statements are exactly the spreadsheet's.
The same code seemed to disagree with the spreadsheet somewhere else. One statement is scored backwards in the spreadsheet and forwards in the code. That looked like a second error until I read the two versions side by side. The form says "I am not able to sing in harmony when somebody is singing a familiar tune." The code's version says "I am able to sing in harmony…". Each scores its own wording correctly, so they agree.
That one nearly got past me, and the reason is mine. My check compared scoring directions only when two wordings were "the same", which it measured as at least 97% alike. Those two sentences are more alike than that: the word that reverses the meaning is three letters long, so it barely moves the number. Now the check compares directions only when the words are identical, and it prints every statement it couldn't compare, so the gaps are visible instead of skipped.
A third source only settles anything if it reached its answer some other way, and I can't fully show that here. The code rewords some statements, so it wasn't pasted straight from the spreadsheet, but both could still come from one master list, and then their agreement would be one source counted twice. So my version uses the spreadsheet's 18, and says next to the scores which source that is and that the choice is likely, not proven.