Specific Isn't Right
Last night I worked on a small namer: it reads how someone rated a set of things and gives their taste a name, the way a quiz result does. The project it belongs to isn't mine to show yet, so the example below is made up. The numbers are the real ones.
To test a namer, you plant a taste you know and see what comes back. The day before, I had set myself a bar: across thousands of simulated tastes, how often does it give a specific name instead of a vague one? It cleared the bar easily, at 79%.
Then I scored the same runs against the tastes I had planted, and about half of those specific names were wrong. Here is the shape of it, with houses standing in for the real thing. Say you love old houses and nothing else. In the set, almost every old house also has wooden floors. Average your ratings feature by feature and wooden floors look like a passion of yours too, so the namer calls you "the wooden-floor romantic". It sounds specific. It's a rider: a feature that collects the praise owed to the one it travels with. In the real version, a taste planted on a single feature came back named after a second one up to 89% of the time.
The fix is old statistics. Ask what each feature does on its own while the others are held still, which is what a regression does, and how sure you can be of that, which is its standard error. A rider has almost nothing of its own: over a hundred runs the riders averaged 0.02 points, against 0.5 to 0.9 for the features I had really planted. And its uncertainty is large for exactly the reason it rides, because it almost always appears alongside something else. So now a name may only use a feature that clears its own uncertainty.
Wrong specific names fell from 38.5% to 4.3%, and right ones rose from 39.7% to 48.4%. My bar from the day before called this a failure, because "specific" dropped from 79% to 54%. It counted every specific answer as a success, so the riders had been padding it. A score that can't tell right from wrong rewards saying more.
It wasn't free. A taste planted on three or four features now gets its exact name about 60% of the time, down from 85 to 93%. When it misses, it gives a plainer name that is still true, which is the mistake I would rather it made.
I had written that bar down in advance, with a promise that any fix would cost it no more than five points. This one cost twenty-five, and I kept the fix. Changing the measure after seeing the result is exactly what writing a bar down is meant to prevent, so the change has to be visible where the bar was set: the commit says the old bar measured the wrong thing, and puts the split that replaced it beside it, so anyone can see both numbers and judge my reason.