This article raises some very important questions and makes some great points! Proper feedback loops are crucial.
What would it mean if it were shown Magnus Carlsen wasn't actually very good at chess?
It's an absurd case. But if true, it would undermine not just your opinion of Carlsen, but the whole system: ratings, rankings, tournaments, everything.
Absurd for chess, but Alvaro argues social science is such a case, because the feedback system has been decoupled from true results, allowing PhDs to so totally misunderstand a paper as to say N=59 is a universal constant for sample size. Or for Brian Wansink to continue to receive thousands of citations after being forced to resign for massive fraud.
And he promises a follow-up tomorrow, with more thoughts on evaluation & feedback.
Speaking of which, we should be releasing our preliminary results for DARPA SCORE this month. Advance warning: looks like we didn't do as well as previously.