On one hand we hope that journals will help us find things that are relevant. On the other hand, we would hope the things they help us find are actually true. It’s not at all obvious how to solve these kind of classification problems economically, but we kind of hope that peer review does it.
Keywords: “file-drawer process” and the “publication sieve”, which are the large-scale models of how this works in a scientific community and “researcher degrees of freedom” which is the model for how this works at the individual scale.
This is particularly pertinent in social psychology, where it turns out the there is too much bullshit with \(P\leq 0.05\).
Sanjay Srivastava, Everything is fucked, the syllabus.
On the easier problem of local theories
On the other hand, we can all agree that finding small-effect universal laws in messy domains like human society is a hard problem. In machine learning we frequently give up on that and just try to solve a local problem — does this work in this domain with enough certainty to help this problem? Then we still need to solve a problem about domain adaptation when we try to work out if we are still working on this problem, or at least one similar enough to this. But that feels like it might be easier by virtue of being less ambitious.