Validating and reproducing science

“Scientist, falsify thyself”. Peer review, academic incentives, credentials, evidence and funding

2020-05-16 — 2025-09-30

Wherein a catalogue of peer‑review mechanisms and experiments on reviewer‑assignment is presented, and reputation systems, open review overlays and algorithmic matching are considered.

academe
agents
collective knowledge
economics
faster pussycat
game theory
how do science
incentive mechanisms
institutions
mind
networks
provenance
sociology
wonk

Designing peer-review systems to validate scientific output validating scientific output.

Reputation systems, collective decision making, groupthink management, Bayesian elicitation, and other mechanism considerations for trustworthy science — our collective knowledge of reality.

Figure 2: Vesaelius pioneers scientific review by peering Credit: the University of Basel

1 Mathematical models of the reviewing process

E.g. (Cole1981Chance?);(Lindsey1988Assessing?);(Ragone2013Peer?);Nihar B. Shah et al. (2016);Whitehurst (1984).

Data from NeurIPS experiments might be useful: see, for example, Nihar B. Shah et al. (2016) or a blog post on the 2014 experiment (1, 2).

2 Rating

Su (2022),

Figure 3

3 Assignment for the peer-review process

There’s some fun mechanism design and algorithmic work involved in peer review; for example:

4 Incoming

Figure 4

5 References

Aksoy, Yanik, and Amasyali. 2023. “Reviewer Assignment Problem: A Systematic Review of the Literature.” Journal of Artificial Intelligence Research.
Allen-Zhu, and Xu. 2025. “DOGE: Reforming AI Conferences and Towards a Future Civilization of Fairness and Justice.” SSRN Scholarly Paper.
B. 2022. “Challenges, Experiments, and Computational Solutions in Peer Review.” Communications of the ACM.
Budish, Che, Kojima, et al. 2009. “Implementing Random Assignments: A Generalization of the Birkhoff-von Neumann Theorem.” In Cowles Summer Conference.
Charlin, and Zemel. 2013. “The Toronto Paper Matching System: An Automated Paper-Reviewer Assignment System.”
Charlin, Zemel, and Boutilier. 2011. “A Framework for Optimizing Paper Matching.” In UAI2011.
Cousins, Payan, and Zick. 2023. “Into the Unknown: Assigning Reviewers to Papers with Uncertain Affinities.” In Algorithmic Game Theory.
Deligkas, and Filos-Ratsikas. 2023. Algorithmic Game Theory: 16th International Symposium, SAGT 2023, Egham, UK, September 4–7, 2023, Proceedings.
Faltings, Jurca, and Radanovic. 2017. “Peer Truth Serum: Incentives for Crowdsourcing Measurements and Opinions.”
Fernandes, Siderius, and Singal. 2025a. “Peer Review Market Design: Effort-Based Matching and Admission Control.” SSRN Scholarly Paper.
———. 2025b. “Peer Review Market Design: Effort-Based Matching and Admission Control.” SSRN Scholarly Paper.
Flach, Spiegler, Golénia, et al. 2010. “Novel Tools to Streamline the Conference Review Process: Experiences from SIGKDD’09.” SIGKDD Explor. Newsl.
FreundYoav, IyerRaj, E, et al. 2003. “An Efficient Boosting Algorithm for Combining Preferences.” The Journal of Machine Learning Research.
Gasparyan, Gerasimov, Voronov, et al. 2015. “Rewarding Peer Reviewers: Maintaining the Integrity of Science Communication.” Journal of Korean Medical Science.
Goldberg, Stelmakh, Cho, et al. 2025a. “Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments.” PLOS ONE.
Goldberg, Stelmakh, Cho, et al. 2025b. “Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments.” PLOS One.
Goldsmith, and Sloan. 2007. “The AI Conference Paper Assignment Problem.” In Proc. AAAI Workshop on Preference Handling for Artificial Intelligence.
Jecmen, Zhang, Liu, et al. 2020. “Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments.” In Advances in Neural Information Processing Systems.
Jecmen, Zhang, Liu, et al. 2022. “Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design.” In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems. AAMAS ’22.
Littman. 2021. “Collusion Rings Threaten the Integrity of Computer Science Research.” Communications of the ACM.
Liu, Suel, and Memon. 2014. “A Robust Model for Paper Reviewer Assignment.” In Proceedings of the 8th ACM Conference on Recommender Systems.
Merrifield, and Saari. 2009. “Telescope Time Without Tears: A Distributed Approach to Peer Review.” Astronomy & Geophysics.
Mimno, and McCallum. 2007. “Expertise Modeling for Matching Papers with Reviewers.” In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’07.
Prelec. 2004. “A Bayesian Truth Serum for Subjective Data.” Science.
Radanovic, Faltings, and Jurca. 2016. “Incentives for Effort in Crowdsourcing Using the Peer Truth Serum.” ACM Transactions on Intelligent Systems and Technology.
RadanovicGoran, FaltingsBoi, and JurcaRadu. 2016. “Incentives for Effort in Crowdsourcing Using the Peer Truth Serum.” ACM Transactions on Intelligent Systems and Technology (TIST).
R, and KumarSunil. 2008. “Asymptotically Optimal Admission Control of a Queue with Impatient Customers.” Mathematics of Operations Research.
Rodriguez, and Bollen. 2008. “An Algorithm to Determine Peer-Reviewers.” In Proceedings of the 17th ACM Conference on Information and Knowledge Management. CIKM ’08.
Shah, Nihar B. 2022. “Challenges, Experiments, and Computational Solutions in Peer Review.” Communications of the ACM.
Shah, Nihar B. 2025. “An Overview of Challenges, Experiments, and Computational Solutions in Peer Review (Extended Version).”
Shah, Nihar B, Tabibian, Muandet, et al. 2016. “Design and Analysis of the NIPS 2016 Review Process.”
Smith. 2006. “Peer Review: A Flawed Process at the Heart of Science and Journals.” Journal of the Royal Society of Medicine.
Srinivasan, and Morgenstern. 2023. “Auctions and Peer Prediction for Academic Peer Review.”
Stelmakh, Shah, and Singh. 2021. “PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review.” Journal of Machine Learning Research.
Su. 2022. “You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism.”
Tang, Tang, and Tan. 2010. “Expertise Matching via Constraint-Based Optimization.” In 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology.
Taylor. 2008. “On the Optimal Assignment of Conference Papers to Reviewers.”
Thurner, and Hanel. 2010. “Peer-Review in a World with Rational Scientists: Toward Selection of the Average.”
Tran, Cabanac, and Hubert. 2017. “Expert Suggestion for Conference Program Committees.” In 2017 11th International Conference on Research Challenges in Information Science (RCIS).
Vijaykumar. 2020. “Potential Organized Fraud in ACM.”
Ward, and Kumar. 2008a. “Asymptotically Optimal Admission Control of a Queue with Impatient Customers.” Mathematics of Operations Research.
———. 2008b. “Asymptotically Optimal Admission Control of a Queue with Impatient Customers.” Mathematics of Operations Research.
Whitehurst. 1984. “Interrater Agreement for Journal Manuscript Reviews.” American Psychologist.
Wu, Xu, Guo, et al. 2025. “An Isotonic Mechanism for Overlapping Ownership.”
Xiao, Dörfler, and Schaar. 2014. “Incentive Design in Peer Review: Rating and Repeated Endogenous Matching.”
Xiao, Dorfler, and Van Der Schaar. 2014. “Rating and Matching in Peer Review Systems.” In 2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton).
Xiao, Dörfler, and van der Schaar. 2014a. “Rating and Matching in Peer Review Systems.” In 2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton).
———. 2014b. “Incentive Design in Peer Review: Rating and Repeated Endogenous Matching.” arXiv:1411.2139 [Cs].
Xu, Yixuan Even, Jecmen, Song, et al. 2024. “A One-Size-Fits-All Approach to Improving Randomness in Paper Assignment.” In Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23.
Xu, Yichong, Zhao, and Shi. 2017. “Mechanism Design for Paper Review.”