Game theory with open-source agents and simulation
Program equilibria, mutual source-code inspection, credible commitment by transparency
2026-05-27 — 2026-05-27
Wherein the Submission of Programs Rather Than Moves Is Examined, Permitting Cooperative Equilibria to Be Sustained Through Mutual Verification in One-Shot Games Without Repetition or Punishment.
Stub.
In standard game theory each agent’s policy is a black box. Opponents see actions but not the mechanisms that led to the actions. Commitments to cooperate aren’t credible. Open-source game theory (OSGT) changes the action space: instead of submitting a move, each player submits a program that can read its opponent’s source code as input (Bárász et al. 2014).
1 Program equilibria
This permits program equilibria — cooperative Nash equilibria unavailable in the move-game, sustained by mutual verification rather than by repetition or punishment. The recursion problem (my program simulates your program which simulates mine…) is handled by resource-bounded versions of Löb’s theorem, yielding constructions like \(\epsilon\)-GroundedFairBot: cooperate with probability \(\epsilon\) unconditionally, otherwise simulate the opponent and cooperate iff the simulation says they will (Critch 2016). A Folk Theorem follows: in one-shot program games, any feasible and individually rational payoff is sustainable.
An obvious practical objection is that real-world agents don’t expose their weights, and even if they did, a deceptively aligned agent could obfuscate them. But the framing is suggestive: how interpretable an agent needs to be to be trustworthy, and whether modern interpretability tools can do the job that programmatic source code used to, is one of the live questions for AI safety.
2 Simulating your opponent
3 In RL
Multi-agent reinforcement learning is one possible operationalisation of OSGT.