Multi agent causality

Game theory and decision theory for lots of interacting agents

2025-03-09 — 2025-03-09

Wherein agents’ decisions are modeled via a mechanised multi-agent influence diagram (MMAID) extending causal DAGs to iterated games, and the problem of commitment races is examined for AI safety.

adaptive
agents
AI safety
causal
cooperation
economics
evolution
extended self
game theory
graphical models
incentive mechanisms
learning
mind
networks
social graph
utility
wonk
Figure 1

Notes on decision theory and causality where agents make decisions, in the context of iterated games in multi-agent systems, with applications to AI safety.

Extending causal DAGs to include agents and decisions.

0.1 Multi-agent graphs

There seems to be a long series of works attempting this (Heckerman and Shachter 1994; Dawid 2002; Koller and Milch 2003). I am working from Hammond et al. (2023) and MacDermott, Everitt, and Belardinelli (2023), which introduce the One Ring that unifies them all in the form of something called a Mechanised Multi-Agent Influence Diagram, a.k.a. a MMAID.

cf Liu et al. (2024).

1 Commitment races

See commitment for a discussion of the commitment problem in the context of multi-agent systems.

2 References

Benford. 2010. “What Does Newcomb’s Paradox Teach Us?”
Brunet, and Doolittle. 2015. “Multilevel Selection Theory and the Evolutionary Functions of Transposable Elements.” Genome Biology and Evolution.
Cai, Daskalakis, and Weinberg. 2013. “Understanding Incentives: Mechanism Design Becomes Algorithm Design.” arXiv:1305.4002 [Cs].
Dawid. 2002. “Influence Diagrams for Causal Modelling and Inference.” International Statistical Review.
Fernández-Loría, and Provost. 2021. “Causal Decision Making and Causal Effect Estimation Are Not the Same… and Why It Matters.” arXiv:2104.04103 [Cs, Stat].
Fox, MacDermott, Hammond, et al. 2023. “On Imperfect Recall in Multi-Agent Influence Diagrams.” Electronic Proceedings in Theoretical Computer Science.
Hammond, Chan, Clifton, et al. 2025. “Multi-Agent Risks from Advanced AI.”
Hammond, Fox, Everitt, et al. 2023. “Reasoning about Causality in Games.” Artificial Intelligence.
Harley. 1981. “Learning the Evolutionarily Stable Strategy.” Journal of Theoretical Biology.
Heckerman, and Shachter. 1994. “A Decision-Based View of Causality.” In Proceedings of the Tenth International Conference on Uncertainty in Artificial Intelligence. UAI’94.
Hetzer, and Sornette. 2013. “An Evolutionary Model of Cooperation, Fairness and Altruistic Punishment in Public Good Games.” PLoS ONE.
Kalai, and Lehrer. 1993. “Rational Learning Leads to Nash Equilibrium.” Econometrica.
Koller, and Milch. 2003. “Multi-Agent Influence Diagrams for Representing and Solving Games.” Games and Economic Behavior, First World Congress of the Game Theory Society,.
Liu, Wang, Li, et al. 2024. “Attaining Human Desirable Outcomes in Human-AI Interaction via Structural Causal Games.”
MacDermott, Everitt, and Belardinelli. 2023. “Characterising Decision Theories with Mechanised Causal Graphs.”
MacDermott, Fox, Belardinelli, et al. 2024. “Measuring Goal-Directedness.”
Mu, Guo, Chen, et al. 2024. “Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas.”
Nowak. 2006. “Five Rules for the Evolution of Cooperation.” Science.
Sanders, Galla, and Shapiro. 2011. “Effects of Noise on Convergent Game Learning Dynamics.” arXiv:1109.4853.
Wolpert, and Benford. 2013. “The Lesson of Newcomb’s Paradox.” Synthese.