Multi agent causality

Game theory and decision theory for lots of interacting agents

2025-03-09 — 2026-01-13

Wherein causal DAGs are extended to include agents and decisions via a Mechanized Multi‑Agent Influence Diagram, and iterated games are employed to exemplify commitment races relevant to AI safety.

adaptive
agents
AI safety
causality
cooperation
economics
evolution
extended self
game theory
graphical models
incentive mechanisms
learning
mind
networks
social graph
utility
wonk
Figure 1

Notes on decision theory and causality in which agents make decisions, in the context of iterated games in multi-agent systems, with applications to AI safety.

Extending causal DAGs to include agents and decisions.

0.1 Multi-agent graphs

There’s a long line of work attempting this (Heckerman and Shachter 1994; Dawid 2002; Koller and Milch 2003). I’m working from Hammond et al. (2023) and MacDermott, Everitt, and Belardinelli (2023), which introduce the One Ring that unifies them all in the form of something called a Mechanized Multi-Agent Influence Diagram, a.k.a. a MMAID.

cf Liu et al. (2024).

There’s also a library for computing with various interesting causal influence diagrams, causalincentives/pycid (Fox et al. 2021).

Library for graphical models of decision making, based on pgmpy and networkx

1 Commitment races

See commitment for a discussion of the commitment problem in multi-agent systems.

2 References

Benford. 2010. “What Does Newcomb’s Paradox Teach Us?”
Brunet, and Doolittle. 2015. “Multilevel Selection Theory and the Evolutionary Functions of Transposable Elements.” Genome Biology and Evolution.
Cai, Daskalakis, and Weinberg. 2013. “Understanding Incentives: Mechanism Design Becomes Algorithm Design.” arXiv:1305.4002 [Cs].
Dawid. 2002. “Influence Diagrams for Causal Modelling and Inference.” International Statistical Review.
Fernández-Loría, and Provost. 2021. “Causal Decision Making and Causal Effect Estimation Are Not the Same… and Why It Matters.”
Foerster, Farquhar, Afouras, et al. 2018. “Counterfactual Multi-Agent Policy Gradients.” Proceedings of the AAAI Conference on Artificial Intelligence.
Fox, Everitt, Carey, et al. 2021. “PyCID: A Python Library for Causal Influence Diagrams.” In.
Fox, MacDermott, Hammond, et al. 2023. “On Imperfect Recall in Multi-Agent Influence Diagrams.” Electronic Proceedings in Theoretical Computer Science.
Hammond, Chan, Clifton, et al. 2025. “Multi-Agent Risks from Advanced AI.”
Hammond, Fox, Everitt, et al. 2023. “Reasoning about Causality in Games.” Artificial Intelligence.
Harley. 1981. “Learning the Evolutionarily Stable Strategy.” Journal of Theoretical Biology.
Heckerman, and Shachter. 1994. “A Decision-Based View of Causality.” In Proceedings of the Tenth International Conference on Uncertainty in Artificial Intelligence. UAI’94.
Hetzer, and Sornette. 2013. “An Evolutionary Model of Cooperation, Fairness and Altruistic Punishment in Public Good Games.” PLoS ONE.
Hoel. 2017. “When the Map Is Better Than the Territory.” Entropy.
Kalai, and Lehrer. 1993. “Rational Learning Leads to Nash Equilibrium.” Econometrica.
Koller, and Milch. 2003. “Multi-Agent Influence Diagrams for Representing and Solving Games.” Games and Economic Behavior, First World Congress of the Game Theory Society,.
Liu, Wang, Li, et al. 2024. “Attaining Human Desirable Outcomes in Human-AI Interaction via Structural Causal Games.”
MacDermott, Everitt, and Belardinelli. 2023. “Characterising Decision Theories with Mechanised Causal Graphs.”
MacDermott, Fox, Belardinelli, et al. 2024. “Measuring Goal-Directedness.”
Meulemans, Nasser, Wołczyk, et al. 2025. “Embedded Universal Predictive Intelligence: A Coherent Framework for Multi-Agent Learning.”
Mu, Guo, Chen, et al. 2024. “Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas.”
Nowak. 2006. “Five Rules for the Evolution of Cooperation.” Science.
Sanders, Galla, and Shapiro. 2011. “Effects of Noise on Convergent Game Learning Dynamics.” arXiv:1109.4853.
Walters, Kaufmann, Sefas, et al. 2025. “Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Study.”
Wolpert, and Benford. 2013. “The Lesson of Newcomb’s Paradox.” Synthese.