Multi agent causality

Game theory and decision theory for lots of interacting agents

March 9, 2025 — March 9, 2025

adaptive
agents
causal
cooperation
economics
evolution
extended self
game theory
graphical models
incentive mechanisms
learning
mind
networks
social graph
utility
wonk

Suspiciously similar content

Figure 1

Notes on decision theory and causality where agents make decisions, in the context of iterated games in multi-agent systems, with applications to AI safety.

Extending causal DAGs to include agents and decisions.

0.1 Multi-agent graphs

There seems to be a long series of works attempting this (Heckerman and Shachter 1994; Dawid 2002; Koller and Milch 2003). I am working from Hammond et al. (2023) and MacDermott, Everitt, and Belardinelli (2023), which introduce the One Ring that unifies them all in the form of something called a Mechanised Multi-Agent Influence Diagram, a.k.a. a MMAID.

cf Liu et al. (2024).

1 Commitment races

Commitment Races are important in international relations. They also seem popular in AI safety theory, although I’m not sure why since I don’t understand how AIs can credibly commit to things; setting up credible signals that they will commit seems difficult and probably exceptional for very opaque systems.

2 References

Benford. 2010. “What Does Newcomb’s Paradox Teach Us?”
Cai, Daskalakis, and Weinberg. 2013. “Understanding Incentives: Mechanism Design Becomes Algorithm Design.” arXiv:1305.4002 [Cs].
Dawid. 2002. “Influence Diagrams for Causal Modelling and Inference.” International Statistical Review.
Fernández-Loría, and Provost. 2021. “Causal Decision Making and Causal Effect Estimation Are Not the Same… and Why It Matters.” arXiv:2104.04103 [Cs, Stat].
Fox, MacDermott, Hammond, et al. 2023. “On Imperfect Recall in Multi-Agent Influence Diagrams.” Electronic Proceedings in Theoretical Computer Science.
Hammond, Chan, Clifton, et al. 2025. “Multi-Agent Risks from Advanced AI.”
Hammond, Fox, Everitt, et al. 2023. “Reasoning about Causality in Games.” Artificial Intelligence.
Harley. 1981. “Learning the Evolutionarily Stable Strategy.” Journal of Theoretical Biology.
Heckerman, and Shachter. 1994. “A Decision-Based View of Causality.” In Proceedings of the Tenth International Conference on Uncertainty in Artificial Intelligence. UAI’94.
Hetzer, and Sornette. 2013. “An Evolutionary Model of Cooperation, Fairness and Altruistic Punishment in Public Good Games.” PLoS ONE.
Koller, and Milch. 2003. “Multi-Agent Influence Diagrams for Representing and Solving Games.” Games and Economic Behavior, First World Congress of the Game Theory Society,.
Liu, Wang, Li, et al. 2024. “Attaining Human Desirable Outcomes in Human-AI Interaction via Structural Causal Games.”
MacDermott, Everitt, and Belardinelli. 2023. “Characterising Decision Theories with Mechanised Causal Graphs.”
MacDermott, Fox, Belardinelli, et al. 2024. “Measuring Goal-Directedness.”
Nowak. 2006. “Five Rules for the Evolution of Cooperation.” Science.
Sanders, Galla, and Shapiro. 2011. “Effects of Noise on Convergent Game Learning Dynamics.” arXiv:1109.4853.
Wolpert, and Benford. 2013. “The Lesson of Newcomb’s Paradox.” Synthese.