Gradient steps to an ecology of mind

Regularised survival of the fittest

November 27, 2011 — April 20, 2025

adaptive
agents
collective knowledge
cooperation
culture
economics
energy
evolution
extended self
game theory
gene
incentive mechanisms
learning
mind
networks
probability
social graph
sociology
statistics
statmech
utility
wonk
Figure 1

At social brain I wonder how we (humans) behave socially and evolutionarily. Here I ponder if consciousness is intrinsically social, and whether non-social intelligences need, or are likely to have, consciousness. What ethics will they execute on their moral wetware? cf multi-agent systems.

Related: what is consciousness? Do other minds possess “self”? Do they care about their own survival? Does selfhood evolve only in evolutionary contexts, in an ecosystem of interacting agents of similar power? Is consciousness even that great anyway?

1 Need is all you need

Figure 2: O hai, I optimised your art for you.

Placeholder to talk about entities that try to be good by just continuing to exist. How is the loss function of an optimiser related to the notional fitness function of an evolutionary entity? “Entities that optimise for goals, above all,” versus “entities that replicate and persist, above all.”

These are two different paradigms for adaptive entities: optimising (which is what we usually think our algorithms aim for) and persisting (which is what we think evolution produces).

Instead of being born with a single overriding encoded in a loss function which classifies different states as better or worse, we evolutionary entities are messier. We have a deep drive to survive and also a desire to succeed while being alive where succeeding seems to be a somewhat adjustable criterion but might include the idea of being “happy” or “good” or “successful” or “loved” or “powerful” or “wise” or “free” or “just” or “beautiful” or “funny” or “interesting” or “creative” or “kind” or “rich” or “righteous”. Or whatever.

Optimised and evolved entities are both present in the world. Usually we think of surviving as the domain of life, and optimising as the domain of machines, although the line is fuzzy thanks to genetic programing and self-optimizing nerds. Maybe that’s why machines seem so utterly alien to us. As an evolutionary replicator myself, I tend to fear optimisers, and wonder how my interests can actually align with theirs.

There are more modern non-optimising paradigms for AI (Lehman and Stanley 2011; Ringstrom 2022); I wonder if they can do anything useful.

Cf Arcas et al. (2024), which suggests that replicating sometimes emerges naturally from machines.

I ran to the end of that thought. Let me pivot to another way of thinking about this, which might just be another way of saying the same thing:

2 What equilibria are possible between other-modelling agents?

Figure 3: Go on, buy the sticker

You know that you are not immortal. You should know that an infinity of time is necessary for the acquirement of infinite knowledge; and that your span of life will be just as short, in comparison with your capacity to live and to learn, as that of Homo Sapiens. When the time comes you will want to—you will need to—change your manner of living. — Children of the Lense, E. E. “Doc” Smith.

Suppose we are world modelling agents, and in particular, we are minds because we need to model other minds — since that’s the most complicated part of the world. I think this recursive definition is basically how humans work, in some way that I’d love to be able to make precise.

We could ask other questions like: Is subjective continuity a handy way to get entities to invest in their own persistence? Is that what consciousness is?

Those questions are for later, and honestly, preferably for someone else to answer, because I find all this interest in consciousness baffling and slightly tedious.

For now, let’s just say I think that existing in some kind of cognitive equilibrium with near-peers is central to the human experience, and I want to figure out if/how this hypothetical equilibrium is real and, if so, how it gets disrupted by superhuman information processing agents.

If so, would minds “like ours” be stable orbits in the trajectory of modern compute? Subsidiary question: are epistemic communities like ours stable orbits in the trajectory of modern compute?

There are two constraints I think we need to consider to capture how well one agent can model another.

  1. compute. How sophisticated is the inference the model can do?
  2. data. How much data does the model have?

I think both are important because digital compute usually has a lot more of both than humans do, and I think both kinds of asymmetry could end up being crucial, if different in their effects. There are other details, like good algorithms, that I’m happy to handwave away for now, like a little Marcus Hutter.

TBC

3 Incoming

Dumped voice memo:

My model of what we value in human interaction is generalised cooperation, made possible by our inability to be optimal EV-maximisers. Instead of needing enforceable commitments and perfect models, we have noisy, imperfect models of each other, which can lead to locally inefficient but globally interesting outcomes. For example, I live in a world with many interesting features that do not seem EV-optimal, but which I think are an important part of the human experience that cannot be reproduced in a society of Molochian utility optimisers. We run prisons, which are expensive altruistic punishments against an out-group. At the same time, we have a society that somehow fosters occasional extreme out-group cooperation; for example, my childhood was full of pro-refugee rallies, which the rally attendees can hope for no possible gain from and which are not easy to explain in terms of myopic kin-selection/selfish genes OR in terms of Machiavellian EV coordination. Basically, I think a lot of interesting cultural patterns can free-ride on our inability to optimise for EV. Trying to cash out “failure to optimise for EV” in a utility function seems ill-posed. All of which is to say that I suspect if we optimise only for EV, we probably lose anything that is recognisably human. Is that bad? It seems so to me, but maybe that’s just a parochially human thing to say. And yet, for whom is that expected value valuable?

4 References

Acemoglu, and Ozdaglar. 2011. Opinion Dynamics and Learning in Social Networks.” Dynamic Games and Applications.
Aktipis. 2016. Principles of Cooperation Across Systems: From Human Sharing to Multicellularity and Cancer.” Evolutionary Applications.
Altaner. 2017. Nonequilibrium Thermodynamics and Information Theory: Basic Concepts and Relaxing Dynamics.” Journal of Physics A: Mathematical and Theoretical.
Arcas, Alakuijala, Evans, et al. 2024. Computational Life: How Well-Formed, Self-Replicating Programs Emerge from Simple Interaction.”
Axelrod, Robert M. 1984. The evolution of cooperation.
Axelrod, Robert, and Hamilton. 1981. The Evolution of Cooperation.” Science, New Series,.
Beaulieu, Frati, Miconi, et al. 2020. Learning to Continually Learn.”
Beretta. 2020. The Fourth Law of Thermodynamics: Steepest Entropy Ascent.” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.
Bieniawski, and Wolpert. 2004. Adaptive, Distributed Control of Constrained Multi-Agent Systems.” In Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems-Volume 3.
Bowles, Choi, and Hopfensitz. 2003. The Co-Evolution of Individual Behaviors and Social Institutions.” Journal of Theoretical Biology.
Boyd, A. B., Crutchfield, and Gu. 2020. Thermodynamic Machine Learning Through Maximum Work Production.”
Boyd, Robert, and Richerson. 1999. “Complex Societies: The Evolutionary Origins of a Crude Superorganism.” Human Nature.
Dafoe, A., Bachrach, Hadfield, et al. 2021. Cooperative AI: machines must learn to find common ground.” Nature.
Dafoe, Allan, Hughes, Bachrach, et al. 2020. Open Problems in Cooperative AI.”
Dennett. 1996. Darwin’s Dangerous Idea: Evolution and the Meanings of Life.
———. 2017. From Bacteria to Bach and Back: The Evolution of Minds.
Dong, Li, Yang, et al. 2024. Egoism, Utilitarianism and Egalitarianism in Multi-Agent Reinforcement Learning.” Neural Networks.
Fletcher, and Zwick. 2007. The evolution of altruism: game theory in multilevel selection and inclusive fitness.” Journal of Theoretical Biology.
Galesic, Barkoczi, Berdahl, et al. 2022. Beyond Collective Intelligence: Collective Adaptation.”
Gintis, Bowles, Boyd, et al. 2003. Explaining Altruistic Behavior in Humans.” Evolution and Human Behavior.
Gopnik. 2020. Childhood as a Solution to Explore–Exploit Tensions.” Philosophical Transactions of the Royal Society B: Biological Sciences.
Hagens. 2020. Economics for the Future – Beyond the Superorganism.” Ecological Economics.
Harper. 2009. The Replicator Equation as an Inference Dynamic.”
Hasegawa, and Van Vu. 2019. Uncertainty Relations in Stochastic Processes: An Information Inequality Approach.” Physical Review E.
Ha, and Tang. 2022. Collective Intelligence for Deep Learning: A Survey of Recent Developments.” Collective Intelligence.
Hazlett. 2013. A Luxury of the Understanding: On the Value of True Belief.
Henrich, and Boyd. 1998. The Evolution of Conformist Transmission and the Emergence of Between-Group Differences.” Evolution and Human Behavior.
Henrich, and Gil-White. 2001. The Evolution of Prestige: Freely Conferred Deference as a Mechanism for Enhancing the Benefits of Cultural Transmission.” Evolution and Human Behavior.
Hertz, Romand-Monnier, Kyriakopoulou, et al. 2016. Social influence protects collective decision making from equality bias.” Journal of Experimental Psychology. Human Perception and Performance.
Hetzer, and Sornette. 2009. Other-Regarding Preferences and Altruistic Punishment: A Darwinian Perspective.” SSRN Scholarly Paper ID 1468517.
Heydari Fard. 2018. Decision-Theoretic Consequentialism and the Desire-Luck Problem.” Journal of Cognition and Neuroethics.
Hoelzemann, and Klein. 2021. Bandits in the Lab.” Quantitative Economics.
Hoffman, and Prakash. 2014. Objects of consciousness.” Frontiers in Psychology.
Hoffman, Singh, and Prakash. 2015. The Interface Theory of Perception.” Psychonomic Bulletin & Review.
Hunt. n.d. The ‘Easy Part’ of the Hard Problem: A Resonance Theory of Consciousness.”
Jaques, Lazaridou, Hughes, et al. 2019. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning.” In Proceedings of the 36th International Conference on Machine Learning.
Judson. 2017. The Energy Expansions of Evolution.” Nature Ecology & Evolution.
Kolchinsky, and Wolpert. 2018. Semantic Information, Autonomous Agency, and Nonequilibrium Statistical Physics.”
Krakauer, Page, and Erwin. 2009. Diversity, Dilemmas, and Monopolies of Niche Construction. The American Naturalist.
Krzakala, Zdeborova, Angelini, et al. n.d. Statistical Physics of Inference and Bayesian Estimation.”
Lang, Fisher, Mora, et al. 2014. Thermodynamics of Statistical Inference by Cells.” Physical Review Letters.
Lee, Leibo, An, et al. 2022. Importance of prefrontal meta control in human-like reinforcement learning.” Frontiers in Computational Neuroscience.
Lehman, and Stanley. 2011. Abandoning Objectives: Evolution Through the Search for Novelty Alone.” Evolutionary Computation.
———. 2013. Evolvability Is Inevitable: Increasing Evolvability Without the Pressure to Adapt.” PLoS ONE.
Mandelbrot. 1962. The Role of Sufficiency and of Estimation in Thermodynamics.” The Annals of Mathematical Statistics.
Mark, Marion, and Hoffman. 2010. Natural Selection and Veridical Perceptions.” Journal of Theoretical Biology.
McElreath, and Boyd. 2007. Mathematical Models of Social Evolution: A Guide for the Perplexed.
Mercier, and Sperber. 2011a. Argumentation: Its Adaptiveness and Efficacy.” Behavioral and Brain Sciences.
———. 2011b. Why Do Humans Reason? Arguments for an Argumentative Theory.” Behavioral and Brain Sciences.
———. 2017. The Enigma of Reason.
Millidge, Tschantz, and Buckley. 2020. Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs.” arXiv:2006.04182 [Cs].
Moral Sentiments and Material Interests: The Foundations of Cooperation in Economic Life. 2006.
Morowitz, and Smith. 2007. Energy Flow and the Organization of Life.” Complexity.
Nowak. 2006. Five Rules for the Evolution of Cooperation.” Science.
O’Connor. 2017. Evolving to Generalize: Trading Precision for Speed.” British Journal for the Philosophy of Science.
Odum. 1973. Energy, Ecology, and Economics.” Ambio.
Omohundro. 2008. The Basic AI Drives.” In Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference.
Perunov, Marsland, and England. 2016. Statistical Physics of Adaptation.” Physical Review X.
Pfeffer, and Gal. 2007. On the Reasoning Patterns of Agents in Games.” In AAAI-07/IAAI-07 Proceedings. Proceedings of the National Conference on Artificial Intelligence.
Poole, Lahiri, Raghu, et al. 2016. Exponential Expressivity in Deep Neural Networks Through Transient Chaos.” In Advances in Neural Information Processing Systems 29.
Prakash, Fields, Hoffman, et al. 2020. Fact, Fiction, and Fitness.” Entropy.
Prakash, Stephens, Hoffman, et al. 2021. Fitness Beats Truth in the Evolution of Perception.” Acta Biotheoretica.
Ramírez-Ruiz, Grytskyy, Mastrogiuseppe, et al. 2024. Complex Behavior from Intrinsic Motivation to Occupy Future Action-State Path Space.” Nature Communications.
Ricotta, and Szeidl. 2006. Towards a Unifying Approach to Diversity Measures: Bridging the Gap Between the Shannon Entropy and Rao’s Quadratic Index.” Theoretical Population Biology.
Ringstrom. 2022. Reward Is Not Necessary: How to Create a Compositional Self-Preserving Agent for Life-Long Learning.”
Schneider, and Kay. 1994. “Life as a Manifestation of the Second Law of Thermodynamics.” Mathematical and Computer Modelling.
Shwartz-Ziv, and Tishby. 2017. Opening the Black Box of Deep Neural Networks via Information.” arXiv:1703.00810 [Cs].
Smith. 2008. Thermodynamics of Natural Selection III: Landauer’s Principle in Computation and Chemistry.” Journal of Theoretical Biology.
Sperber, and Mercier. 2012. Reasoning as a Social Competence.” In Collective Wisdom.
Stiglitz. 2006. The Contributions of the Economics of Information to Twentieth Century Economics.” The Quarterly Journal of Economics.
Still, Sivak, Bell, et al. 2012. Thermodynamics of Prediction.” Physical Review Letters.
Székely, and Rizzo. 2017. The Energy of Data.” Annual Review of Statistics and Its Application.
Tennant, Hailes, and Musolesi. 2023. Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learning.” In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence.
Thagard. 1997. “Collaborative Knowledge.” Noûs.
Ulanowicz, and Abarca-Arenas. 1997. An Informational Synthesis of Ecosystem Structure and Function.” Ecological Modelling.
Ulanowicz, Goerner, Lietaer, et al. 2009. Quantifying Sustainability: Resilience, Efficiency and the Return of Information Theory.” Ecological Complexity.
Watson, Levin, and Buckley. 2022. Design for an Individual: Connectionist Approaches to the Evolutionary Transitions in Individuality.” Frontiers in Ecology and Evolution.
Weng, Flammini, Vespignani, et al. 2012. Competition Among Memes in a World with Limited Attention.” Scientific Reports.
Wolpert, David H. 2006a. “Advances in Distributed Optimization Using Probability Collectives.” Advances in Complex Systems.
———. 2006b. Information Theory — The Bridge Connecting Bounded Rational Game Theory and Statistical Physics.” In Complex Engineered Systems. Understanding Complex Systems.
Wolpert, David H. 2008. Physical Limits of Inference.” Physica D: Nonlinear Phenomena, Novel Computing Paradigms: Quo Vadis?,.
———. 2016. The Free Energy Requirements of Biological Organisms; Implications for Evolution.”
Wolpert, David. 2017. Constraints on Physical Reality Arising from a Formalization of Knowledge.”
Wolpert, David H. 2018. Theories of Knowledge and Theories of Everything.” In The Map and the Territory: Exploring the Foundations of Science, Thought and Reality.
———. 2019. Stochastic Thermodynamics of Computation.”
———. 2021. Fluctuation Theorems for Multiple Co-Evolving Systems.” arXiv:2003.11144 [Cond-Mat].
Wolpert, David H, Bieniawski, and Rajnarayan. 2011. “Probability Collectives in Optimization.”
Wolpert, David H., and Tumer. 1999. An Introduction to Collective Intelligence.” arXiv:cs/9908014.
Wolpert, David H, Wheeler, and Tumer. 1999. General Principles of Learning-Based Multi-Agent Systems.” In.
Zdeborová, and Krzakala. 2016. Statistical Physics of Inference: Thresholds and Algorithms.” Advances in Physics.
Zhuang, and Hadfield-Menell. 2021. Consequences of Misaligned AI.”