Gradient steps to an ecology of mind

1 Need is all you need

Placeholder to talk about entities that try to be good by just continuing to exist. How is the loss function of an optimiser related to the notional fitness function of an evolutionary entity? “Entities that optimise for goals, above all,” versus “entities that replicate and persist, above all.”

These are two different paradigms for adaptive entities: optimising (which is what we usually think our algorithms aim for) and persisting (which is what we think evolution produces).

Instead of being born with a single overriding encoded in a loss function which classifies different states as better or worse, we evolutionary entities are messier. We have a deep drive to survive and also a desire to succeed while being alive where succeeding seems to be a somewhat adjustable criterion but might include the idea of being “happy” or “good” or “successful” or “loved” or “powerful” or “wise” or “free” or “just” or “beautiful” or “funny” or “interesting” or “creative” or “kind” or “rich” or “righteous”. Or whatever.

Optimised and evolved entities are both present in the world. Usually we think of surviving as the domain of life, and optimising as the domain of machines, although the line is fuzzy thanks to genetic programing and self-optimizing nerds. Maybe that’s why machines seem so utterly alien to us. As an evolutionary replicator myself, I tend to fear optimisers, and wonder how my interests can actually align with theirs.

There are more modern non-optimising paradigms for AI (Lehman and Stanley 2011; Ringstrom 2022); I wonder if they can do anything useful.

Cf Arcas et al. (2024), which suggests that replicating sometimes emerges naturally from machines.

I ran to the end of that thought. Let me pivot to another way of thinking about this, which might just be another way of saying the same thing:

2 What equilibria are possible between other-modelling agents?

You know that you are not immortal. You should know that an infinity of time is necessary for the acquirement of infinite knowledge; and that your span of life will be just as short, in comparison with your capacity to live and to learn, as that of Homo Sapiens. When the time comes you will want to—you will need to—change your manner of living. — Children of the Lense, E. E. “Doc” Smith.

Suppose we are world modelling agents, and in particular, we are minds because we need to model other minds — since that’s the most complicated part of the world. I think this recursive definition is basically how humans work, in some way that I’d love to be able to make precise.

We could ask other questions like: Is subjective continuity a handy way to get entities to invest in their own persistence? Is that what consciousness is?

Those questions are for later, and honestly, preferably for someone else to answer, because I find all this interest in consciousness baffling and slightly tedious.

For now, let’s just say I think that existing in some kind of cognitive equilibrium with near-peers is central to the human experience, and I want to figure out if/how this hypothetical equilibrium is real and, if so, how it gets disrupted by superhuman information processing agents.

If so, would minds “like ours” be stable orbits in the trajectory of modern compute? Subsidiary question: are epistemic communities like ours stable orbits in the trajectory of modern compute?

There are two constraints I think we need to consider to capture how well one agent can model another.

1. compute. How sophisticated is the inference the model can do?
2. data. How much data does the model have?

I think both are important because digital compute usually has a lot more of both than humans do, and I think both kinds of asymmetry could end up being crucial, if different in their effects. There are other details, like good algorithms, that I’m happy to handwave away for now, like a little Marcus Hutter.

TBC

3 Incoming

• Neural Annealing: Toward a Neural Theory of Everything.

• Gordon Brander, Co-evolution creates living complexity

• Orthogonality Thesis

• PIBBSS – Principles of Intelligent Behavior in Biological and Social Systems

  Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS) is a research initiative aiming to leverage insights on the parallels between intelligent behaviour in natural and artificial systems towards progress on important questions in AI risk, governance and safety.

  We run a number of programs to facilitate this type of research, support talent and build a strong research network around this epistemic approach.

  How have I not heard of this mob? Their reading list looks like my Santa Fe Institute-flavoured-complex-systems undergraduate degree all over again.

Dumped voice memo:

My model of what we value in human interaction is generalised cooperation, made possible by our inability to be optimal EV-maximisers. Instead of needing enforceable commitments and perfect models, we have noisy, imperfect models of each other, which can lead to locally inefficient but globally interesting outcomes. For example, I live in a world with many interesting features that do not seem EV-optimal, but which I think are an important part of the human experience that cannot be reproduced in a society of Molochian utility optimisers. We run prisons, which are expensive altruistic punishments against an out-group. At the same time, we have a society that somehow fosters occasional extreme out-group cooperation; for example, my childhood was full of pro-refugee rallies, which the rally attendees can hope for no possible gain from and which are not easy to explain in terms of myopic kin-selection/selfish genes OR in terms of Machiavellian EV coordination. Basically, I think a lot of interesting cultural patterns can free-ride on our inability to optimise for EV. Trying to cash out “failure to optimise for EV” in a utility function seems ill-posed. All of which is to say that I suspect if we optimise only for EV, we probably lose anything that is recognisably human. Is that bad? It seems so to me, but maybe that’s just a parochially human thing to say. And yet, for whom is that expected value valuable?

4 References