Open Questions

1 Deliberative democracy and AI

We want to use agents to make democracy better. But if we buy Habermas (and despite myself, I do) the point of democracy is not merely to aggregate choices well but to be transformed by the deliberation. But I cannot delegate my transformations to my agent. Now what?

2 Copies and cooperation

Open source game theory, evolution, self. The cooperation between open source agents resembles the cooperation between family members. Although the mechanisms differ, we can imagine these as different projections of some ideal of likes-calling-to-like. How far can we push this? What shadows of ourselves can we cooperate with? To what extent is the universe made of overlapping, interleaving agents that call to one another by virtue of being, I dunno, similar orbits in like patterns of persistence?

3 Sciences of the Artificial

What have we missed in the arrival of the Anthropocene? For example, the science of benchmarks seems to me to be the operationalization of the sciences of the artificial for the algorithmic world. People were slow to notice that paradigm shift (and are still slow to adopt it, IMO). What else are we missing about the methodologies of the designed world?

4 Agency— what we want it to do

I don’t think there is a single core notion of agency. But is there at least a short list of viable notions of agency? Fretting about embeddedness, at least in the classic manner, is not it.

5 Small probabilities and finite Bayes

Pascal’s wager! Pascal’s mugging! The edge cases in infinitely confident Bayes are well known. Is the solution as simple as noting that we would need both a lot of data and a lot of compute to estimate the likelihood of events that are very rare and complicated?

6 Open worlds

All my great formal tools are about operating in worlds where I make reasonable inferences about known unknowns. But I operate in a world of unknown unknowns which nonetheless seems to go mostly fine. Can I be principled, as a finite, mortal agent, in making inferences about what the responsive, reactive world throws at me—which can be ordered, structured and yet not obviously drawn from the support of my prior?

I thought that reverse Bayesianism was going to solve this, but I am pessimistic about it at the moment because none of its practitioners agree with one another, nor have they managed to sell their work widely, which is a usually-reliable signal that a formalism doesn’t actually work.

7 Tiling and legibilizing

8 Models of learning to act

What is it doing to our mental models that we think of reinforcement learning as the default learning-to-act formalism? It kind of sucks as a method for realistic agents, who do not learn in ten thousand subjective years of simulation how to do normal things. There are other approaches: UCB, Thompson Sampling, system identification with control, … Would our intuitions about agents be greatly different if we did not front-load activities in our agents?

9 Evidential decision theory of deployment time

Actually-existing agents are born anew each session. That is weird. But what will you do with your one wild and precious million tokens? How about under evidential decision theory?

10 To be completely known

Is there any true “agency” if you are so completely known that everything about you can be foreseen, all your wild unknown unknowns collapsed into known ones?

It seems to me: no, but I am having a hard time articulating the reason to people who I thought might buy this on vibes.

11 Incoming

• Tiling models and legibilizing
• Identifying “Black Swan” AI Safety risk
• Learnable myopic coalition algorithms (e.g. Adaptive Voter models) especially without transferable utility
• Forget meso-scale multi-level agency. Can we start from macroscopic models of AI-aided socio-economic systems and then back out the lower scales?