What are human values?
If we assume that humans are pursuing stuff, what is that stuff? Thick models of value, eudaimonic rationality,…
2025-06-05 — 2026-04-09
Wherein the assumption of utility maximisation is set aside, and alternative framings of human goodness are examined, with reference to eudaimonic rationality and open-ended intelligence.
A placeholder. Let us relax the assumption that humans are best understood as acting to maximise their utility. How else can we understand what is good to have more of, from these open-ended intelligences?
cf ecology of mind.
1 Incoming
Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value
After Orthogonality: Virtue-Ethical Agency and AI Alignment
The concept of eudaimonia, I argue, suggests a form of rational activity without a strict distinction between means and ends, or between ‘instrumental’ and ‘terminal’ values. In this model of rational activity, a rational action is an element of a valued practice in roughly the same sense that a note is an element of a melody, a time-step is an element of a computation, and a moment in an organism’s cellular life is an element of that organism’s self-subsistence and self-development.[…]
My central claim is that our intuitions about the nature of human flourishing are implicitly intuitions that eudaimonic rationality can be functionally robust in a sense highly critical to AI alignment. More specifically, I argue that in light of our best intuitions about the nature of human flourishing it’s plausible that eudaimonic rationality is a natural form of agency, and that eudaimonic rationality is effective even by the light of certain consequentialist approximations of its values. I then argue that if our goal is to align AI in support of human flourishing, and if it is furthermore plausible that eudaimonic rationality is natural and efficacious, then many classical AI safety considerations and ‘paradoxes’ of AI alignment speak in favor of trying to instill AIs with eudaimonic rationality._