What are human values?
If we assume that humans are pursuing stuff, what is that stuff? Thick models of value, eudaimonic rationality,…
2025-06-05 — 2026-06-29
Wherein the Limits of Utility Maximisation Are Examined and Thick Models of Value Are Proposed as an Alternative Framework for Understanding Human Motivation and Collective Flourishing.
A placeholder. Let us relax the assumption that humans are best understood as acting to maximise their utility. How else can we understand what is good to have more of, from such open-ended intelligences as we?
cf ecology of mind.
1 Thick models of value
An interesting concept used instrumentally in delegated agent governance. For now, consider Zhi-Xuan et al. (2025), Klingefjord, Lowe, and Edelman (2024) and the manifesto Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value:
Full-Stack Alignment is a collaborative project led by the Meaning Alignment Institute and a small group of outside researchers. Our goal is to align AI and institutions with what people value, from each individual’s pursuit of their vision of the good life to the collective achievement of shared values and ideals. In other words, we want AI systems and institutions that fit human values and sociality well, where the AI systems and their institutions are compatible.
We argue that our current societal stack is misaligned in many places; we have markets that favor things that are addictive and isolating, and our democratic institutions are polarizing us. Things will get worse as AI displace workers entirely, outspeed regulation, and outmanouver lawyers. In our near-term future we risk sudden or gradual disempowerment as our economic and democratic agency erodes.
This challenge may seem insurmountable. We disagree, and argue this intractability stems from how we model what humans care about in our AI, markets and democracies—in our position paper, we summarize these approaches as “Preferentist models of value” (PMV) and “Values-as-text” (VAT). Both of these fail to capture the richness of human motivation. Consequently, a desire for “meaningful connection” becomes “engagement metrics” to recommender systems, which becomes “daily active users” to companies, and “quarterly revenue” in markets. Instead, we propose a new paradigm — “Thick models of value” (TMV).
2 Incoming
After Orthogonality: Virtue-Ethical Agency and AI Alignment
The concept of eudaimonia, I argue, suggests a form of rational activity without a strict distinction between means and ends, or between ‘instrumental’ and ‘terminal’ values. In this model of rational activity, a rational action is an element of a valued practice in roughly the same sense that a note is an element of a melody, a time-step is an element of a computation, and a moment in an organism’s cellular life is an element of that organism’s self-subsistence and self-development.[…]
My central claim is that our intuitions about the nature of human flourishing are implicitly intuitions that eudaimonic rationality can be functionally robust in a sense highly critical to AI alignment. More specifically, I argue that in light of our best intuitions about the nature of human flourishing it’s plausible that eudaimonic rationality is a natural form of agency, and that eudaimonic rationality is effective even by the light of certain consequentialist approximations of its values. I then argue that if our goal is to align AI in support of human flourishing, and if it is furthermore plausible that eudaimonic rationality is natural and efficacious, then many classical AI safety considerations and ‘paradoxes’ of AI alignment speak in favor of trying to instill AIs with eudaimonic rationality._