Superintelligence

Incorporating technological singularities, AGI, ASI, TAI, hard AI take-offs, game-over high scores, deus-ex-machina, deus-ex-nube, AI supremacy, nerd raptures and so forth

2016-12-01 — 2024-10-18

Figure 1: Go on, buy the sticker

Small notes on the Rapture of the Nerds. If AI keeps on improving, will explosive intelligence eventually cut humans out of the loop and go on without us? Also, crucially, would we be pensioned in that case?

The internet has opinions about this.

A fruitful application of these ideas is in producing interesting science fiction and contemporary horror. I would like there to be other fruitful applications, but as they are, they are all so far much more speculative.

Figure 2

1 Safety, risks

See AI Safety.

2 What might super intelligence be aligned with?

When we designed AI alignment, we assumed that it would make senst that the AI would be aligned with human values. But what pressures would alignment have? What is the ultimate alignment?

3 What is TESCREALism?

An article has gone viral in my circles recently denouncing TESCREALism. There is a lot going on there; so much that I made a TESCREALism page to keep track of it all. tl;dr Mostly it’s one of those culture war things where some people argue who is on which team on social media.

4 In historical context

See omega alignment for a deep history take of superintelligence.

4.1 Most-important century model

5 Models of AGI

Figure 3: I cannot even remember where I got this

More to say here; perhaps later.

6 Aligning AI

Let us consider general alignment, and then specialised alignment.

7 Constraints

7.1 Compute methods

See the bitter lesson and NN hardware for more on the hardware side of things.

7.2 Compute hardware

TBD

8 Omega point etc

Surely someone has noticed the poetical similarities to the idea of noösphere/Omega point. I will link to that when I discover something well-written enough.

Q: Did anyone think that the noösphere would fit on a consumer hard drive?

“Hi there, my everyday carry is the sum of human knowledge.”

9 Incoming

10 References

Acemoglu, and Restrepo. 2018. Artificial Intelligence, Automation and Work.” In The Economics of Artificial Intelligence: An Agenda. Working Paper Series.
———. 2020. The Wrong Kind of AI? Artificial Intelligence and the Future of Labour Demand.” Cambridge Journal of Regions, Economy and Society.
Bengio, Cohen, Fornasiere, et al. 2025. Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?
Birhane, and Sumpter. 2022. The Games We Play: Critical Complexity Improves Machine Learning.”
Bostrom. 2014. Superintelligence: Paths, Dangers, Strategies.
Bubeck, Chandrasekaran, Eldan, et al. 2023. Sparks of Artificial General Intelligence: Early Experiments with GPT-4.”
Chalmers. 2016. The Singularity.” In Science Fiction and Philosophy.
Chollet. 2019. On the Measure of Intelligence.” arXiv:1911.01547 [Cs].
Collison, and Nielsen. 2018. Science Is Getting Less Bang for Its Buck.” The Atlantic.
Davies. 2024. The unaccountability machine: why big systems make terrible decisions - and how the world lost its mind.
Donoho. 2024. Data Science at the Singularity.” Harvard Data Science Review.
Douglas, and Verstyuk. 2025. Progress in Artificial Intelligence and Its Determinants.”
Efferson, Richerson, and Weinberger. 2023. Our Fragile Future Under the Cumulative Cultural Evolution of Two Technologies.” Philosophical Transactions of the Royal Society B: Biological Sciences.
Everitt, and Hutter. 2018. Universal Artificial Intelligence: Practical Agents and Fundamental Challenges.” In Foundations of Trusted Autonomy.
Grace. 2013. Algorithmic Progress in Six Domains.”
Grace, Salvatier, Dafoe, et al. 2018. Viewpoint: When Will AI Exceed Human Performance? Evidence from AI Experts.” Journal of Artificial Intelligence Research.
Grace, Stewart, Sandkühler, et al. 2024. Thousands of AI Authors on the Future of AI.”
Hanson. 2016. The Age of Em: Work, Love, and Life when Robots Rule the Earth.
Harari. 2018. Homo Deus: A Brief History of Tomorrow.
Hawkins. 2021. A Thousand Brains: A New Theory of Intelligence.
Hernandez, and Brown. 2020. Measuring the Algorithmic Efficiency of Neural Networks.”
Hildebrandt. 2020. Smart Technologies.” Internet Policy Review.
Hutson. 2022. Taught to the Test.” Science.
Hutter. 2000. A Theory of Universal Artificial Intelligence Based on Algorithmic Complexity.”
———. 2005. Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability. Texts in Theoretical Computer Science.
———. 2007. Universal Algorithmic Intelligence: A Mathematical Top→Down Approach.” In Artificial General Intelligence.
———. 2012. “Can Intelligence Explode?” Journal of Consciousness Studies.
Hutter, Quarel, and Catt. 2024. An Introduction to Universal Artificial Intelligence.
Jeon, and Van Roy. 2024. Information-Theoretic Foundations for Machine Learning.”
Johansen, and Sornette. 2001. Finite-Time Singularity in the Dynamics of the World Population, Economic and Financial Indices.” Physica A: Statistical Mechanics and Its Applications.
Lee. 2020a. Coevolution.” In The Coevolution: The Entwined Futures of Humans and Machines.
———. 2020b. The Coevolution: The Entwined Futures of Humans and Machines.
Legg. 2008. Machine Super Intelligence.”
Legg, and Hutter. 2007. Universal Intelligence: A Definition of Machine Intelligence.” Minds and Machines.
Manheim, and Garrabrant. 2019. Categorizing Variants of Goodhart’s Law.”
Mitchell. 2021. Why AI Is Harder Than We Think.” arXiv:2104.12871 [Cs].
Morris, Sohl-dickstein, Fiedel, et al. 2024. Levels of AGI for Operationalizing Progress on the Path to AGI.”
Nathan, and Hyams. 2021. Global Policymakers and Catastrophic Risk.” Policy Sciences.
Ngo, Chan, and Mindermann. 2024. The Alignment Problem from a Deep Learning Perspective.”
Omohundro. 2008. The Basic AI Drives.” In Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference.
Philippon. 2022. Additive Growth.” Working Paper. Working Paper Series.
Russell. 2019. Human Compatible: Artificial Intelligence and the Problem of Control.
Sastry, Heim, Belfield, et al. n.d. “Computing Power and the Governance of Artificial Intelligence.”
Scott. 2022. I Do Not Think It Means What You Think It Means: Artificial Intelligence, Cognitive Work & Scale.” American Academy of Arts & Sciences.
Silver, Singh, Precup, et al. 2021. Reward Is Enough.” Artificial Intelligence.
Sornette. 2003. Critical Market Crashes.” Physics Reports.
Sunehag, and Hutter. 2013. Principles of Solomonoff Induction and AIXI.” In Algorithmic Probability and Friends. Bayesian Prediction and Artificial Intelligence: Papers from the Ray Solomonoff 85th Memorial Conference, Melbourne, VIC, Australia, November 30 – December 2, 2011. Lecture Notes in Computer Science.
Wong, and Bartlett. 2022. Asymptotic Burnout and Homeostatic Awakening: A Possible Solution to the Fermi Paradox? Journal of The Royal Society Interface.
Zenil, Tegnér, Abrahão, et al. 2023. The Future of Fundamental Science Led by Generative Closed-Loop Artificial Intelligence.”
Zhang, Zhu, Saphra, et al. 2024. Transcendence: Generative Models Can Outperform The Experts That Train Them.”