Spamularity

Dark forest, Zombie internet, cheapfakes, textpocalypse

2015-10-05 — 2025-06-12

adversarial
AI safety
bounded compute
communicating
cooperation
culture
economics
faster pussycat
language
machine learning
mind
neural nets
NLP
security
technology
wonk
Figure 1

What happens when we run on an attention economy and during an AI arms race?

1 The Spamularity

Charlie Stross’s 2010 Spamularity stuck with me:

We are currently in the early days of an arms race, between the spammers and the authors of spam filters. The spammers are writing software to generate personalized, individualized wrappers for their advertising payloads that masquerade as legitimate communications. The spam cops are writing filters that automate the process of distinguishing a genuinely interesting human communication from the random effusions of a ’bot. And with each iteration, the spam gets more subtly targeted, and the spam filters get better at distinguishing human beings from software, […]

We have one faction that is attempting to write software that can generate messages that can pass a Turing test, and another faction that is attempting to write software that can administer an ad-hoc Turing test. Each faction has a strong incentive to beat the other. This is the classic pattern of an evolutionary predator/prey arms race: and so I deduce that if symbol-handling, linguistic artificial intelligence is possible at all, we are on course for a very odd destination indeed—the Spamularity, in which those curious lumps of communicating meat give rise to a meta-sphere of discourse dominated by parasitic viral payloads pretending to be meat…

Sam Kriss calls it the language of god:

What is machine language? Firstly, machine language is vampiric, shamanic, xenophagic, mocking. It’s a changeling. Often it tries to imitate human discourse; the machine wants you to think that it’s human. This is the first level of deception. Often this isn’t enough: machines will use various methods to take over other text-producing systems, so that without your knowledge you end up advertising weight loss pills to all your old school friends. First axiom: all language has the potential to become machine language. To become infected. 10 Award-Winning GIFs That Will Leave You Wanting More. I Could Watch #4 For Days. This is the second level of deception. In the third level of deception, the machine convinces itself that it has a physically extended body, that it has an independent mind, that it really wants to produce the text it generates. This might happen very soon. It might have already happened, somewhere on a dusty plain in western Africa, somewhere that never really existed, tens of thousands of years ago.

Sam Kriss is a smartarse though.

2 Dark Forest

Figure 2

The Dark Forest theory of the Internet is a metaphor for the current state of the web, where genuine human interactionis increasingly obscured by bots, advertisers, and other automated entities. It suggests that the web has become a “dark forest” where humans must hide to avoid being overwhelmed by these entities. The term dark forest is a shout out to Liu Cixin.

I’m not sure of the origin, but here are the think pieces I saw it in:

  • Yancey Strickler, Dark Forest theory of the Internet

  • Maggie Appleton’s commentary, 1, 2 in support of the cozy web movement:

    The dark forest theory of the web points to the increasingly life-like but life-less state of being online. Most open and publicly available spaces on the web are overrun with bots, advertisers, trolls, data scrapers, clickbait, keyword-stuffing “content creators,” and algorithmically manipulated junk.

    It’s like a dark forest that seems eerily devoid of human life—all the living creatures are hidden beneath the ground or up in trees. If they reveal themselves, they risk being attacked by automated predators.

    Humans who want to engage in informal, unoptimised, personal interactions have to hide in closed spaces like invite-only Slack channels, Discord groups, email newsletters, small-scale blogs, and digital gardens. Or make themselves illegible and algorithmically incoherent in public venues.

  • against the dark forest

I feel like I’m going to lose this battle, but for the record, I do not love the term “textpocalypse”.

3 Scraper arms races

What if in the internet scraper race all the tarpits and text generators we created ended up training AIs to simulate Markov text generators? That is one possible outcome of the current battle, if not the most liekly then certainly the funniest.

The resulting Markov garbage is soothing IMO I am developing quite the parasocial relationship with it.

4 Scams

Persuasion of vulnerable people at scale. Also a business model here. See AI persuasion for more on individual persuasion.

In The Economics of Spam, Bruce Schneier argues, based on Kanich et al. (), that spam is probably optimized for traffic, not conversion, i.e. quantity over quality, at least at that time. I suspect that to be truer of email spam at the time, than of Dark Forest Internet of today.

5 Bot or not?

Who is people? Let’s hope there is a verifiable identity solution to this.

6 Paths out of the dark forest

Public sphere business models??

7 References

Akerlof, and Shiller. 2015. Phishing for Phools: The Economics of Manipulation and Deception.
Bradshaw, and Howard. 2017. Troops, Trolls and Troublemakers: A Global Inventory of Organized Social Media Manipulation.”
Jaidka, Chen, Chesterman, et al. 2024. Misinformation, Disinformation, and Generative AI: Implications for Perception and Policy.” Digit. Gov.: Res. Pract.
Kanich, Kreibich, Levchenko, et al. 2008. Spamalytics: An Empirical Analysis of Spam Marketing Conversion.” In Proceedings of the 15th ACM Conference on Computer and Communications Security. CCS ’08.
Marwick, and Lewis. 2017. Media Manipulation and Disinformation Online.”
Rao, and Reiley. 2012. The Economics of Spam.” Journal of Economic Perspectives.
Swartz, Marwick, and Larson. 2025. ScamGPT: GenAI and the Automation of Fraud.