The language game

1 Semantics is solved you guys

Have you had a talk with philosophers recently? Philosophers be all like

• Do machines have free will?
• Are machines conscious?
• Can I see the same colours as a mantis shrimp?

Great news, guys, questions like this are all totally solved! In case you missed the memo, I will vouchsafe it to you here: the canonical solution to semantics.

Collectively, speakers of a given language have learned a mapping between words and their referents, by observing a lot of phenomena and talking about them. If you will forgive me bulldozing some interesting fine detail, I assert that a good enough model for the phenomena is a large fixed data set which comprises more-or-less everyday human experience (smells, tastes, scenes, proprioceptive twinges). Now, obviously there is a lot going on when we learn to map from sensory experience to words. That is a whole thing which I will not attempt to solve here, but, you know, we humans clearly do pretty great at it, as do machines these days.

We approximate this process of learning words as a classifier.

Write \(\mathcal{X}\) for the space of phenomena and let each phenomenon have a frequency of occurrence in the world, \(p(x)\). We use \(V\) for a finite vocabulary to describe these. Not all things have names, and not all names are useful, so we will assume that \(|V|\) is much smaller than \(|\mathcal{X}|\), and “I dunno” is a valid label for a phenomenon. Suppose some stimuli occur often enough and are useful enough that humans name them. Everyday human experience is a joint distribution \(p(x, w)\) over \(\mathcal{X}\times V\), the propensity of a competent speaker subject to stimulus \(x\) to assign label \(w\) to it. \(w\) means \(x\) if \(p(w\mid x)\) is high.

In practice, a learner sees a finite sample \(\mathcal{D}=\{(x_i,w_i)\}_{i=1}^{n}\) of (stimulus, word) pairs, and fits a tractable model \(\hat p_{\mathcal{D}}(w\mid x)\) to it — the member of some family \(\mathcal{F}\) of candidate classifiers \(q(w\mid x)\) that maximizes the likelihood of the classifications of words actually heard,

\[\hat p_{\mathcal{D}} \;=\; \arg\max_{q\in\mathcal{F}}\;\sum_{(x_i,w_i)\in\mathcal{D}} \log q(w_i \mid x_i).\]

The meaning of a given word is the set of phenomena that are most likely to be assigned that word by the learned model,

\[[\![\,w\,]\!]_{\mathcal{D}} \;=\; \bigl\{\, x\in\mathcal{X} \;:\; w=\arg\max_{w'\in V}\,\hat p_{\mathcal{D}}(w'\mid x) \,\bigr\}.\]

Imagine you and I come from the same town and the main language spoken in the town is English. Your corpus \(\mathcal{D}'\) and mine \(\mathcal{D}\) are not identical, but a shared town and language means both are samples from almost the same \(p(x,w)\). My meaning for a given word is the region \([\![\,w\,]\!]_{\mathcal{D}}\) defined above; yours is \([\![\,w\,]\!]_{\mathcal{D}’}\), i.e. the same up to a difference in the training data we each sample from the world. We understand each other about \(w\) to the extent those two regions coincide, and we disagree on the phenomena that fall in one region but not the other, depending on the idiosyncrasies of our respective corpora.

Aggregating over the vocabulary, our miscommunication rate is the probability that we assign different labels to the same phenomenon,¹

\[\Pr_{x\sim p(x)}\!\left[\,\hat w_{\mathcal{D}}(x)\neq\hat w_{\mathcal{D}'}(x)\,\right],\qquad \hat w_{\mathcal{D}}(x)=\arg\max_{w'\in V}\hat p_{\mathcal{D}}(w'\mid x).\]

Since \(\mathcal{D}\) and \(\mathcal{D}'\) are drawn from the same \(p(x,w)\), that probability is small: we typically assign the same words to the same phenomena. The disagreements that remain are the rare phenomena, where neither corpus held enough examples to settle which word applies, or maybe where the usage itself is ambiguous. So I might think a whale is not a fish because it is a mammal, and you might think it is a fish because it lives in the water.

Generally our vocabularies nearly agree because our corpora nearly do, since we learned words from similar experiences by living in the same time and place.

OK, I didn’t say what nameable “phenomena” were. In practice, labelling phenomena in the world is a social and sensory process. The label is something we usually agree on, and the phenomenon itself is something we can readily perceive, like recognizing that “furry things that like sardines and purr” are “cats”. In machine learning, phenomena like “fur” and “purr” are features. Insofar as they co-occur often enough in the world, and we care to discuss them with other speakers of our language, we might label their co-occurrence in English as cat. If nature is kind to us, then there might be some latent cause that explains the co-occurrence of those features.

Now, cheetahs have fur and can purr. Are they truly cats?

The canonical answer is this is an underspecified question.

The we learned the word “cat” without reference to cheetahs its scope is simply undefined when we learn the world contains also cheetahs. We then get to choose whether to extend the word to cover cheetahs, or to invent a new word for them, or to, like, invent a phylogenetic justification for whichever label we ultimately choose.

Words are undefined outside their training distribution. We can give them particular meanings by extending them to new phenomena, but we cannot discover their meanings. No-one is a realist about words.

I am getting this printed on cards for emergency escapes from any debate about what “true” agency is, or real free will, or whether consciousness is real, or whether the same colours are seen by mantis shrimp and humans.

Now I have a hard one for you:

• Does a dog have Buddha nature?

2 Formalizations

When do we need to use words? What words can be shared between people? (Baronchelli et al. 2010; Loreto, Mukherjee, and Tria 2012) present a toy model for colour words — a clever choice of domain with something like a continuously indexed ground truth, 5 stars.

(Steyvers and Tenenbaum 2005) draws a connection to categorical stochastics and evolutionary models.

Eisenstat (2025) proposes an information-theoretic model of meaning learning as “shared latents” in the probabilistic ML sense. I think of this as a convergent abstraction argument.

3 Incoming

4 References