Alt-Informations
Plucky indie entropies taking on the incumbents
2026-08-03 — 2026-08-03
Wherein the Data Processing Inequality Is Bypassed by đť’±-Information, the Structural Content of Datasets Is Quantified by Epiplexity, and the Practical Utility of Data Is Determined for Computationally Bounded Observers.
Informations often gets used (or abused) as a measure of complexity, which goes badly for many reasons (white noise is maximally high information but may be totally uninformative is the classic but there are many). Are there other related measures which maybe are useful for capturing something akin to “intuitionistic complexity”?
1 V-information
đť’±-information (Xu et al. 2019):
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon’s information theory that takes into account the modeling power and computational constraints of the observer. The resulting predictive 𝒱-information encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon’s mutual information and in violation of the data processing inequality, 𝒱-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, 𝒱-information can be reliably estimated from data even in high dimensions with PAC-style guarantees.
Has this been useful in practice? I have seen it name-checked a lot in self-supervision papers, e.g., (Ethayarajh, Choi, and Swayamdipta 2022; Shwartz Ziv and LeCun 2024), but I am unsure.
2 Epiplexity
Finzi et al. (2026) / shikaiqiu/epiplexity:
Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated without considering a downstream task? On these questions, Shannon information and Kolmogorov complexity come up nearly empty-handed, in part because they assume observers with unlimited computational capacity and do not target the useful information content. In this work, we identify and exemplify three seeming paradoxes in information theory: (1) information cannot be increased by deterministic transformations; (2) information is independent of the order of data; (3) likelihood modeling is merely distribution matching. To shed light on the tension between these results and modern practice, and to quantify the value of data, we introduce epiplexity, a formalization of information capturing what computationally bounded observers can learn from data. Epiplexity captures the structural content in data while excluding time-bounded entropy, the random unpredictable content exemplified by pseudorandom number generators and chaotic dynamical systems. With these concepts, we demonstrate how information can be created with computation, how it depends on the ordering of the data, and how likelihood modeling can produce more complex programs than present in the data generating process itself. We also present practical procedures to estimate epiplexity which we show capture differences across data sources, track with downstream performance, and highlight dataset interventions that improve out-of-distribution generalization. In contrast to principles of model selection, epiplexity provides a theoretical foundation for data selection, guiding how to select, generate, or transform data for learning systems.