Multivariate information decomposition
2026-01-06 — 2026-01-06
Wherein the mutual information held by multiple sources about a target is shown to be partitioned into redundant, unique, and synergistic atoms via a lattice and Möbius inversion, while the original minimum‑information redundancy is critiqued.
Placeholder for a particular trick in information theory:
Williams and Beer (2010) introduced partial information decomposition (PID) as a way to split the mutual information that a set of sources has about a target into non‑negative “atoms” corresponding to redundant, unique, and synergistic information. Their framework has become a standard reference point, but their specific redundancy measure has been heavily critiqued. There are now many alternative proposals and generalizations.
1 Williams & Beer’s original PID
Williams & Beer consider a target \(Y\) and sources \(X_1, X_2, \dots\), and aim to decompose \(I(X_1, X_2, \dots : Y)\) into atoms corresponding to:
Redundant information (shared by multiple sources about \(Y\))
Unique information (available only from one source)
Synergistic information (available only from sources in combination).
They formalize this by:
Introducing a redundancy function \(I_{\cap}(X_{1:n} : Y)\) defined on sets of sources.
Requiring this redundancy measure to satisfy three axioms: symmetry, self‑redundancy, and monotonicity.
Using the lattice of “information antichains” over source subsets, they show that, given such a redundancy measure, the mutual information can be decomposed by Moebius inversion over this lattice, yielding guaranteed non‑negative information atoms.
As a concrete proposal, they define redundancy via the “minimum information” measure \(I_{\min}\), which roughly takes the minimal specific information any source provides about each outcome of \(Y\), averaged over outcomes.
2 Subsequent theoretical developments
While the framework (axioms + lattice) was widely accepted, the specific Williams–Beer redundancy \(I_{\min}\) quickly attracted criticism.
Much subsequent work has tried to preserve the Williams–Beer axioms and lattice structure while replacing \(I_{\min}\) with better‑behaved redundancy or intersection‑information measures.