# External validity

When does what I learn on one data set apply to another?

October 17, 2020 — February 16, 2023

For me it seems natural to consider learning well-factored causal graphical models containing the necessary interaction effects as the platonic ideal here and everything else is just an approximation to that.

Although maybe I should be thinking about feedback effects also — if everyone uses my algorithms, does this change the environment in which my algorithm is operating? For e.g. traffic routing algorithms the answer is clearly yes.

The reason this is hot topic in neural nets, I suspect, is that it is convenient for massive, low-human-effort neural networks to ignore graphical structure to get predictively good results from regressions in observational data by ignoring that structure, and this leads us into strife when the situation changes. In the IRM section there is a spicy take by Ermin Orhan that reframes this to say: The problem is *you* if you don’t have so much data that you can integrate out all such difficulties.

To recover the causal consistency in a black-box model is even more tedious than a classical one. Also, it fits the social conventions of neural network research to reinvent methods to fix such problems without reference to previous conventions, for better and worse.

One thing that the machine learning set up gives us which is an additional emphasis: external validity, the *statistical* framing, would ask you whether the model you have learnt is still useful on new data. The transfer learning set up invites use to consider if we can *transfer* some of the *computational effort* from learning on one data set to learning on new dataset, and if so, how much. Maybe that is a useful insight?

This connects also to semi-supervised learning and fairness, argues (Schölkopf et al. 2012; Schölkopf 2022).

Possibly the same underlying idea, we could argue that interaction effects are probably what we want to learn.

## 1 Standard graphical models

We can just try some basic graphical model technology and see how far we get. If the right independences are enforced, presumably we are doing something not too far from learning a transferable model? Or, if we work out that the necessary parameters are not identifiable, then we discover that we cannot in fact learn a transferable model, right? (But maybe we can learn a somewhat transferable model?) I guess the key weakness is that graphical models will miss some types of transferability, specifically, independences that are dependent on particular values of the nodes, so this might be less powerful.

## 2 External validity in policy

See also anthropic principles, science for policy.

I have lots of ideas about policy for the world and I think that some of the ideas are good because of some mix of scientific research and personal experience.^{1} So let us suppose that I am broadly sympathetic to some policy instrument (state ownership of power utilities? diversity quotas in hiring? etc) because I have seen them work in the past. The question is, *how universally should I be in favour of that policy?* How do I find out *what* are the circumstances that make these policy instruments achieve my desired outcomes? A recent example from a workplace I was in: Presumably a diversity quota requiring a certain percentage of the workforce be, say, women, would be pointless in a society with perfect gender equality, and ineffectual in a society which has failed to train any women at all with the required skills. Most societies will not be at either of those extremes, but what is the range of gender inequity where the hiring quotas *would* be a useful policy intervention? What other predictors will change their effectiveness? This policy is not a good idea *in and of itself* but rather in a *particular context*. Burying that essential context is common in debates observationally.

Rather than universal policy prescriptions, it is worth wondering *what specificity policies have* and constantly checking if they apply *here*.

## 3 Learning under covariate shift

See Covariate shift.

## 4 Incoming

## 5 References

*Journal of Causal Inference*.

*Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1*. NIPS’14.

*Proceedings of the National Academy of Sciences*.

*arXiv:1611.06221 [Cs, Stat]*.

*Statistical Science*.

*arXiv:2011.03395 [Cs, Stat]*.

*Annual Review of Statistics and Its Application*.

*arXiv:2104.04103 [Cs, Stat]*.

*Behavioral and Brain Sciences*.

*Frontiers in Applied Mathematics and Statistics*.

*Advances in Neural Information Processing Systems 30*.

*arXiv:2012.07421 [Cs]*.

*Proceedings of the National Academy of Sciences*.

*2018 IEEE Data Science Workshop (DSW)*.

*Frontiers in Big Data*.

*Statistical Science*.

*Elements of Causal Inference: Foundations and Learning Algorithms*. Adaptive Computation and Machine Learning Series.

*Dataset Shift in Machine Learning*.

*arXiv:2106.02589 [Math, Stat]*.

*arXiv:1801.06229 [Stat]*.

*Uncertainty in Artificial Intelligence*.

*Nature Communications*.

*Probabilistic and Causal Inference: The Works of Judea Pearl*.

*arXiv:1505.03036 [Astro-Ph, Stat]*.

*ICML 2012*.

*Journal of Economic Methodology*.

*Proceedings of the 33rd International Conference on Neural Information Processing Systems*.

*The 22nd International Conference on Artificial Intelligence and Statistics*.

*Advances in Neural Information Processing Systems*.

*arXiv:2109.03795 [Cs, Stat]*.

## Footnotes

Although I realistically copied some ideas from my acquaintances, but maybe even those ideas have the same sort of empirical basis. Let us optimistically assume so for now 🤞.↩︎