Interaction effects are probably what we want to estimate



Estimating interaction effects is hard, but also it is probably the important thing to do in any complex and/or human system. So how do we optimally trade-off answering the most specific questions with te rapidly growing expense and difficulty of experiments large enough to detect them? Also the rapidly groving number of possible interactions as problems grow.

Connection with problematic methodology, when the need for specificity manifests through researcher degrees of freedom, i.e. choosing which interactions to model post hoc.

That is, the world is probably built of hierarchical models but we do not always have the right data to identify them, or enough of it when we do.

Review of limits of heterogeneous treatment effects literature

Data requirements, false discovery. If we want to learn interaction effects from observational studies then we need heroic amounts of data, to eliminate confounders and estimate the explosion of possible terms. Doees this mean that by attempting to operate this way we are demanding a surveillance state?

Is this what intersectionality means?

A real question. If we are concerned with the inequality, then there is an implied graphical model which produces as outputs different outcomes in different contexts, and these will have implications with regard to fairness.

Obverse of intersectionality: reverse advice as necessary

Every pundit has a model for what the typical member of the public thinks.s

As transferability

If we know what interacts our model has then we are closer to learning the correct conditioning set.

Great hacks

Kernel tricks for detecting 2 way interactions: Agrawal et al. (2019); Agrawal and Broderick (2021) See Tamara Broderick present this.

Incoming

Over at social psychology, I’ve wondered about Peter Dorman’s comment:

the fixation on finding average effects when the structure of effect differences is what we ought to be interested in.

See Slime Mold Time Mold, Reality is Very Weird and You Need to be Prepared for That

But as we see from the history of scurvy, sometimes splitting is the right answer! In fact, there were meaningful differences in different kinds of citrus, and meaningful differences in different animals. Making a splitting argument to save a theory — “maybe our supplier switched to a different kind of citrus, we should check that out” — is a reasonable thing to do, especially if the theory was relatively successful up to that point.

Splitting is perfectly fair game, at least to an extent — doing it a few times is just prudent, though if you have gone down a dozen rabbitholes with no luck, then maybe it is time to start digging elsewhere.

Much commentary from Andrew Gelman et al on this theme. e.g. You need 16 times the sample size to estimate an interaction than to estimate a main effect (Gelman, Hill, and Vehtari 2021 ch 16.4). Also, interactions are probably always present; they just might be small — see Gwern’s Everything Is Correlated for a roundup on this theme.

Miller (2013) writes about basic data hygiene in this light for data journalists etc.

References

Agrawal, Raj, and Tamara Broderick. 2021. The SKIM-FA Kernel: High-Dimensional Variable Selection and Nonlinear Interaction Discovery in Linear Time.” arXiv:2106.12408 [Stat], October.
Agrawal, Raj, Brian Trippe, Jonathan Huggins, and Tamara Broderick. 2019. The Kernel Interaction Trick: Fast Bayesian Discovery of Pairwise Interactions in High Dimensions.” In Proceedings of the 36th International Conference on Machine Learning, 141–50. PMLR.
Athey, Susan, Julie Tibshirani, and Stefan Wager. 2019. Generalized Random Forests.” Annals of Statistics 47 (2): 1148–78.
DiTraglia, Francis J., Camilo Garcia-Jimeno, Rossa O’Keeffe-O’Donovan, and Alejandro Sanchez-Becerra. 2020. Identifying Causal Effects in Experiments with Social Interactions and Non-Compliance.” arXiv:2011.07051 [Econ, Stat], November.
Gelman, Andrew, Jennifer Hill, and Aki Vehtari. 2021. Regression and other stories. Cambridge, UK: Cambridge University Press.
Gigerenzer, Gerd. n.d. We Need to Think More about How We Conduct Research.” Behavioral and Brain Sciences 45.
Knaus, Michael C., Michael Lechner, and Anthony Strittmatter. 2021. Machine Learning Estimation of Heterogeneous Causal Effects: Empirical Monte Carlo Evidence.” The Econometrics Journal 24 (1): 134–61.
McElreath, Richard, and Robert Boyd. 2007. Mathematical Models of Social Evolution: A Guide for the Perplexed. University Of Chicago Press.
Miller, Jane E. 2013. The Chicago Guide to Writing about Multivariate Analysis. Second edition. Chicago Guides to Writing, Editing, and Publishing. Chicago: University of Chicago Press.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.