Planning under uncertainty

Transcending planning-though-hope-and-haruspicy



We talk a lot about risk management but what do we do at the outer boundaries of certainty, where statistics meets irreducible complexity?

Scott Alexander:

Gallant wouldn’t have waited for proof. He would have checked prediction markets and asked top experts for probabilistic judgments. If he heard numbers like 10 or 20 percent, he would have done a cost-benefit analysis and found that putting some tough measures into place, like quarantine and social distancing, would be worthwhile if they had a 10 or 20 percent chance of averting catastrophe.

Managing black swans

TBC. Nassim Taleb has a whole career based on handling heavy-tailed risk and managing out-of-sample downsides (Taleb 2007, 2020). This subsection probably needs a better name and a notebook of its own. Contrariness dictates I will not use Taleb’s terms

Statistical pragmatism

Ben Recht on what probability theory even means in application introduces David Freedman:

The dominant convention in experimental sciences is to assert the existence of a probability distribution from which all observations in an experiment are samples. That is, most analyses begin with the assumption that the population is itself a stochastic entity whose statistical properties can be accurately modeled. Such probabilistic modeling is immediately problematic. We are forced to confront the notion of probability itself. What does the probability of an event mean? Is a Bayesian or Frequentist viewpoint more correct? What does it mean to sample a natural process?

My perspective on the pitfalls of probabilistic modeling has been heavily influenced by the writings of David Freedman. Freedman advocates for a pragmatic approach to statistical modeling. “Probability is just a property of a mathematical model intended to describe some features of the natural world. For the model to be useful, it must be shown to be in good correspondence with the system it describes.”

There are some philosophical questions there. See also Effect size is significantly more important than statistical significance. – arg min blog.

References

Dawes, Robyn M., David Faust, and Paul E. Meehl. 1989. “Clinical Versus Actuarial Judgment.” Science, March. https://doi.org/10.1126/science.2648573.
Freedman, David A., and Philip B Stark. 2009. “What Is the Chance of an Earthquake?” In Statistical Models and Causal Inference: A Dialogue with the Social Sciences, edited by David Collier, Jasjeet S. Sekhon, and Philip B. Stark. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511815874.
Smith, Gordon C. S., and Jill P. Pell. 2003. “Parachute Use to Prevent Death and Major Trauma Related to Gravitational Challenge: Systematic Review of Randomised Controlled Trials.” BMJ 327 (7429): 1459–61. https://doi.org/10.1136/bmj.327.7429.1459.
Taleb, Nassim Nicholas. 2007. “Black Swans and the Domains of Statistics.” The American Statistician 61 (3): 198–200. https://doi.org/10.1198/000313007X219996.
———. 2020. “On the Statistical Differences Between Binary Forecasts and Real-World Payoffs.” International Journal of Forecasting, April. https://doi.org/10.1016/j.ijforecast.2019.12.004.

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.