Probably actually reading/writing

Triage

• CXAI 2023
• Etalumis: Scientific simulators as probabilistic programs - Oxford AI4Science Lab

Currently writing

Not all published yet, expect broken links.

1. Human superorganisms

2. Movement design

3. Returns on hierarchy

4. Effective collectivism

5. alignment

6. Emancipating my tribe and the cruelty of collectivism (and why I love it anyway)

7. Institutions for angels

8. Beliefs and rituals of tribes,

9. Where to deploy taboo

10. The Great Society

11. Something about the fungibility of hipness and cash

12. What even are GFlownets?

13. Power, and measuring difficult things

    • Explode on Impact by Toby Lowe

      TL:DR — It is impossible for organisations to “demonstrate their impact” if they work in complex environments. Asking them to do so requires them to create a fantasy version of the story of their work. This corruption of data makes doing genuine change work harder because it is difficult to learn and adapt from corrupted data.

    • Decision Theory Remains Neglected - by Robin Hanson

      I say that their motives are more political: execs and their allies gain more by using other more flexible decision making frameworks for key decisions, frameworks with more wiggle room to help them justify whatever decision happens to favor them politically. Decision theory, in contrast, threatens to more strongly recommend a particular hard-to-predict decision in each case. As execs gain when the orgs under them are more efficient, they don’t mind decision theory being used down there. But they don’t want it up at their level and above, for decisions that say if they and their allies win or lose.

• Misidentifying talent

1. The Human learner series

2. Our moral wetware

3. Superstimuli

4. Clickbait bandits

5. ~~Comfort traps ~~ ✅ Good enough for now

6. Myths ✅ a few notes is enough

7. Classification and society series

   1. Affirming the consequent, and thermodynamic tribalism.
   2. Classifications are not very informative
   3. Adversarial categorization
   4. AUC and collateral damage
   5. bias and base rates
   6. Decision rules

8. Shouting at each other on the internet series (Teleological liberalism)

9. The Activist and decoupling games, and game-changing.

10. lived evidence deductions and/or ad hominem for discussing genetic arguments.

11. diffusion of responsibility — is this distinct from messenger shooting?

12. Iterative game theory of communication styles

13. Invasive arguments

14. Coalition games

15. All We Need Is Hate

16. Speech standards

17. Startup justice warriors/move fast and cancel things

18. Pluralism ✅

19. Epistemic community design

20. Scientific community

21. Messenger shooting

22. Experimental ethics and surveillance

23. Steps to an ecology of mind

24. DIY and the feast of fools

25. Enclosing the intellectual commons as economic dematerialisation

26. Academic publications as Veblen goods

27. Ensemble strategies at the population level. I don’t need to guess right, we need a society in which people in aggregate guess in a calibrated way.

28. Epistemic bottlenecks and bandwidth problems

    1. Information versus learning as a fundamental question of ML. When do we store exemplars on disk? When do we gradient updates? How much compute to spend on compressing?
    2. What is special about science? Transmissibility. Can chatGPT do transmission? Or is it 100% tacit? How does explainabilitiy relate to transmissibility?

29. Stein variational gradient descent

30. Edge of chaos, history of

31. Interaction effects

32. Ethical consumption

33. X is Yer than Z

34. But what can I do?

35. Haunting and exchangeability

36. Black swan farming

37. Doing complicated things naively

38. Conspiracies as simulations

39. Something about the limits of legible fairness versus metis in common property regimes

40. The uncanny ally

41. Elliptical belief propagation

42. Akrasia in stochastic processes: What time-integrated happiness should we optimise? Connection: contrastive learning

43. Strategic ignorance

44. privilege accountancy

45. anthropic principles ✅ Good enough

46. You can’t talk about us without us ❌ what did I even mean

47. subculture dynamics ✅ Good enough

48. Opinion dynamics (memetics for beginners) ✅ Good enough

49. Table stakes versus tokenism ✅

50. Iterative game theory under bounded rationality ❌ too general

51. Memetics ❌ (too big, will never finish)

52. Cradlesnatch calculator ✅ Good enough

Workflow optimisation

• Shell Integration - Documentation - iTerm2 - macOS Terminal Replacement
• josStorer/chatGPTBox: Integrating ChatGPT into your browser deeply, everything you need is here
• GPT Everywhere: The Ultimate AI Assistant App for Seamless Integration with Your Files, Keyboard Shortcuts, Whisper AI, and Custom LLaMAs

Commoncog

• Start Here: Commoncog’s Best Posts - Commoncog
• The 2022 Commoncog Recap - Commoncog
• Setting the Business Expertise Series Free - Commoncog

graphical models

• versus autodiff: There and Back Again: A Tale of Slopes and Expectations | Mathematics for Machine Learning
• Zhoubin
• Montanari
• Machine Learning — Graphical Model Exact inference (Variable elimination, Belief propagation, Junction tree) | by Jonathan Hui
• scribe_note_lecture13.pdf
• Belief propagation -
• gss2013_11344.pdf

“transfer” learning

• Bernhard Schölkopf: From statistical to causal learning
• Bernhard Schölkopf: Learning Causal Mechanisms (ICLR invited talk)
• thuml/Transfer-Learning-Library: Transfer Learning Library for Domain Adaptation, Task Adaptation, and Domain Generalization
• Transfer Learning — Transfer Learning Library 0.0.24 documentation
• thuml/A-Roadmap-for-Transfer-Learning

Internal

• iPhD Researchers - MyCSIRO
• Industry PhD Program Information Session 2023 28/3/2023
• LineLab Production Modeling Software
• Turing Talk: AI, data science and environmental change
• QUT - Research student internships

ICML 2023 workshop

• Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators | ICML 2023 Workshop, Honolulu, Hawaii — ICML 2023
• Structured Probabilistic Inference & Generative Modeling @ ICML — ICML 2023
• Duality Principles for Modern ML — ICML 2023
• Synergy of Scientific and Machine Learning Modeling — ICML 2023

Neurips 2022 follow-ups

• The Symbiosis of Deep Learning and Differential Equations (DLDE)
• NeurIPS 2022 Workshop DLDE | OpenReview

1. Arya et al. (2022) — stochastic gradients are more general than deterministic ones because they are defined on discrete vars
2. Rudner et al. (2022)
3. Phillips et al. (2022) — diffusions in the spectral domain allow us to handle continuous function valued inputs
4. Gahungu et al. (2022)
5. Wu, Maruyama, and Leskovec (2022) LE-PDE is a learnable low-rank approximation method
6. Holl, Koltun, and Thuerey (2022) — Physics loss via forward simulations, without the need for sensitivity.
7. Neural density estimation
8. Metrics for inverse design and inverse inference problems - the former is in fact easier. Or is it? can we simply attain forward prediction loss?
9. Noise injection in emulator learning (see refs in Su et al. (2022))

Conf, publication venues

• International Joint Conference on Neural Networks 2023
• Transactions on Machine Learning Research

Neurips 2022

• NeurIPS Workshop on Gaussian Processes, Spatiotemporal Modeling, and Decision-making Systems
• SBM @ NeurIPS
• Causal dynamics
• The Symbiosis of Deep Learning and Differential Equations (DLDE)
• Machine Learning and the Physical Sciences, NeurIPS 2022
• AI for Science: Progress and Promises
• Machine Learning Street Talk

Neurips 2021

• Storchastic: A Framework for General Stochastic Automatic Differentiation
• Causal Inference & Machine Learning: Why now?
• Real-Time Optimization for Fast and Complex Control Systems
• [2104.13478] Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Cheng Soon Ong, Marc Peter Deisenroth | There and Back Again: A Tale of Slopes and Expectations (“Lets unify automatic differentiation, integration and mesage passing”
• David Duvenaud, J. Zico Kolter, Matt Johnson | Deep Implicit Layers: Neural ODEs, Equilibrium Models and Beyond

Music

Nestup / cutelabnyc/nested-tuplets: Fancy javascript for manipulating nested tuplets.

Hot topics

• Equations of Motion from a Time Series
• Path Integrals and Feynman Diagrams for Classical Stochastic Processes
• Inference for Stochastic Differential Equations
• George Ho, Modern Computational Methods for Bayesian Inference: A Reading List is a good curation of modern Bayes methods posts. The next links come from there
• will wolf on neural methods in Simulation-Based Inference
• will wolf, Deriving Expectation-Maximization
• will wolf, Deriving Mean-Field Variational Bayes
• Reality Is Just a Game Now
• Michael Bronstein, Graph Neural Networks as gradient flows, re: [2206.10991] Graph Neural Networks as Gradient Flows: understanding graph convolutions via energy
• M Bronstein’s ICLR 2021 Keynote, Geometric Deep Learning: The Erlangen Programme of ML
• How to write a great research paper
• The Notion of “Double Descent”
• Jaan on translating between variational terminology in physics and ML
• Sander on waveform audio
• yuge shi’s ELBO gradient post is excellent
• Francis Bach, the many faces of integration by parts.
• Bubeck on hot results in learning theory takes him far from the world of mirror descent, where i first met him. Also lectures well, IMO.
• Causality for Machine Learning

Stein stuff

• Stein Variational Gradient Descent · Depth First Learning
• Qiang on kernelized Stein discrepancy
• UT Statistical Learning & AI Group
• Stein’s Method for Practical Machine Learning
• A Stein Variational Framework for Deep Probabilistic Modeling
• Probabilistic Learning and Inference Using Stein Discrepancy

newsletter migration

• Editing "Links blog"
• Substack Dashboard
• Ghost(Pro)

GP research

• https://www.patreon.com/posts/new-linearized-69325387
• Regression-based covariance functions for nonstationary spatial modeling
• kalman-jax/sde_gp.py at master · AaltoML/kalman-jax
• AaltoML/kalman-jax: Approximate inference for Markov Gaussian processes using iterated Kalman smoothing, in JAX

SDEs, optimisation and gradient flows

Nguyen and Malinsky (2020)

Statistical Inference via Convex Optimization.

Conjugate functions illustrated.

Francis Bach on the use of geometric sums and a different take by Julyan Arbel.

Tutorial to approximating differentiable control problems. An extension of this is universal differential equations.

• An Introduction to Stochastic Calculus

Career tips and metalearning

• Commoncog on learning

• jbhuang0604/awesome-tips

• Making the right moves: A Practical Guide to Scientific Management for Postdocs and New Faculty

• Things You Are Allowed To Do, Academic Edition

• There is a Q&A site about this, Academia stackexchange

• For early career types, classic blog Thesis Whisperer

• How to teach statistics

• How to do academic online

• publishing is messed up

• Read Academic work-life-balance survey to feel like not bothering with academe.

• Andrej Karpathy’s guide

• AI research: the unreasonably narrow path and how not to be miserable

• How To Be an Academic Hyper-Producer

• How to Become the Best in the World at Something

  This is how skill stacking works. It’s easier and more effective to be in the top 10% in several different skills — your “stack” — than it is to be in the top 1% in any one skill.

Ensembles and particle methods

• The pseudo-marginal approach for efficient Monte Carlo computations
• 2303.13930.pdf

Foundations of ML

So much Michael Betancourt.

• Probability Theory (For Scientists and Engineers)
• Course Notes 7: Gaussian Process Engineering | Michael Betancourt on Patreon
• Conditional Probability Theory (For Scientists and Engineers)
• Autodiff for Implicit Functions Paper Live Stream Wed 1/12 at 11 AM EST | Michael Betancourt on Patreon
• New Autodiff Paper | Michael Betancourt on Patreon
• Rumble in the Ensemble
• Scholastic Differential Equations | Michael Betancourt on Patreon
• Identity Crisis
• Invited Talk: Michael Bronstein
• Product Placement
• (Not So) Free Samples
• Updated Geometric Optimization Paper
• We Built Sparse City
• Rumble in the Ensemble

nonparametrics

• Tamara Broderick 1: Bayesian Nonparametrics
• Tamara Broderick 2: Bayesian Nonparametrics
• Tamara Broderick 3: Bayesian Nonparametrics

References