optimization

Implementing neural nets 2016-10-14 – 2023-01-27

Reparameterization tricks in inference Pathwise gradient estimation, nNormalizing flows, invertible density models, inference by measure transport, low-dimensional coupling… 2018-04-04 – 2023-01-19

Optimal transport inference I feel the earth mover under my feet, I feel the ψ tumbling down, I feel my heart start to trembling, Whenever you’re around my empirical density in minimal transport cost 2021-03-16 – 2023-01-16

Maximum Mean Discrepancy 2016-08-21 – 2023-01-13

Variational message-passing algorithms in graphical models Cleaving reality at the joint, then summing it at the marginal 2014-11-25 – 2023-01-12

Pytorch #torched 2018-05-04 – 2023-01-12

(Kernelized) Stein variational gradient descent KSVD, SVGD 2022-11-02 – 2023-01-09

Probability divergences Metrics, contrasts and divergences and other ways of quantifying how similar are two randomnesses 2014-11-25 – 2023-01-06

Transforms of Gaussian noise Delta method, error propagation, unscented transform, Taylor expansion… 2014-11-25 – 2022-12-23

Biomimetic algorithms 2015-12-22 – 2022-12-06

Gradient flows infinitesimal optimization 2020-01-30 – 2022-11-02

Online learning 2018-09-30 – 2022-10-20

Neural tangent kernel 2020-12-09 – 2022-10-14

Multi-objective optimisation 2021-07-14 – 2022-10-10

(Weighted) least squares fits 2016-09-22 – 2022-10-04

Automatic differentiation 2016-07-27 – 2022-10-04

Neural denoising diffusion models Denoising diffusion probabilistic models (DDPMs), score-based generative models, generative diffusion processes, neural energy models… 2021-11-11 – 2022-09-24

Score matching 2021-11-11 – 2022-09-23

Gaussian process inference by partial updates 2020-12-03 – 2022-09-22

Penalised/regularised regression 2016-06-23 – 2022-09-19

Gradient descent, first-order, stochastic a.k.a. SGD, as seen in deep learning 2020-01-30 – 2022-09-05

Gaussian belief propagation Least squares at maximal elaboration 2014-11-25 – 2022-09-01

Distributional robustness in inference 2019-07-12 – 2022-08-30

Elliptical belief propagation Generalized least generalized squares 2022-08-22 – 2022-08-23

Recommender systems 2020-11-30 – 2022-08-15

Automatic differentiation in Julia 2016-07-27 – 2022-08-09

Jax Julia for python 2020-09-15 – 2022-08-09

Neural nets with implicit layers Also, declarative networks, bi-level optimization and other ingenious uses of the implicit function theorem 2020-12-08 – 2022-08-09

Scaling laws for very large neural nets Compute/size/data tradeoffs 2021-01-14 – 2022-08-09

ELBO Evidence lower bound, variational free energy etc 2020-10-02 – 2022-08-03

Bayes linear regression and basis-functions in Gaussian process regression a.k.a Fixed Rank Kriging, weight space GPs 2022-02-22 – 2022-07-27

Large sample theory 2015-02-15 – 2022-07-25

Gradient descent, Newton-like 2019-02-05 – 2022-07-25

Fun tricks in non-convex optimisation 2014-10-04 – 2022-07-14

Markov decision problems 2014-11-27 – 2022-06-07

Machine learning and statistics in Julia 2019-11-27 – 2022-05-27

SLAM Simultaneous Location and Mapping 2014-11-25 – 2022-04-28

Gradient descent First order of business 2014-10-04 – 2022-04-28

Vecchia factoring of GP likelihoods Ignore some conditioning in the dependencies and attain a sparse cholesky factor for the precision matrix 2022-04-27

Particle belief propagation Graphical inference using empirical distribution estimates 2014-07-25 – 2022-04-08

Particle Markov Chain Monte Carlo Particle systems as MCMC proposals 2014-07-25 – 2022-04-08

Matrix calculus 2018-07-09 – 2022-04-04

Belief propagation 2014-11-25 – 2022-03-31

Wiener-Khintchine representations Spectral representations of stochastic processes 2019-05-08 – 2022-03-11

Genetic programming 2015-12-22 – 2022-03-09

Sparse coding Wavelets, matching pursuit, overcomplete dictionaries… 2014-11-17 – 2022-03-07

Generative flow 2022-03-07

OODA loops 2021-05-01 – 2022-02-28

M-estimation 2016-07-11 – 2022-02-17

Variational inference On fitting something not too far from a pretty good model that is not too hard 2016-03-22 – 2022-02-10

Evolution 2014-11-04 – 2022-02-07

Neural nets with basis decomposition layers 2021-03-09 – 2022-02-01

Running neural nets backwards 2022-01-29

(Outlier) robust statistics 2014-11-25 – 2022-01-21

Gradient descent, Newton-like, stochastic 2020-01-23 – 2021-12-09

Variational state filtering 2018-03-19 – 2021-12-08

Gaussian Processes as stochastic differential equations Imposing time on things 2019-09-18 – 2021-11-25

Optimisation 2014-10-04 – 2021-11-11

Deep generative models 2020-12-10 – 2021-11-11

Probably Approximately Correct 2014-11-24 – 2021-10-27

Hyperparameter optimization Replacing a hyperparameter problem with a hyperhyperparameter problem which feels like progress I guess 2020-09-25 – 2021-10-20

Gradient descent at scale Practical implementation of large optimisations 2021-07-14 – 2021-09-28

Meta learning Few-shot learning, learning fast weights, learning to learn 2021-09-16

Wirtinger calculus It’s not complicated / It’s complex 2019-05-08 – 2021-09-07

Generalised linear models 2016-03-24 – 2021-08-05

Sequential experiments Especially multiple sequential experiments 2021-08-04

Tensorflow The framework to use for deep learning if you groupthink like Google 2016-07-11 – 2021-07-07

Learning on tabular data 2020-11-30 – 2021-06-21

Random-forest-like methods A selection of randomly stopped clocks is never far from wrong. 2015-09-23 – 2021-06-17

Voice fakes 2018-09-06 – 2021-06-17

Optimal transport metrics Wasserstein distances, Monge-Kantorovich metrics, Earthmover distances 2019-05-30 – 2021-06-08

Energy based models Inference with kinda-tractable un-normalized densities 2021-06-07

Stein’s method His eyes are like angels but his heart is cold / No need to ask / He’s a Stein operator 2021-03-12 – 2021-06-01

Differentiable model selection Differentiable hyperparameter search, and architecture search, and optimisation optimisation by optimisation and so on 2020-09-25 – 2021-04-13

Generically approximating probability distributions 2021-03-12 – 2021-03-22

Generative adversarial networks 2016-10-07 – 2020-12-14

Distribution regression 2020-12-01

Efficient factoring of GP likelihoods 2020-10-16 – 2020-10-26

Automatic design of experiments Minesweeper++ 2017-04-11 – 2020-10-13

AutoML 2017-07-17 – 2020-10-02

Data summarization On maps drawn at smaller than 1:1 scale 2019-01-14 – 2020-09-18

Variational autoencoders 2019-11-04 – 2020-09-10

Natural gradient descent Climbing slower on the tricky bits 2019-07-18 – 2020-05-26

Variational inference On fitting the best model one can be bothered to 2016-03-22 – 2020-05-24

Phase retrieval I’ve got the power. / Like the crack of the whip/ I snap attack/ Front to back 2017-01-16 – 2019-11-07

Optimal control 2015-06-22 – 2019-11-01

Gradient descent, Higher order 2019-10-26

Statistical learning theory for time series 2016-11-03 – 2019-10-01

Semidefinite proramming 2019-06-29

Wiener theorem Now with bonus Bochner! 2019-05-08

Wacky regression 2015-09-23 – 2019-05-02

Nearly sufficient statistics How about “Sufficient sufficiency”? — is that taken? 2018-03-13 – 2019-01-14

Optimisation, combinatorial 2018-08-11

Submodular functions, maximizing 2018-07-09

Gradient descent, continuous, primal/dual formulations. 2017-08-07

Lagrangian mechanics 2015-02-11 – 2017-06-18

Maximum likelihood inference 2015-02-15 – 2016-10-13

Distributed statistica inference 2016-10-11

Statistical learning theory Eventually including structural risk minimisation, risk bounds, hopefully-uniform convergence rates, VC-dimension, generalisation-and-stability framings etc 2016-07-06 – 2016-08-16

Expectation maximisation 2014-08-17 – 2016-04-17

Geometry of fitness landscapes 2011-07-27 – 2015-06-17