# optimization

Implementing neural nets
2016-10-14
– 2023-01-27Reparameterization tricks in inference
Pathwise gradient estimation, nNormalizing flows, invertible density models, inference by measure transport, low-dimensional coupling…
2018-04-04
– 2023-01-19Optimal transport inference
I feel the earth mover under my feet, I feel the ψ tumbling down, I feel my heart start to trembling, Whenever you’re around my empirical density in minimal transport cost
2021-03-16
– 2023-01-16Maximum Mean Discrepancy
2016-08-21
– 2023-01-13Variational message-passing algorithms in graphical models
Cleaving reality at the joint, then summing it at the marginal
2014-11-25
– 2023-01-12Pytorch
#torched
2018-05-04
– 2023-01-12(Kernelized) Stein variational gradient descent
KSVD, SVGD
2022-11-02
– 2023-01-09Probability divergences
Metrics, contrasts and divergences and other ways of quantifying how similar are two randomnesses
2014-11-25
– 2023-01-06Transforms of Gaussian noise
Delta method, error propagation, unscented transform, Taylor expansion…
2014-11-25
– 2022-12-23Biomimetic algorithms
2015-12-22
– 2022-12-06Gradient flows
infinitesimal optimization
2020-01-30
– 2022-11-02Online learning
2018-09-30
– 2022-10-20Neural tangent kernel
2020-12-09
– 2022-10-14Multi-objective optimisation
2021-07-14
– 2022-10-10(Weighted) least squares fits
2016-09-22
– 2022-10-04Automatic differentiation
2016-07-27
– 2022-10-04Neural denoising diffusion models
Denoising diffusion probabilistic models (DDPMs), score-based generative models, generative diffusion processes, neural energy models…
2021-11-11
– 2022-09-24Score matching
2021-11-11
– 2022-09-23Gaussian process inference by partial updates
2020-12-03
– 2022-09-22Penalised/regularised regression
2016-06-23
– 2022-09-19Gradient descent, first-order, stochastic
a.k.a. SGD, as seen in deep learning
2020-01-30
– 2022-09-05Gaussian belief propagation
Least squares at maximal elaboration
2014-11-25
– 2022-09-01Distributional robustness in inference
2019-07-12
– 2022-08-30Elliptical belief propagation
Generalized least generalized squares
2022-08-22
– 2022-08-23Recommender systems
2020-11-30
– 2022-08-15Automatic differentiation in Julia
2016-07-27
– 2022-08-09Jax
Julia for python
2020-09-15
– 2022-08-09Neural nets with implicit layers
Also, declarative networks, bi-level optimization and other ingenious uses of the implicit function theorem
2020-12-08
– 2022-08-09Scaling laws for very large neural nets
Compute/size/data tradeoffs
2021-01-14
– 2022-08-09ELBO
Evidence lower bound, variational free energy etc
2020-10-02
– 2022-08-03Bayes linear regression and basis-functions in Gaussian process regression
a.k.a Fixed Rank Kriging, weight space GPs
2022-02-22
– 2022-07-27Large sample theory
2015-02-15
– 2022-07-25Gradient descent, Newton-like
2019-02-05
– 2022-07-25Fun tricks in non-convex optimisation
2014-10-04
– 2022-07-14Markov decision problems
2014-11-27
– 2022-06-07Machine learning and statistics in Julia
2019-11-27
– 2022-05-27SLAM
Simultaneous Location and Mapping
2014-11-25
– 2022-04-28Gradient descent
First order of business
2014-10-04
– 2022-04-28Vecchia factoring of GP likelihoods
Ignore some conditioning in the dependencies and attain a sparse cholesky factor for the precision matrix
2022-04-27Particle belief propagation
Graphical inference using empirical distribution estimates
2014-07-25
– 2022-04-08Particle Markov Chain Monte Carlo
Particle systems as MCMC proposals
2014-07-25
– 2022-04-08Matrix calculus
2018-07-09
– 2022-04-04Belief propagation
2014-11-25
– 2022-03-31Wiener-Khintchine representations
Spectral representations of stochastic processes
2019-05-08
– 2022-03-11Genetic programming
2015-12-22
– 2022-03-09Sparse coding
Wavelets, matching pursuit, overcomplete dictionaries…
2014-11-17
– 2022-03-07Generative flow
2022-03-07OODA loops
2021-05-01
– 2022-02-28M-estimation
2016-07-11
– 2022-02-17Variational inference
On fitting something not too far from a pretty good model that is not too hard
2016-03-22
– 2022-02-10Evolution
2014-11-04
– 2022-02-07Neural nets with basis decomposition layers
2021-03-09
– 2022-02-01Running neural nets backwards
2022-01-29(Outlier) robust statistics
2014-11-25
– 2022-01-21Gradient descent, Newton-like, stochastic
2020-01-23
– 2021-12-09Variational state filtering
2018-03-19
– 2021-12-08Gaussian Processes as stochastic differential equations
Imposing time on things
2019-09-18
– 2021-11-25Optimisation
2014-10-04
– 2021-11-11Deep generative models
2020-12-10
– 2021-11-11Probably Approximately Correct
2014-11-24
– 2021-10-27Hyperparameter optimization
Replacing a hyperparameter problem with a hyperhyperparameter problem which feels like progress I guess
2020-09-25
– 2021-10-20Gradient descent at scale
Practical implementation of large optimisations
2021-07-14
– 2021-09-28Meta learning
Few-shot learning, learning fast weights, learning to learn
2021-09-16Wirtinger calculus
It’s not complicated / It’s complex
2019-05-08
– 2021-09-07Generalised linear models
2016-03-24
– 2021-08-05Sequential experiments
Especially multiple sequential experiments
2021-08-04Tensorflow
The framework to use for deep learning if you groupthink like Google
2016-07-11
– 2021-07-07Learning on tabular data
2020-11-30
– 2021-06-21Random-forest-like methods
A selection of randomly stopped clocks is never far from wrong.
2015-09-23
– 2021-06-17Voice fakes
2018-09-06
– 2021-06-17Optimal transport metrics
Wasserstein distances, Monge-Kantorovich metrics, Earthmover distances
2019-05-30
– 2021-06-08Energy based models
Inference with kinda-tractable un-normalized densities
2021-06-07Stein’s method
His eyes are like angels but his heart is cold / No need to ask / He’s a Stein operator
2021-03-12
– 2021-06-01Differentiable model selection
Differentiable hyperparameter search, and architecture search, and optimisation optimisation by optimisation and so on
2020-09-25
– 2021-04-13Generically approximating probability distributions
2021-03-12
– 2021-03-22Generative adversarial networks
2016-10-07
– 2020-12-14Distribution regression
2020-12-01Efficient factoring of GP likelihoods
2020-10-16
– 2020-10-26Automatic design of experiments
Minesweeper++
2017-04-11
– 2020-10-13AutoML
2017-07-17
– 2020-10-02Data summarization
On maps drawn at smaller than 1:1 scale
2019-01-14
– 2020-09-18Variational autoencoders
2019-11-04
– 2020-09-10Natural gradient descent
Climbing slower on the tricky bits
2019-07-18
– 2020-05-26Variational inference
On fitting the best model one can be bothered to
2016-03-22
– 2020-05-24Phase retrieval
I’ve got the power. / Like the crack of the whip/ I snap attack/ Front to back
2017-01-16
– 2019-11-07Optimal control
2015-06-22
– 2019-11-01Gradient descent, Higher order
2019-10-26Statistical learning theory for time series
2016-11-03
– 2019-10-01Semidefinite proramming
2019-06-29Wiener theorem
Now with bonus Bochner!
2019-05-08Wacky regression
2015-09-23
– 2019-05-02Nearly sufficient statistics
How about “Sufficient sufficiency”? — is that taken?
2018-03-13
– 2019-01-14Optimisation, combinatorial
2018-08-11Submodular functions, maximizing
2018-07-09Gradient descent, continuous, primal/dual formulations.
2017-08-07Lagrangian mechanics
2015-02-11
– 2017-06-18Maximum likelihood inference
2015-02-15
– 2016-10-13Distributed statistica inference
2016-10-11Statistical learning theory
Eventually including structural risk minimisation, risk bounds, hopefully-uniform convergence rates, VC-dimension, generalisation-and-stability framings etc
2016-07-06
– 2016-08-16Expectation maximisation
2014-08-17
– 2016-04-17Geometry of fitness landscapes
2011-07-27
– 2015-06-17