high_d on Dan MacKinlayhttps://danmackinlay.name/tags/high_d.htmlRecent content in high_d on Dan MacKinlayHugo -- gohugo.ioen-usWed, 12 Jan 2022 20:23:51 +1100Recommender systemshttps://danmackinlay.name/notebook/recommender_systems.htmlWed, 12 Jan 2022 20:23:51 +1100https://danmackinlay.name/notebook/recommender_systems.htmlZoo Tooling Vowpal wabbit For control and addiction of humans References Not my core area, but I need a landing page to refer to because this is coming up in conversations a lot at the moment.
Users who liked apostasy also enjoyed…
Recommender systems are a weird area. There have been some high profile examples of recommender models going back a long time (e.Random rotationshttps://danmackinlay.name/notebook/random_rotations.htmlWed, 01 Dec 2021 11:25:12 +1100https://danmackinlay.name/notebook/random_rotations.htmlUniform random rotations Tiny random rotations Random Givens rotation Doubly Random Givens rotation In an isotropic direction References Placeholder for random measures on the special orthogonal group, which encodes random \(d\)-dimensional rotations. A special type of matrix random variate if you’d like.
Uniform random rotations Key word: Haar measure.
Chatterjee and Meckes (2008) gives us a useful lemma describing the moments of a random rotation:High dimensional statisticshttps://danmackinlay.name/notebook/high_d_statistics.htmlThu, 28 Oct 2021 13:07:54 +1100https://danmackinlay.name/notebook/high_d_statistics.htmlSoap bubbles Convex hulls Empirical processes in high dimensions Markov Chain Monte Carlo in high dimensions References Placeholder to think about the many weird problems arising in very high dimensional statistical inference. There are many approaches to this problem: throwing out dimensions/predictors as in model selection, considering low dimensional projections, viewing objects with matrix structure for concentration or factorisation, or tensor structure even.
Soap bubbles High dimensional distributions are extremely odd, and concentrate in weird ways.Fun with rotational symmetrieshttps://danmackinlay.name/notebook/rotationally_symmetric.htmlThu, 14 Oct 2021 14:26:57 +1100https://danmackinlay.name/notebook/rotationally_symmetric.htmlRadial functions In dot-product kernels Polynomial integrals on radially symmetric domains Integrating radial function on rotationally symmetric domains Generating random points on balls and spheres Directional statistics Random projections Transforms Hankel transforms Integration algebra Misc References There are some related tricks that I used for functions with rotational symmetry, and functions on domains with rotational symmetry. Here is where I write them down to remember.The Gaussian distributionhttps://danmackinlay.name/notebook/gaussian_distribution.htmlSun, 03 Oct 2021 11:20:38 +1100https://danmackinlay.name/notebook/gaussian_distribution.htmlDensity, CDF Mills ratio Differential representations Stein’s lemma ODE representation for the univariate density ODE representation for the univariate icdf Density PDE representation as a diffusion equation Extremes Orthogonal basis Rational function approximations Roughness Entropy Multidimensional marginals and conditionals Fourier representation Transformed variables Metrics Wasserstein Kullback-Leibler Hellinger What is Erf again? Matrix Gaussian References Bell curves
Many facts about the useful, boring, ubiquitous Gaussian.Orthonormal and unitary matriceshttps://danmackinlay.name/notebook/orthonormal_matrices.htmlWed, 22 Sep 2021 20:03:03 +1000https://danmackinlay.name/notebook/orthonormal_matrices.htmlParametrising Take the QR decomposition Iterative normalising Householder reflections Givens rotation Cayley map Parametric sub families Structured Higher rank Random distributions over References In which I think about parameterisations and implementations of finite dimensional energy-preserving operators, a.k.a. matrices. A particular nook in the linear feedback process library, closely related to stability in linear dynamical systems, since every orthonormal matrix is the forward operator of an energy-preserving system, which is an edge case for certain natural types of stability.Learning on tabular datahttps://danmackinlay.name/notebook/tabular_data.htmlMon, 21 Jun 2021 12:44:21 +1000https://danmackinlay.name/notebook/tabular_data.htmlReferences Learning for tabular data, i.e. the stuff you generally store in spreadsheets and relational databases.
Popular in many areas, notably recommender systems.
Not much for now.
(Cheng et al. 2016): jrzaurin/pytorch-widedeep: A flexible package to combine tabular data with text and images using Wide and Deep models in Pytorch. pytorch-widedeep, deep learning for tabular data IV: Deep Learning vs LightGBM | infinitoml
Note that the author of that package advises using gradient boosting machines to get this job done.Isotropic random vectorshttps://danmackinlay.name/notebook/isotropic_vectors.htmlMon, 24 May 2021 15:00:54 +1000https://danmackinlay.name/notebook/isotropic_vectors.htmlSimulating isotropic vectors On the \(d\)-sphere On the \(d\)-ball Marginal distributions Inner products Moments Archimedes Principles Funk-Hecke References wow! This notebook entry is now 10 years old. I have cared about this for ages.
Random variables with radial symmetry; It would be appropriate to define these circularly, so I will, which is to say, (centred) isotropic random vectors are those whose distribution is unchanged under fixed rotations, or random rotations.Randomized low dimensional projectionshttps://danmackinlay.name/notebook/low_d_projections.htmlMon, 24 May 2021 14:16:37 +1000https://danmackinlay.name/notebook/low_d_projections.htmlTutorials Inner products Random projections are kinda Gaussian Random projections are distance preserving Projection statistics Concentration theorems for projections References One way I can get at the confusing behaviours of high dimensional distributions is to instead look at low dimensional projections of them. If I have a (possibly fixed) data matrix and a random dimensional projection, what distribution does the projection have?
This idea pertains to many others: matrix factorisations, restricted isometry properties, Riesz bases, randomised regression, compressed sensing.Matrix measure concentration inequalities and boundshttps://danmackinlay.name/notebook/matrix_concentration.htmlMon, 08 Mar 2021 11:08:41 +1100https://danmackinlay.name/notebook/matrix_concentration.htmlMatrix Chernoff Matrix Chebychev Matrix Bernstein Matrix Efron-Stein Gaussian References Concentration inequalities for matrix-valued random variables.
Recommended overviews are J. A. Tropp (2015); van Handel (2017); Vershynin (2018).
Matrix Chernoff J. A. Tropp (2015) summarises:
In recent years, random matrices have come to play a major role in computational mathematics, but most of the classical areas of random matrix theory remain the province of experts.Sparse model selectionhttps://danmackinlay.name/notebook/sparse_model_selection.htmlFri, 02 Oct 2020 17:50:51 +1000https://danmackinlay.name/notebook/sparse_model_selection.htmlFOCI Stability selection Relaxed Lasso Dantzig Selector Garotte Degrees-of-freedom penalties References On choosing the right model and regularisation parameter in sparse regression, which turn out to be nearly the same, and closely coupled to doing the regression. There are some wrinkles.
🏗 Talk about when degrees-of-freedom penalties work, when cross-validation and so on.
FOCI The new hotness sweeping the world is FOCI, a sparse model selection procedure (Azadkia and Chatterjee 2019) based on Chatterjee’s ξ statistic as an independence test test.(Approximate) matrix factorisationhttps://danmackinlay.name/notebook/matrix_factorisation.htmlFri, 03 Jul 2020 19:51:38 +1000https://danmackinlay.name/notebook/matrix_factorisation.htmlWhy does it ever work Overviews Non-negative matrix factorisations As regression Sketching \([\mathcal{H}]\)-matrix methods Randomized methods Connections to kernel learning Implementations References Forget QR and LU decompositions, there are now so many ways of factorising matrices that there are not enough acronyms in the alphabet to hold them, especially if you suspect your matrix is sparse, or could be made sparse because of some underlying constraint, or probably could, if squinted at in the right fashion, be such as a graph transition matrix, or Laplacian, or noisy transform of some smooth object, or at least would be close to sparse if you chose the right metric, or…Sparse codinghttps://danmackinlay.name/notebook/sparse_coding.htmlTue, 05 Nov 2019 16:28:28 +0100https://danmackinlay.name/notebook/sparse_coding.htmlResources Wavelet bases Matching Pursuits Learnable codings Codings with desired invariances Misc Implementations References Linear expansion with dictionaries of basis functions, with respect to which you wish your representation to be sparse; i.e. in the statistical case, basis-sparse regression. But even outside statistics, you wish simply to approximate some data compactly. My focus here is on the noisy-observation case, although the same results are recycled enough throughout the field.Elliptical distributionshttps://danmackinlay.name/notebook/elliptical_distributions.htmlTue, 23 Jun 2015 18:28:30 +0200https://danmackinlay.name/notebook/elliptical_distributions.htmlReferences TBD
References Anderson, T. W. 2006. An introduction to multivariate statistical analysis. Hoboken, N.J.: Wiley-Interscience. Cambanis, Stamatis, Steel Huang, and Gordon Simons. 1981. “On the Theory of Elliptically Contoured Distributions.” Journal of Multivariate Analysis 11 (3): 368–85. Chamberlain, Gary. 1983. “A Characterization of the Distributions That Imply Mean—Variance Utility Functions.” Journal of Economic Theory 29 (1): 185–201. Culan, Christophe, and Claude Adnet.