Sparse coding

Wavelets, matching pursuit, overcomplete dictionaries…

2014-11-17 – 2022-03-07
Maybe related 🀷:

Linear expansion with dictionaries of basis functions, with respect to which you wish your representation to be sparse; i.e. in the statistical case, basis-sparse regression. But even outside statistics, you wish simply to approximate some data compactly. My focus is on the noisy-observation case, although the same results are recycled enough throughout the field.

There are several senses in which people seem to use sparse coding; these do not necessarily mean the same thing, but they frequently connect.

  • I know that my signal happens to be compressible, in the sense that under some transform its coefficient vector is mostly zeros, even in a plain old orthogonal basis expansion. Or, relatedly, I know that under such a transform, the fidelity of my reproduction of the signal decays rapidly with the number of bits, in some metric.

  • I am using a redundant dictionary such that I won’t need most of it to represent even a dense signal. This means that the representations of the signals might have lots of zeros, but nonetheless may not be compressible in an information-theoretic sense.

I should break these two notions apart here. For now, I’m especially interested in adaptive bases.

This is merely a bunch of links to important articles at the moment; I might write a little exposition one day.

Decomposition of stuff by matching pursuit, wavelets, curvelets, chirplets, framelets, shearlets, camelhairbrushlets, content-specific basis dictionaries, designed or learned. Mammals visual cortexes seem to use something like this, if you squint right at the evidence.

To discuss:

Daniel LaCombe, some sparse basis functions

Resources

Baraniuk’s lab has a comprehensive, but not usefully annotated, selection of articles in this field, which I include less as a useful starting point, than to demonstrate the virtue of a literature review by showing the pathology of its absence.

Wavelet bases

Very popular practical intro is Torrence and Compo.

πŸ—

Scattering transform coefficients

See scattering transform.

Matching Pursuits

I do a lot of this now. I should document it. πŸ—

Learnable codings

See Adaptive sparse coding.

Bayesian

Unsure. See perhaps Schniter, Potter, and Ziniel (2008), ZhouNonparametric2009.

Incoming

Affine tight framelets (Ingrid Daubechies et al. 2003) and their presumably less-computationally-tractable, more flexible cousins, shearlets also sound interesting. For reasons I do not yet understand I am told they can naturally be used on sundry graphs and manifolds, not just lattices, is traditional in DSP. I saw Xiaosheng Zhuang present these (see, e.g. (Y. G. Wang and Zhuang 2016; B. Han, Zhao, and Zhuang 2016), where the latter demonstrates a Fast Framelet Transform which is supposedly as computationally as cheap as the FFT.)

I have some ideas I call learning gamelan which relate to this.

Implementations

Implementations boil down to clever optimisation and/or good use of functional transforms to make the calculations tractable.

  • Shailesh Kumar, Wavelet Transforms in Python with Google JAX introduces CR.Sparse, a JAX/XLA based library of accelerated models and algorithms for inverse problems in sparse representation and compressive sensing.

  • the standard wavelet toolkits.

    • scipy’s wavelet transform has no frills and little coherent explanation, but it goes
    • pywavelets does various fancy wavelets and seems to be a standard for python.
    • Matlab’s Wavelet toolbox seems to be the reference.
    • scikit-learn dictionary learning version here
    • also pydbm
    • Fancy easy GPU wavelet implementation, PyTorchWavelets.

  • SPORCO

    SParse Optimization Research COde (SPORCO) is an open-source Python package for solving optimization problems with sparsity-inducing regularization, consisting primarily of sparse coding and dictionary learning, for both standard and convolutional forms of sparse representation. In the current version, all optimization problems are solved within the Alternating Direction Method of Multipliers (ADMM) framework. SPORCO was developed for applications in signal and image processing, but is also expected to be useful for problems in computer vision, statistics, and machine learning.

  • Sparse-filtering: Unsupervised feature learning based on sparse-filtering

    This implements the method described Jiquan Ngiam, Pang Wei Koh, Zhenghao Chen, Sonia Bhaskar, Andrew Y. Ng: Sparse Filtering. NIPS 2011: 1125-1133 and is based on the Matlab code provided in the supplementary material

  • spams does a huge variety of off-the-shelf sparse codings, although none of them are flexible. Nonetheless it does some neat things fast.

SPAMS (SPArse Modeling Software) is an optimization toolbox for solving various sparse estimation problems.

  • Dictionary learning and matrix factorization (NMF, sparse PCA,…)
  • Solving sparse decomposition problems with LARS, coordinate descent, OMP, SOMP, proximal methods
  • Solving structured sparse decomposition problems (l1/l2, l1/linf, sparse group lasso, tree-structured regularization, structured sparsity with overlapping groups,…).

References

Aharon, M., M. Elad, and A. Bruckstein. 2006. β€œK-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation.” IEEE Transactions on Signal Processing 54 (11): 4311–22.
Arora, Sanjeev, Rong Ge, Tengyu Ma, and Ankur Moitra. 2015. β€œSimple, Efficient, and Neural Algorithms for Sparse Coding.” In Proceedings of The 28th Conference on Learning Theory, 40:113–49. Paris, France: PMLR.
Bach, Francis R., and Michael I. Jordan. 2006. β€œLearning Spectral Clustering, with Application to Speech Separation.” Journal of Machine Learning Research 7 (Oct): 1963–2001.
Baraniuk, Richard G., Volkan Cevher, Marco F. Duarte, and Chinmay Hegde. 2010. β€œModel-Based Compressive Sensing.” IEEE Transactions on Information Theory 56 (4): 1982–2001.
Barron, Andrew R., Albert Cohen, Wolfgang Dahmen, and Ronald A. DeVore. 2008. β€œApproximation and Learning by Greedy Algorithms.” The Annals of Statistics 36 (1): 64–94.
BarthΓ©lemy, Quentin, Anthony Larue, AurΓ©lien Mayoue, David Mercier, and JΓ©rΓ΄me I. Mars. 2012. β€œShift & 2D Rotation Invariant Sparse Coding for Multivariate Signals.” IEEE Transactions on Signal Processing 60 (4): 1597–1611.
Bertin, K., E. Le Pennec, and V. Rivoirard. 2011. β€œAdaptive Dantzig Density Estimation.” Annales de l’Institut Henri PoincarΓ©, ProbabilitΓ©s Et Statistiques 47 (1): 43–74.
Beylkin, G., R. Coifman, and V. Rokhlin. 1991. β€œFast Wavelet Transforms and Numerical Algorithms I.” Communications on Pure and Applied Mathematics 44 (2): 141–83.
Blumensath, Thomas, and Mike Davies. 2004. β€œOn Shift-Invariant Sparse Coding.” In Independent Component Analysis and Blind Signal Separation, edited by Carlos G. Puntonet and Alberto Prieto, 3195:1205–12. Berlin, Heidelberg: Springer Berlin Heidelberg.
β€”β€”β€”. 2006. β€œSparse and Shift-Invariant Representations of Music.” IEEE Transactions on Audio, Speech and Language Processing 14 (1): 50–57.
Bora, Ashish, Ajil Jalal, Eric Price, and Alexandros G. Dimakis. 2017. β€œCompressed Sensing Using Generative Models.” In International Conference on Machine Learning, 537–46.
Boyes, Graham. 2011. β€œDictionary-Based Analysis/Synthesis and Structured Representations of Musical Audio.” McGill University.
Bruna, Joan, and Stephane Mallat. 2013. β€œInvariant Scattering Convolution Networks.” IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (8): 1872–86.
β€”β€”β€”. 2019. β€œMultiscale Sparse Microcanonical Models.” arXiv:1801.02013 [Math-Ph, Stat], May.
Bruna, Joan, StΓ©phane Mallat, Emmanuel Bacry, and Jean-FranΓ§ois Muzy. 2015. β€œIntermittent Process Analysis with Scattering Moments.” The Annals of Statistics 43 (1): 323–51.
Cai, Jian-Feng, Raymond H. Chan, and Zuowei Shen. 2008. β€œA Framelet-Based Image Inpainting Algorithm.” Applied and Computational Harmonic Analysis, Special Issue on Mathematical Imaging – Part II, 24 (2): 131–49.
Carabias-Orti, J. J., T. Virtanen, P. Vera-Candeas, N. Ruiz-Reyes, and F. J. Canadas-Quesada. 2011. β€œMusical Instrument Sound Multi-Excitation Model for Non-Negative Spectrogram Factorization.” IEEE Journal of Selected Topics in Signal Processing 5 (6): 1144–58.
Casazza, Peter G., and Richard G. Lynch. 2015. β€œA Brief Introduction to Hilbert Space Frame Theory and Its Applications.” In Finite Frame Theory: A Complete Introduction to Overcompleteness.
Chen, Shaobing, and David L. Donoho. 1994. β€œBasis Pursuit.” In 1994 Conference Record of the Twenty-Eighth Asilomar Conference on Signals, Systems and Computers, 1994, 1:41–44 vol.1.
Cheng, Sihao, and Brice MΓ©nard. 2021. β€œHow to Quantify Fields or Textures? A Guide to the Scattering Transform.” arXiv.
Cox, Christopher R., and Timothy T. Rogers. 2021. β€œFinding Distributed Needles in Neural Haystacks.” Journal of Neuroscience 41 (5): 1019–32.
Daubechies, I., M. Defrise, and C. De Mol. 2004. β€œAn Iterative Thresholding Algorithm for Linear Inverse Problems with a Sparsity Constraint.” Communications on Pure and Applied Mathematics 57 (11): 1413–57.
Daubechies, Ingrid. 1988. β€œOrthonormal Bases of Compactly Supported Wavelets.” Communications on Pure and Applied Mathematics 41 (7): 909–96.
Daubechies, Ingrid, Bin Han, Amos Ron, and Zuowei Shen. 2003. β€œFramelets: MRA-Based Constructions of Wavelet Frames.” Applied and Computational Harmonic Analysis 14 (1): 1–46.
Davis, Geoffrey M. 1998. β€œA Wavelet-Based Analysis of Fractal Image Compression.” IEEE Transactions on Image Processing 7 (2): 141–54.
Davis, Geoffrey M., Stephane G. Mallat, and Zhifeng Zhang. 1994a. β€œAdaptive Time-Frequency Decompositions.” Optical Engineering 33 (7): 2183–91.
β€”β€”β€”. 1994b. β€œAdaptive Time-Frequency Decompositions with Matching Pursuit.” In Wavelet Applications, 2242:402–14. International Society for Optics and Photonics.
Davis, G., S. Mallat, and M. Avellaneda. 1997. β€œAdaptive Greedy Approximations.” Constructive Approximation 13 (1): 57–98.
DeVore, Ronald A. 1998. β€œNonlinear Approximation.” Acta Numerica 7 (January): 51–150.
Dong, Bin. 2015. β€œSparse Representation on Graphs by Tight Wavelet Frames and Applications.” Applied and Computational Harmonic Analysis.
Donoho, David L., and Iain M. Johnstone. 1995. β€œAdapting to Unknown Smoothness via Wavelet Shrinkage.” Journal of the American Statistical Association 90 (432): 1200–1224.
Donoho, David L., Iain M. Johnstone, Gerard Kerkyacharian, and Dominique Picard. 1995. β€œWavelet Shrinkage: Asymptopia?” Journal of the Royal Statistical Society. Series B (Methodological) 57 (2): 301–69.
Du, Pan, Warren A. Kibbe, and Simon M. Lin. 2006. β€œImproved Peak Detection in Mass Spectrum by Incorporating Continuous Wavelet Transform-Based Pattern Matching.” Bioinformatics 22 (17): 2059–65.
Eggert, J., and E. Korner. 2004. β€œSparse Coding and NMF.” In 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541), 4:2529–2533 vol.4.
Ekanadham, C., D. Tranchina, and E. P. Simoncelli. 2011. β€œRecovery of Sparse Translation-Invariant Signals With Continuous Basis Pursuit.” IEEE Transactions on Signal Processing 59 (10): 4735–44.
Fan, Jianqing, and Runze Li. 2001. β€œVariable Selection via Nonconcave Penalized Likelihood and Its Oracle Properties.” Journal of the American Statistical Association 96 (456): 1348–60.
Frazier, Michael W. 1999. An Introduction to Wavelets Through Linear Algebra. Springer.
Garg, Sahil, Irina Rish, Guillermo Cecchi, and Aurelie Lozano. 2017. β€œNeurogenesis-Inspired Dictionary Learning: Online Model Adaption in a Changing World.” In arXiv:1701.06106 [Cs, Stat].
Gersho, Allen, and Robert M. Gray. 2012. Vector Quantization and Signal Compression. Springer Science & Business Media.
GinΓ©, Evarist, and Richard Nickl. 2009. β€œUniform Limit Theorems for Wavelet Density Estimators.” The Annals of Probability 37 (4): 1605–46.
Giryes, R., G. Sapiro, and A. M. Bronstein. 2016. β€œDeep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?” IEEE Transactions on Signal Processing 64 (13): 3444–57.
Goodwin, M M. 2001. β€œMultiscale Overlap-Add Sinusoidal Modeling Using Matching Pursuit and Refinements.” In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.
Goodwin, M M, and M Vetterli. 1999. β€œMatching Pursuit and Atomic Signal Models Based on Recursive Filter Banks.” IEEE Transactions on Signal Processing 47 (7): 1890–1902.
Goodwin, M., and M. Vetterli. 1997. β€œAtomic Decompositions of Audio Signals.” In 1997 IEEE ASSP Workshop on Applications of Signal Processing to Audio and Acoustics, 1997.
Gray, R. 1984. β€œVector Quantization.” IEEE ASSP Magazine 1 (2): 4–29.
Gregor, Karol, and Yann LeCun. 2010. β€œLearning fast approximations of sparse coding.” In Proceedings of the 27th International Conference on Machine Learning (ICML-10), 399–406.
β€”β€”β€”. 2011. β€œEfficient Learning of Sparse Invariant Representations.” arXiv:1105.5307 [Cs], May.
Grosse, Roger, Rajat Raina, Helen Kwong, and Andrew Y. Ng. 2007. β€œShift-Invariant Sparse Coding for Audio Classification.” In The Twenty-Third Conference on Uncertainty in Artificial Intelligence (UAI2007), 9:8.
Gupta, Pawan, and Marianna Pensky. 2016. β€œSolution of Linear Ill-Posed Problems Using Random Dictionaries.” arXiv:1605.07913 [Math, Stat], May.
Hahn, William Edward, Stephanie Lewkowitz, Daniel C. Lacombe, and Elan Barenholtz. 2015. β€œDeep Learning Human Actions from Video via Sparse Filtering and Locally Competitive Algorithms.” Multimedia Tools and Applications 74 (22): 10097–110.
Han, Bin, Zhenpeng Zhao, and Xiaosheng Zhuang. 2016. β€œDirectional Tensor Product Complex Tight Framelets with Low Redundancy.” Applied and Computational Harmonic Analysis, Sparse Representations with Applications in Imaging Science, Data Analysis, and Beyond, Part IISI: ICCHAS Outgrowth, part 2, 41 (2): 603–37.
Han, Bin, and Xiaosheng Zhuang. 2015. β€œSmooth Affine Shear Tight Frames with MRA Structure.” Applied and Computational Harmonic Analysis 39 (2): 300–338.
Han, Kyunghee, and Hyejin Shin. n.d. β€œFunctional Linear Regression for Functional Response via Sparse Basis Selection.”
Harte, Christopher, Mark Sandler, and Martin Gasser. 2006. β€œDetecting Harmonic Change in Musical Audio.” In Proceedings of the 1st ACM Workshop on Audio and Music Computing Multimedia, 21–26. AMCMM ’06. New York, NY, USA: ACM.
Henaff, Mikael, Kevin Jarrett, Koray Kavukcuoglu, and Yann LeCun. 2011. β€œUnsupervised Learning of Sparse Features for Scalable Audio Classification.” In ISMIR.
Hoyer, Patrik O. n.d. β€œNon-Negative Matrix Factorization with Sparseness Constraints.” Journal of Machine Learning Research 5 (9): 1457–69.
Hoyer, P.O. 2002. β€œNon-Negative Sparse Coding.” In Proceedings of the 2002 12th IEEE Workshop on Neural Networks for Signal Processing, 2002, 557–65.
Huang, Cong, G. L. H. Cheang, and Andrew R. Barron. 2008. β€œRisk of Penalized Least Squares, Greedy Selection and L1 Penalization for Flexible Function Libraries.”
HyvΓ€rinen, Aapo, and Patrik Hoyer. 2000. β€œEmergence of Phase- and Shift-Invariant Features by Decomposition of Natural Images into Independent Feature Subspaces.” Neural Computation 12 (7): 1705–20.
HyvΓ€rinen, Aapo, Jarmo Hurri, and Patrick O. Hoyer. 2009. Natural Image Statistics: A Probabilistic Approach to Early Computational Vision. Vol. 39. Springer Science & Business Media.
Jafari, M. G., and M. D. Plumbley. 2011. β€œFast Dictionary Learning for Sparse Representations of Speech Signals.” IEEE Journal of Selected Topics in Signal Processing 5 (5): 1025–31.
Jaillet, F., R. Gribonval, M. D. Plumbley, and H. Zayyani. 2010. β€œAn L1 Criterion for Dictionary Learning by Subspace Identification.” In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, 5482–85.
Jung, Alexander. 2013. β€œAn RKHS Approach to Estimation with Sparsity Constraints.” In Advances in Neural Information Processing Systems 29.
Kim, H., and H. Park. 2008. β€œNonnegative Matrix Factorization Based on Alternating Nonnegativity Constrained Least Squares and Active Set Method.” SIAM Journal on Matrix Analysis and Applications 30 (2): 713–30.
Koch, Parker, and Jason J. Corso. 2016. β€œSparse Factorization Layers for Neural Networks with Limited Supervision.” arXiv:1612.04468 [Cs, Stat], December.
Koppel, Alec, Garrett Warnell, Ethan Stump, and Alejandro Ribeiro. 2016. β€œParsimonious Online Learning with Kernels via Sparse Projections in Function Space.” arXiv:1612.04111 [Cs, Stat], December.
Lattner, Stefan, Monika Dorfler, and Andreas Arzt. 2019. β€œLearning Complex Basis Functions for Invariant Representations of Audio.” In Proceedings of the 20th Conference of the International Society for Music Information Retrieval, 8.
Lee, Honglak, Alexis Battle, Rajat Raina, and Andrew Y. Ng. 2007. β€œEfficient Sparse Coding Algorithms.” Advances in Neural Information Processing Systems 19: 801.
Lee, Wee Sun, Peter L. Bartlett, and Robert C. Williamson. 1996. β€œEfficient Agnostic Learning of Neural Networks with Bounded Fan-in.” IEEE Transactions on Information Theory 42 (6): 2118–32.
Lewicki, M S, and T J Sejnowski. 1999. β€œCoding Time-Varying Signals Using Sparse, Shift-Invariant Representations.” In NIPS, 11:730–36. Denver, CO: MIT Press.
Lewicki, Michael S., and Terrence J. Sejnowski. 2000. β€œLearning Overcomplete Representations.” Neural Computation 12 (2): 337–65.
Liu, Tongliang, Dacheng Tao, and Dong Xu. 2016. β€œDimensionality-Dependent Generalization Bounds for \(k\)-Dimensional Coding Schemes.” arXiv:1601.00238 [Cs, Stat], January.
Liu, T., and D. Tao. 2015. β€œOn the Performance of Manhattan Nonnegative Matrix Factorization.” IEEE Transactions on Neural Networks and Learning Systems PP (99): 1–1.
Lyu, Hanbaek, Deanna Needell, and Laura Balzano. 2020. β€œOnline Matrix Factorization for Markovian Data and Applications to Network Dictionary Learning.” Journal of Machine Learning Research 21 (251): 1–49.
MailhΓ©, Boris, RΓ©mi Gribonval, Pierre Vandergheynst, and FrΓ©dΓ©ric Bimbot. 2011. β€œFast Orthogonal Sparse Approximation Algorithms over Local Dictionaries.” Signal Processing, Advances in Multirate Filter Bank Structures and Multiscale Representations, 91 (12): 2822–35.
Mairal, Julien, Francis Bach, and Jean Ponce. 2014. Sparse Modeling for Image and Vision Processing. Vol. 8.
Mairal, Julien, Francis Bach, Jean Ponce, and Guillermo Sapiro. 2009. β€œOnline Dictionary Learning for Sparse Coding.” In Proceedings of the 26th Annual International Conference on Machine Learning, 689–96. ICML ’09. New York, NY, USA: ACM.
β€”β€”β€”. 2010. β€œOnline Learning for Matrix Factorization and Sparse Coding.” The Journal of Machine Learning Research 11: 19–60.
Mallat, Stephane G. 1989. β€œMultiresolution Approximations and Wavelet Orthonormal Bases of LΒ²(R).” Transactions of the American Mathematical Society 315 (1): 69–87.
Mallat, StΓ©phane. 2008. A Wavelet Tour of Signal Processing: The Sparse Way. Academic Press.
β€”β€”β€”. 2012. β€œGroup Invariant Scattering.” Communications on Pure and Applied Mathematics 65 (10): 1331–98.
Mallat, StΓ©phane G., and Zhifeng Zhang. 1993. β€œMatching Pursuits with Time-Frequency Dictionaries.” IEEE Transactions on Signal Processing 41 (12): 3397–3415.
Mallat, S., and Z. Zhang. 1992. β€œAdaptive Time-Frequency Decomposition with Matching Pursuits.” In Time-Frequency and Time-Scale Analysis, 1992., Proceedings of the IEEE-SP International Symposium, 7–10.
Marcus, Gary, Adam Marblestone, and Thomas Dean. 2014. β€œThe atoms of neural computation.” Science 346 (6209): 551–52.
Mlynarski, Wiktor. 2013. β€œSparse, Complex-Valued Representations of Natural Sounds Learned with Phase and Amplitude Continuity Priors.” arXiv Preprint arXiv:1312.4695.
Mondal, Debashis, and Donald B. Percival. 2010. β€œM-Estimation of Wavelet Variance.” Annals of the Institute of Statistical Mathematics 64 (1): 27–53.
MΓΈrup, Morten, Mikkel N. Schmidt, and Lars K. Hansen. 2007. β€œShift Invariant Sparse Coding of Image and Music Data.” Journal of Machine Learning Research.
Ngiam, Jiquan, Zhenghao Chen, Sonia A. Bhaskar, Pang W. Koh, and Andrew Y. Ng. 2011. β€œSparse Filtering.” In Advances in Neural Information Processing Systems 24, edited by J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, 1125–33. Curran Associates, Inc.
Olshausen, B. A., and D. J. Field. 1996. β€œNatural image statistics and efficient coding.” Network (Bristol, England) 7 (2): 333–39.
Olshausen, Bruno A., and David J. Field. 1996. β€œEmergence of Simple-Cell Receptive Field Properties by Learning a Sparse Code for Natural Images.” Nature 381 (6583): 607–9.
Olshausen, Bruno A, and David J Field. 2004. β€œSparse Coding of Sensory Inputs.” Current Opinion in Neurobiology 14 (4): 481–87.
Opsomer, Jean, Yuedong Wang, and Yuhong Yang. 2001. β€œNonparametric Regression with Correlated Errors.” Statistical Science 16 (2): -134-153.
Oyallon, Edouard, Eugene Belilovsky, and Sergey Zagoruyko. 2017. β€œScaling the Scattering Transform: Deep Hybrid Networks.” arXiv Preprint arXiv:1703.08961.
Pfister, Luke, and Yoram Bresler. 2017. β€œAutomatic Parameter Tuning for Image Denoising with Learned Sparsifying Transforms.” In.
Plumbley, Mark D., Samer A. Abdallah, Thomas Blumensath, and Michael E. Davies. 2006. β€œSparse Representations of Polyphonic Music.” Signal Processing, Sparse Approximations in Signal and Image ProcessingSparse Approximations in Signal and Image Processing, 86 (3): 417–31.
Qian, Shie, and Dapang Chen. 1994. β€œSignal Representation Using Adaptive Normalized Gaussian Functions.” Signal Processing 36 (1): 1–11.
Ravishankar, Saiprasad, and Yoram Bresler. 2015. β€œEfficient Blind Compressed Sensing Using Sparsifying Transforms with Convergence Guarantees and Application to MRI.” arXiv:1501.02923 [Cs, Stat], January.
Rubinstein, Ron, A.M. Bruckstein, and Michael Elad. 2010. β€œDictionaries for Sparse Representation Modeling.” Proceedings of the IEEE 98 (6): 1045–57.
Rubinstein, Ron, Michael Zibulevsky, and Michael Elad. 2008. β€œEfficient Implementation of the K-SVD Algorithm Using Batch Orthogonal Matching Pursuit.” CS Technion.
Saad, Yousef. 2003. Iterative Methods for Sparse Linear Systems: Second Edition. 2nd ed. SIAM.
Schniter, Philip, Lee C. Potter, and Justin Ziniel. 2008. β€œFast Bayesian Matching Pursuit.” In 2008 Information Theory and Applications Workshop, 326–33. San Diego, CA, USA: IEEE.
Shen, Z. 2010. β€œWavelet Frames and Image Restorations.” In Scopus, 2834–63. World Scientific.
Simoncelli, Eero P, and Bruno A Olshausen. 2001. β€œNatural Image Statistics and Neural Representation.” Annual Review of Neuroscience 24 (1): 1193–1216.
Smith, Evan C., and Michael S. Lewicki. 2006. β€œEfficient Auditory Coding.” Nature 439 (7079): 978–82.
Soh, Yong Sheng, and Venkat Chandrasekaran. 2017. β€œA Matrix Factorization Approach for Learning Semidefinite-Representable Regularizers.” arXiv:1701.01207 [Cs, Math, Stat], January.
Sprechmann, Pablo, Joan Bruna, and Yann LeCun. 2014. β€œAudio Source Separation with Discriminative Scattering Networks.” arXiv:1412.7022 [Cs], December.
Sulam, Jeremias, Aviad Aberdam, Amir Beck, and Michael Elad. 2020. β€œOn Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks.” IEEE Transactions on Pattern Analysis and Machine Intelligence 42 (8): 1968–80.
Tang, Yuan Y. 2000. Wavelet Theory and Its Application to Pattern Recognition. World Scientific.
Torrence, Christopher, and Gilbert P Compo. 1998. β€œA Practical Guide to Wavelet Analysis.” Bulletin of the American Meteorological Society 79 (1): 61–78.
ToΕ‘iΔ‡, Ivana, and Pascal Frossard. 2011. β€œDictionary Learning: What Is the Right Representation for My Signal?” IEEE Signal Processing Magazine 28 (2): 27–38.
Tropp, J. A., and S. J. Wright. 2010. β€œComputational Methods for Sparse Solution of Linear Inverse Problems.” Proceedings of the IEEE 98 (6): 948–58.
Tropp, Joel A. 2006. β€œAlgorithms for Simultaneous Sparse Approximation. Part II: Convex Relaxation.” Signal Processing, Sparse Approximations in Signal and Image ProcessingSparse Approximations in Signal and Image Processing, 86 (3): 589–602.
Tsaig, Yaakov, and David L. Donoho. 2006a. β€œBreakdown of Equivalence Between the Minimal -Norm Solution and the Sparsest Solution.” Signal Processing, Sparse Approximations in Signal and Image ProcessingSparse Approximations in Signal and Image Processing, 86 (3): 533–48.
β€”β€”β€”. 2006b. β€œExtensions of Compressed Sensing.” Signal Processing, Sparse Approximations in Signal and Image ProcessingSparse Approximations in Signal and Image Processing, 86 (3): 549–71.
TΓΌrkmen, Ali Caner. 2015. β€œA Review of Nonnegative Matrix Factorization Methods for Clustering.” arXiv:1507.03194 [Cs, Stat], July.
Vainsencher, Daniel, Shie Mannor, and Alfred M. Bruckstein. 2011. β€œThe Sample Complexity of Dictionary Learning.” Journal of Machine Learning Research 12 (Nov): 3259–81.
Vetterli, Martin. 1999. β€œWavelets: Approximation and Compression–a Review.” In AeroSense’99, 3723:28–31. International Society for Optics and Photonics.
Wang, Yu Guang, and Houying Zhu. 2017. β€œLocalized Tight Frames and Fast Framelet Transforms on the Simplex.” arXiv:1701.01595 [Cs, Math], January.
Wang, Yu Guang, and Xiaosheng Zhuang. 2016. β€œTight Framelets and Fast Framelet Transforms on Manifolds.” arXiv:1608.04026 [Math], August.
Wang, Yu-Xiang, Alex Smola, and Ryan J. Tibshirani. 2014. β€œThe Falling Factorial Basis and Its Statistical Applications.” In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, 730–38. ICML’14. Beijing, China: JMLR.org.
Weidmann, Claudio, and Martin Vetterli. 2012. β€œRate Distortion Behavior of Sparse Sources.” IEEE Transactions on Information Theory 58 (8): 4969–92.
Wohlberg, Brendt. 2017. β€œSPORCO: A Python Package for Standard and Convolutional Sparse Representations.” In.
Wright, John, and Yi Ma. 2022. High-dimensional data analysis with low-dimensional models: Principles, computation, and applications. S.l.: Cambridge University Press.
Yaghoobi, M., L. Daudet, and M. E. Davies. 2009. β€œParametric Dictionary Design for Sparse Coding.” IEEE Transactions on Signal Processing 57 (12): 4800–4810.
Yaghoobi, M., Sangnam Nam, R. Gribonval, and M.E. Davies. 2013. β€œConstrained Overcomplete Analysis Operator Learning for Cosparse Signal Modelling.” IEEE Transactions on Signal Processing 61 (9): 2341–55.
Yuan, Xiaotong, Ping Li, and Tong Zhang. 2014. β€œGradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization.” In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, 127–35. Beijing, China: JMLR.org.
Zhang, Kaiqi, and Yu-Xiang Wang. 2022. β€œDeep Learning Meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?” arXiv.
Zhou, Mingyuan, Haojun Chen, John Paisley, Lu Ren, Guillermo Sapiro, and Lawrence Carin. 2009. β€œNon-Parametric Bayesian Dictionary Learning for Sparse Image Representations.” In Proceedings of the 22nd International Conference on Neural Information Processing Systems, 22:2295–2303. NIPS’09. Red Hook, NY, USA: Curran Associates Inc.
Zhuang, Xiaosheng. 2016. β€œDigital Affine Shear Transforms: Fast Realization and Applications in Image/Video Processing.” SIAM Journal on Imaging Sciences 9 (3): 1437–66.

Download bibliography

No comments yet. Why not leave one?

GitHub-flavored Markdown & a sane subset of HTML is supported.