# Natural language processing

Automatic processing of words and sentences and such

January 11, 2018 — September 16, 2021

Computational language translation, parsing, search, generation and understanding.

A mare’s nest of intersecting computational philosophical and mathematical challenges (e.g. semantics, grammatical inference, language complexity, learning theory) that humans seem to be able to handle subconsciously and which we therefore hope to train machines on. Moreover it is a problem of great commercial benefit so it is likely we can muster the resources to tackle it. The interesting thing right now is the NLP explosion, where it looks like if anything has a good chance of producing artificial general intelligence it might be neural NLP, where certain architectures (especially highly evolved attention mechanisms) are producing eerily good results (Brown et al. 2020).

## 1 What is Natural Language Processing?

- Sebastian Ruder, Recent history of NLP a.k.a “how natural language processing turned into a deep learning thing too”
- See also Sebastian’s newsletter
- Peter Norvig on Chomsky and statistical versus explanatory models of natural language syntax. Full of sick burns.
- I guess the famous Stochastic Parrots paper (Bender et al. 2021) is a new kind of rejoinder, with a particular focus on transformers

See also Feral, Thomas Urquhart…

## 2 Biological basis of language

See biology of language.

## 3 Software

See NLP software.

## 4 References

*Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-Gram Model? On the Future of Language Modeling for HLT*. WLM ’12.

*Handbook of Formal Languages, Vol. 1*.

*Modern Information Retrieval*.

*Proceedings of the National Academy of Sciences*.

*Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency*.

*Journal of Machine Learning Research*.

*Handbook of Theoretical Computer Science, Vol. A: Algorithms and Complexity*.

*arXiv:2002.11319 [Cs, q-Bio]*.

*PLoS Biol*.

*IEEE Transactions on Computers*.

*arXiv:1102.1808 [Cs]*.

*arXiv:2005.14165 [Cs]*.

*Grammatical Inference: Algorithms and Applications*.

*Statistical Language Learning*.

*Trends in Cognitive Sciences*.

*EMNLP 2014*.

*Algorithmic Learning Theory*. Lecture Notes in Computer Science.

*Machine Learning: ECML 2006*. Lecture Notes in Computer Science 4212.

*Grammatical Inference: Algorithms and Applications*. Lecture Notes in Computer Science 4201.

*IJCAI 2020*.

*Advances in Neural Information Processing Systems 14*.

*Information and Control*.

*Syntactic Pattern Recognition: An Introduction*.

*arXiv:1506.02516 [Cs]*.

*J. ACM*.

*Introduction to Automata Theory, Languages and Computation*.

*Algorithmic Learning Theory*. Lecture Notes in Computer Science 4264.

*Theoretical Computer Science*, Algorithmic Learning Theory,.

*Proceedings of the Eighteenth International Conference on Machine Learning*. ICML ’01.

*IJCAI 2020*.

*arXiv:1506.00019 [Cs]*.

*Probabilistic Linguistics*.

*Introduction to Information Retrieval*.

*Foundations of Statistical Natural Language Processing*.

*Eleventh Annual Conference of the International Speech Communication Association*.

*arXiv:1309.4168 [Cs]*.

*arXiv:1705.01509 [Cs]*.

*Proceedings of the 12th Biennial European Conference on Artificial Intelligence (ECAI-96), Workshop on Extended Finite State Models of Language*.

*Computer Speech & Language*.

*Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014)*.

*Brain and Language*, The Neurobiology of Syntax,.

*arXiv:2003.07082 [Cs]*.

*Annual Review of Statistics and Its Application*.

*arXiv:1811.12143 [Cs, Stat]*.

*Proceedings of the National Academy of Sciences of the United States of America*.

*Journal of Machine Learning Research*.

*Information Retrieval*.

*ACM Comput. Surv.*

*Journal of Universal Computer Science*.