# Neurons

Neural networks made of real neurons, in functioning brains

November 3, 2014 — February 14, 2022

How do brains work?

I mean, how do brains work at the level slightly higher than a synapse, but much lower than, e.g. psychology. “How is thought done?” etc.

Notes pertaining to *large, artificial* networks are filed under artificial neural networks. The messy, biological end of the stick is here. Since brains seem to be the seat of the most flashy and important bit of the computing taking place in our bodies, we understandably want to know how they works, in order to

- fix Alzheimers disease
- steal cool learning tricks
- endow the children of elites with superhuman mental prowess to cement their places as Übermenschen fit to rule the thousand year Reich
- …or whatever.

Real brains are different to the “neuron-inspired” computation of the simulacrum in many ways, not just the usual difference between model and reality. The similitude between “neural networks” and neurons is intentionally weak for reasons of convenience.

For one example, most simulated neural networks are based on a continuous activation potential and discrete time, unlike spiking biological ones which are driven by discrete events in continuous time.

Also real brains support heterogeneous types of neuron, have messier layer organisation, use less power, don’t have well-defined backpropagation (or not in the same way), and many other things that I as a non-specialist do not know.

To learn more about:

- Just saw a talk by Dan Cireșan in which he mentioned the importance of “Foveation” - blurring the edge of an image when training classifiers on it, thus encouraging us to ignore stuff in order to learn better. What is, rigorously speaking, happening there? Nice actual crossover between biological neural nets and fake ones.
- Algorithmic statistics of neurons sounds interesting.
- Modelling mind as machine learning.

## 1 Fun data

## 2 How computationally complex is a neuron?

Empirically quantifying computation is hard, but people try to do it all the time for brains. Classics try to estimate structure in neural spike trains, (Crumiller et al. 2011; Haslinger, Klinkner, and Shalizi 2010; Nemenman, Bialek, and de Ruyter van Steveninck 2004) often by empirical entropy estimates.

If we are prepared to accept “size of a neural network needed to approximate X” as an estimate of the complexity of X, then there are some interesting results: Allison Whitten, How Computationally Complex Is a Single Neuron? (Beniaguev, Segev, and London 2021). OTOH, finding the *smallest* neural network that can approximate something is itself computationally hard and not in general even easy to check.

## 3 Pretty pictures of neurons

The names I am looking for here for beutiful hand drawn early neuron diagrams are Camillo Golgi and Santiago Ramón y Cajal, especially the latter.

## 4 References

*Neural Computation*.

*Journal of Neuroscience Methods*.

*Neuron*.

*Trends in Cognitive Sciences*.

*Neural Computation*.

*PLoS Comput Biol*.

*Nature Reviews Neuroscience*.

*PLoS Comp. Biol.*

*Journal of Psychopharmacology*.

*Frontiers in Neuroscience*.

*Frontiers in Cellular Neuroscience*.

*Neural Computation*.

*Cognitive Science*.

*Cognition*.

*Annals of the New York Academy of Sciences*.

*Journal of The Royal Society Interface*.

*The Annals of Statistics*.

*Current Biology*.

*Neural Computation*.

*Neural Computation*.

*Physical Review E*.

*PLoS Comput Biol*.

*PLOS Computational Biology*.

*Annual Review of Statistics and Its Application*.

*PLoS ONE*.

*Cell*.

*arXiv Preprint arXiv:1508.06818*.

*BMC Neuroscience*.

*Advances in Neural Information Processing Systems*.

*Science*.

*Physical Review E*.

*Nature*.

*Current Opinion in Neurobiology*.

*Neural Computation*.

*Parallel Distributed Processing: Explorations in the Microstructure of Cognition*. 1986.

*Scientific Reports*.

*Neuroscience*.

*PLoS ONE*.

*arXiv:1610.06551 [Stat]*.

*Annual Review of Neuroscience*.

*Neural Computation*.

*Advances in Neural Information Processing Systems*.

*Neural Computation*.

*Nature*.

*Proceedings of the National Academy of Sciences*.

*Phys. Rev. Lett.*

*Neural Computation*.

*PLoS Computational Biology*.

*Frontiers in Human Neuroscience*.