Neural networks which are more biologically plausible than what we typically refer to as (artificial) neural networks. Synthetic brains, if you’d like.
1 Forward-forward networks
Neural networks without backprop are “more” biologically plausible. Here is a class of such networks (Hinton, n.d.; Ren et al. 2022).
The aim of this paper is to introduce a new learning procedure for neural networks and to demonstrate that it works well enough on a few small problems to be worth serious investigation. The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself. Each layer has its own objective function which is simply to have high goodness for positive data and low goodness for negative data. The sum of the squared activities in a layer can be used as the goodness but there are many other possibilities, including minus the sum of the squared activities. If the positive and negative passes can be separated in time, the negative passes can be done offline, which makes the learning much simpler in the positive pass and allows video to be pipelined through the network without ever storing activities or stopping to propagate derivatives.
2 NEMO
Christos Papadimitriou, How Does the Brain Create Language?
There is little doubt that cognitive phenomena are the result of neural activity. However, there has been slow progress toward articulating an overarching computational theory of how exactly this happens. I will discuss a simplified mathematical model of the brain, which we call NEMO, involving brain areas, spiking neurons, random synapses, local inhibition, Hebbian plasticity, and long-range interneurons. Emergent behaviours of the resulting dynamical system—established both analytically and through simulations—include assemblies of neurons, sequence memorization, one-shot learning, and universal computation. NEMO can also be seen as a software-based neuromorphic system that can be simulated efficiently at the scale of tens of millions of neurons, emulating certain high-level cognitive phenomena such as planning and parsing of natural language. I will describe current work aiming at creating through NEMO a neuromorphic language organ: a neural tabula rasa which, on input consisting of a modest amount of grounded language, is capable of language acquisition: lexicon, syntax, semantics, comprehension, and generation. Finally, and on the plane of scientific methodology, I will argue that experimenting with such brain-like devices, devoid of backpropagation, can reveal novel avenues to learning, and may end up advancing AI.
3 Spiking
TBD