Thomas Natschläger, December 1998
Each living creature has to act in a suitable way in response to the various situations it encounters in its environment. Nowadays it is accepted that the nervous system (including the spinal cord as well as the neocortex) controls the behavior. In a simplified picture one could view the nervous system as a device which receives input from various senses like auditory input from the ears and produces (computes) some output in the form of movement or speech. To understand how this computations are performed is one of the most interesting and challenging tasks in science. Thus it is not surprising that the field of neuroscience1 attracts a lot of researchers not only from biology but also from physics and computer science.
In this article we try to give the reader an intuition what computer scientists (especially at the Institute for Theoretical Computer Science at the Technical University of Graz) can contribute to the question how the brain works. The central questions of a theoretical computer scientist are: How much (computational) power is needed to compute a certain function2? Which functions can be computed by a given „device“? In the context of this article a concrete question could be: How many nerve cells are needed to be able to distinguish between a picture of your mother and your father? However, the questions addressed in this article are still on a more abstract level since theoretical computer scientists try to make general statements about computation. An article3 written by W. Maass4 investigates this kind of questions in a more mathematical manner.
We start with a short cartoon kind description of the human nervous system which provides the basis of all models of neural networks (for a more detailed description see the textbook by Sheperd, 19945). The nervous system consists on the order of 1010 of nerve cells, so called neurons (see Figure 1). Each of these neurons is connected to about 10000 other neurons via synapses. This gigantic network of neurons and synapses builds up the „hardware“ (or often called „wetware“) which carries out the computations underlying the human behavior.
One of the most important parts of a neuron is its membrane, which separates the interior and the exterior of a neuron spatially and electrically. A prominent feature of the cell membrane is, that it maintains a voltage difference between the interior and the exterior. This so called membrane potential is subject to change if a neuron receives some input via its synapses (manly located at the dendritic tree of a neuron) from other neurons. There are two basic types of input: excitatory input (via excitatory synapses) increases the membrane potential whereas inhibitory input (via inhibitory synapses) decreases the membrane potential (see Figure 2). If enough excitatory input accumulates the membrane potential at the axon hillock eventually reaches a threshold. If this event occurs the membrane follows a stereotyped trajectory, the so called action potential or spike. This spike travels along the axon (the “output wire” of a neuron) to all its terminals which connects the axon to other neurons (postsynaptic neurons) via synapses. A complicated chemical process transforms the spike into a change of the membrane potential of the postsynaptic neuron, the so called postsynaptic potential (PSP) which comprise the above mentioned input of the next neuron.
Figure 1: Schematic picture of a nerve cell.
Figure 2: Drawing of the canonical shape of a postsynaptic potential and the action potential. Excitatory synapses produce an EPSP whereas inhibitory synapses produce an IPSP.
If one wants to understand how the nervous system computes a function one has to think about how information about the environment or internal states is represented and transmitted. The fact that the shape of an action potential is always the same one can exclude the possibility that the voltage trajectory of an action potential carries relevant information. Thus a central question in the field of neuroscience is how neurons encode information in the sequence of action potential they emit (see Figure 3). In this article we characterize neural network models by the assumptions about the encoding scheme.
1943 McCulloch and Pitts proposed the first neuron model: the threshold gate. The characteristics of their model was that they treated a neuron as a binary device. That is they distinguished only between the occurrence and absence of a spike. The threshold gate is used as building block for various network types including multilayer perceptrons, Hopfield networks and the Boltzman machine6. It turned out that the threshold gate is a computational powerful device. That is one can compute complex functions with rather small networks made up of threshold gates7. From the theoretical point of view the threshold gate is a very interesting model but it is unlikely that real biological systems use such a binary encoding scheme. A prerequisite for such a binary coding scheme is a kind of global clocking mechanism but it is very unlikely that such a mechanism exists in biological systems.
Another possibility is that the number of spikes per second (called the firing rate) encodes relevant information. This idea lead to a model neuron known as sigmoidal gate. The output of a sigmoidal gate is a number which is thought to represent the firing rate of the neuron. There exists a huge amount of literature which discusses in detail all aspects of this kind of neural network models8. We just want to note that networks of sigmoidal gates can in principle compute any analog function and that along with this type of models the question of learning in neural networks was intensively investigated for the first time.
It is well known that firing rates play an important role in the nervous system especially in the primary sensory areas where low level information processing takes place. However, recent experiments revealed that the human nervous system is able to perform complex visual tasks in time intervals as short as 150–ms (1–ms = 0.001 second)9. The pathway from the retina to higher areas in the neocortex along which a visual stimuli is processed consists of about 10 “processing stages”. Furthermore the firing rates in the areas involved in such complex computations are well below 100 spikes per second (see Figure 3). But to estimate such low firing rates one has to wait at least 30 to 50–ms which is a contradiction to the finding that a computation involving 10 “processing stages” can be carried out within 150°ms. Further experimental results indicate that some biological neural systems indeed use the exact timing of individual spikes which further confirms the idea that the firing rate alone does not carry all the relevant information10.
Figure 3: Simultaneous recordings of 30 neurons in the visual cortex of a monkey11. Each tick mark denotes the occurrence of a spike. The gray shaded bar marks a time interval of 150–ms. As noted in the text such a short time interval suffices to perform complex computations. However, one sees that there are only a few spikes within this time interval.
These results from experimental neurobiology gave rise to a new class of neural network models where one also incorporates the timing of individual spikes12. Thus time plays a central role in SNNs whereas in most other neural network models there is even no notion of time.
A simple way how to use time to encode information is depicted in Figure 4. The inputs to a network of spiking neurons are encoded in the time differences between a spike and a reference time point. The output is also encoded by the time difference between an output spike and some other reference time point. The reference time points can be given by other spikes but also through other mechanisms, e.g. background oscillations. There exists empirical evidence that such a type of encoding is used for example in the olfactory cortex13.
Figure 4: A SNN using a simple form of temporal coding. An analog input x is encoded by a time difference Tin-cx,. where Tin can be given by another spike and c is an arbitrary scaling factor. The output y is also encoded by a difference Tout-cy.
What class of functions can be computed by a SNN using this type of temporal coding? An important result regarding this question is reported by Maass, 199714. In this work it is shown that a small network of spiking neurons using temporal coding can easily compute the same functions as sigmoidal gate. As noted above a network of sigmoidal function can in principle compute any given function and hence also a network of spiking neurons can compute any given function regardless how complex this function might be. An interesting detail of this result is that the time which the SNN needs to perform such a computation is in the order of 100 ms. At least from a theoretical point of view this result shows a possibility how complex computations in the human nervous system might be organized. In another article it shown that SNNs have even more computational power than networks of sigmoidal gates. That means that there are function which can easily be computed by a small network of spiking neurons but one needs a large net of sigmoidal gates3.
In the previous subsection we pointed out that SNNs have enough computational power to compute arbitrary functions. However the proofs underlying these results are done on a rather abstract level and furthermore do not provide hints how to construct a network of spiking neurons which computes a concrete function. But on the other hand if it would had turned out that an SNN using temporal coding has very low computational power this would rule out such a temporal code as potential mechanism used in real biological systems.
The question about the computational power is not the only one which is investigated at the Institute for Theoretical Computer Science. We are also interested how concrete function which are likely to be performed in real biological neural systems could be implemented efficiently in a network of spiking neurons. An example of such a function is the associative recall of stored memories. In the work of Maass and Natschläger, 199715 it is show how such a associative memory can be implemented with a biological rather realistic network of spiking neurons. Another example of a concrete problem is given by Hopfield, 1995. In his work he comes up with a hypothesis how a simple form of pattern recognition might be implemented with a network of spiking neurons13. A good survey about models for concrete functions or subsystems of the nervous systems can be found in the textbook by Koch and Segev, 199816. Such hypothesis about the functionality of certain subsystems of the nervous system may led to new experiments the results of which allow one to make more realistic models. Thus the interplay between the experiments done by biologists and theoretical models made by physicists or computer scientists is a very fruitful interdisciplinary interplay.
One of the most challenging problems in neuroscience is the question of learning. Although substantial progress has been made regarding learning in neural networks of the second generation these “learning schemes” do not directly apply to biological systems (see the textbook by Bishop8 for an overview). What is commonly accepted is that learning takes place in the connections between to neurons, i.e. the synapses are the place where changes are made if we learn something. Until recently it was assumed that the changes in a synapse depend basically on the relation between the firing rates of the two neurons a synapse connects. However results reported by Markram et. al.17 indicate that the changes in a synapse depend strongly on the timing of the individual spikes the two neurons produce. Based on this experimental results several models have been developed how such a time depended learning scheme my be used in a network of spiking neurons to learn something meaningful. An example is the work of Natschläger and Ruf, 199718 where it is shown that such a learning scheme might be useful to extract important features from a stimulus. Although there exist many other hypothesis regarding such a time depended learning scheme non of these hypothesis can explain high level memory phenomena. This is a characteristic phenomena in the whole field of neuroscience. One knows a lot about biological neural systems from the molecular to the cellular level but we have basically no idea how these bits and pieces together form the most powerful computational device which we are aware of.
Network models of the second generation turned out to be a very useful tool for a wide variety of problems8. This gave rise to the development of fast hardware implementations of such models to be able to solve large problems in a reasonable time. One approach is to use analogue devices instead of microprocessors to carry out the computations. It turned out to be advantageous if such “silicon neurons” communicate via voltage pulses. An example of such a hardware implementation is EPSILON II, a chip developed at the University of Edinburgh. This chip does not only use pulses for communication purposes but also employees the timing of individual pulses for computation. This example shows that the principles underlying the information processing in biological neural systems also inspires engineers when designed new hardware. In the last view years people all over the world started to develop such “neuro-inspired” (also called neuromorphic) systems. The idea behind this efforts is to reveal solutions for problems that can not be solved yet efficiently but it is evident that mother nature has developed very efficient solutions for.
We briefly addressed in this article a new class of models for neural networks which are more closely related to real biological neural systems then the models of the first and second generation. We have seen how questions about the computational power of such network lead to possible explanations how fast and complex computations in the nervous system might organized. We also discussed how the question about the implementation of concrete function gives rise to hypothesis for the functionality of certain subsystems of the nervous system. We also touched one of the most interesting aspects of the human behavior namely the ability to from experience. As indicated in the subsection about learning we know a lot about the low level processes in the nervous system but there is this big missing link between the low level physiological processes and higher level phenomena like long term memory. To bridge this gap between low and high level processes will be the challenge not only of biologists but also for researchers from other disciplines like physicists and computer scientists.
1 Researchers in the field of neuroscience try to answer the question: how does the brain work?
2 In the field of theoretical computer science a function is a mapping from inputs to outputs.
3 W. Maass, „Networks of spiking neurons: The third generation of neural network models“, Neural Networks, 1997.
4 Wolfgang Maass is the head of the Institute for Theoretical Computer Science at the Technical University of Graz, Austria.
5 G. M. Shepard, Neurobiology, Oxford University Press, 1994.
6 A. Anderson and E. Rosenfeld, Neurocomputing, MIT-Press, Cambridge, 1998.
7 K. Y. Siu, V. Roychowdhury and T. Kailath, Discrete Neural Computation: A Theoretical Foundation, MIT-Press, Cambridge 1995.
8 C. M. Bishop, Neural Networks for Pattern Recognition, Clarendon Press, Oxford, 1995.
9 S. Thorpe, D. Fize and C. Marlot, “Speed of processing in the human visual system”, Nature, 381:520-522, 1996.
10 F. Rieke, D. Warland, W. Bialek and Van Stevenik , SPIKES: Exploring the Neural Code, MIT-Press, Cambridge 1996.
11 J. Krüger and F. Aiple, “Multielectrode investigation of monkey striate cortex. Spike train correlations in the infragranular layers”, J. of Neurophysiology, 60:798-828, 1988.
12 W. Maass and C. M. Bishop, Pulsed Neural Networks, MIT-Press, Cambridge ,1998.
13 J. J. Hopfield, “Pattern recognition computation using action potential timing for stimulus representation”, Nature, 376:3-36, 1995.
14 W. Maass, “Fast sigmoidal neural networks via spiking neurons”, Neural Computation, 9:279-304, 1997.
15 W. Maass and T. Natschläger, “Networks of spiking Neurons can emulate arbitrary Hopfield nets in temporal coding”, Network: Computations in Neural Systems, 8(4):355-372, 1997.
16 C. Koch and I. Segev, Methods in Neural Modeling: From Ions to Networks, MIT-Press, Cambridge, 1998.
17 H. Markram, J. Lübke, M. Frotscher, A. Roth and B. Sakmann, “Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs”, Science, 275:213-215, 1997.
18 T. Natschläger and B. Ruf, “Spatial and temporal pattern analysis with via spiking neurons”, Network: Computations in Neural Systems, 9(3):319-332, 1998.