Read The Singularity Is Near: When Humans Transcend Biology Online
Authors: Ray Kurzweil
Tags: #Non-Fiction, #Fringe Science, #Retail, #Technology, #Amazon.com
The massive parallelism of the human brain is the key to its pattern-recognition ability, which is one of the pillars of our species’ thinking. Mammalian neurons engage in a chaotic dance (that is, with many apparently random interactions), and if the neural network has learned its lessons well, a stable pattern will emerge, reflecting the network’s decision. At the present, parallel designs for computers are somewhat limited. But there is no reason why functionally equivalent nonbiological re-creations of biological neural networks cannot be built using these principles. Indeed, dozens of efforts around the world have already succeeded in doing so. My own technical field is pattern recognition, and the projects that I have been involved in for about forty years use this form of trainable and nondeterministic computing.
Many of the brain’s characteristic methods of organization can also be effectively simulated using conventional computing of sufficient power. Duplicating the design paradigms of nature will, I believe, be a key trend in future computing. We should keep in mind, as well, that digital computing can be functionally equivalent to analog computing—that is, we can perform all of the functions of a hybrid digital-analog network with an all-digital computer. The reverse is not true: we can’t simulate all of the functions of a digital computer with an analog one.
However, analog computing does have an engineering advantage: it is potentially thousands of times more efficient. An analog computation can be
performed by a few transistors or, in the case of mammalian neurons, specific electrochemical processes. A digital computation, in contrast, requires thousands or tens of thousands of transistors. On the other hand, this advantage can be offset by the ease of programming (and modifying) digital computer-based simulations.
There are a number of other key ways in which the brain differs from a conventional computer:
The software programs for an operating system, language compilers, and assemblers are reasonably complex, but modeling a particular program—for example, a speech-recognition program based on Markov modeling—may be described in only a few pages of equations. Nowhere in such a description would be found the details of semiconductor physics. A similar observation also holds true for the brain. A particular neural arrangement that detects a particular invariant visual feature (such as a face) or that performs a band-pass filtering (restricting input to a specific frequency range) operation on auditory information or that evaluates the temporal proximity of two events can be described with far greater simplicity than the actual physics and chemical relations controlling the neurotransmitters and other synaptic and dendritic variables involved in the respective processes. Although all of this neural complexity will have to be carefully considered before advancing to the next higher level (modeling the brain), much of it can be simplified once the operating principles of the brain are understood.
Trying to Understand Our Own Thinking
The Accelerating Pace of Research
We are now approaching the knee of the curve (the period of rapid exponential growth) in the accelerating pace of understanding the human brain, but our attempts in this area have a long history. Our ability to reflect on and build models of our thinking is a unique attribute of our species. Early mental models were of necessity based on simply observing our external behavior (for example, Aristotle’s analysis of the human ability to associate ideas, written 2,350 years ago).
19
At the beginning of the twentieth century we developed the tools to examine the physical processes
inside
the brain. An early breakthrough was the measurement of the electrical output of nerve cells, developed in 1928 by neuroscience pioneer E. D. Adrian, which demonstrated that there were electrical processes taking place inside the brain.
20
As Adrian wrote, “I had arranged electrodes on the optic nerve of a toad in connection with some experiments on the retina. The room was nearly dark and I was puzzled to hear repeated noises in the loudspeaker attached to the amplifier, noises indicating that a great deal of impulse activity was going on. It was not until I compared the noises with my own movements around the room that I realized I was in the field of vision of the toad’s eye and that it was signaling what I was doing.”
Adrian’s key insight from this experiment remains a cornerstone of neuroscience today: the frequency of the impulses from the sensory nerve is proportional to the intensity of the sensory phenomena being measured. For example, the higher the intensity of light, the higher the frequency (pulses per second) of the neural impulses from the retina to the brain. It was a student of Adrian, Horace Barlow, who contributed another lasting insight, “trigger features” in neurons, with the discovery that the retinas of frogs and rabbits had single neurons that would trigger on “seeing” specific shapes, directions, or velocities. In other words, perception involves a series of stages, with each layer of neurons recognizing more sophisticated features of the image.
In 1939 we began to develop an idea of how neurons perform: by accumulating (adding) their inputs and then producing a spike of membrane conductance (a sudden increase in the ability of the neuron’s membrane to conduct a signal) and voltage along the neuron’s axon (which connects to other neurons via a synapse). A. L. Hodgkin and A. F. Huxley
described their theory of the axon’s “action potential” (voltage).
21
They also made an actual measurement of an action potential on an animal neural axon in 1952.
22
They chose squid neurons because of their size and accessible anatomy.
Building on Hodgkin and Huxley’s insight W. S. McCulloch and W. Pitts developed in 1943 a simplified model of neurons and neural nets that motivated a half century of work on artificial (simulated) neural nets (using a computer program to simulate the way neurons work in the brain as a network). This model was further refined by Hodgkin and Huxley in 1952. Although we now realize that actual neurons are far more complex than these early models, the original concept has held up well. This basic neural-net model has a neural “weight” (representing the “strength” of the connection) for each synapse and a nonlinearity (firing threshold) in the neuron soma (cell body).
As the sum of the weighted inputs to the neuron soma increases, there is relatively little response from the neuron until a critical threshold is reached, at which point the neuron rapidly increases the output of its axon and fires. Different neurons have different thresholds. Although recent research shows that the actual response is more complex than this, the McCulloch-Pitts and Hodgkin-Huxley models remain essentially valid.
These insights led to an enormous amount of early work in creating artificial neural nets, in a field that became known as connectionism. This was perhaps the first self-organizing paradigm introduced to the field of computation.
A key requirement for a self-organizing system is a nonlinearity: some means of creating outputs that are not simple weighted sums of the inputs. The early neural-net models provided this nonlinearity in their replica of the neuron nucleus.
23
(The basic neural-net method is straightforward.)
24
Work initiated by Alan Turing on theoretical models of computation around the same time also showed that computation requires a nonlinearity. A system that simply creates weighted sums of its inputs cannot perform the essential requirements of computation.
We now know that actual biological neurons have many other nonlinearities resulting from the electrochemical action of the synapses and the morphology (shape) of the dendrites. Different arrangements of biological neurons can perform computations, including adding, subtracting, multiplying, dividing, averaging, filtering, normalizing, and thresholding signals, among other types of transformations.
The ability of neurons to perform multiplication is important because
it allows the behavior of one network of neurons in the brain to be modulated (influenced) by the result of computations of another network. Experiments using electrophysiological measurements on monkeys provide evidence that the rate of signaling by neurons in the visual cortex when processing an image is increased or decreased by whether or not the monkey is paying attention to a particular area of that image.
25
Human fMRI studies have also shown that paying attention to a particular area of an image increases the responsiveness of the neurons processing that image in a cortical region called V5, which is responsible for motion detection.
26
Another key breakthrough occurred in 1949 when Donald Hebb presented his seminal theory of neural learning, the “Hebbian response”: if a synapse (or group of synapses) is stimulated repeatedly, that synapse becomes stronger. Over time this conditioning of the synapse produces a learning response. The connectionism movement designed simulated neural nets based on this model, and this gave momentum to such experiments during the 1950s and 1960s.
The connectionism movement experienced a setback in 1969 with the publication of the book
Perceptrons
by MIT’s Marvin Minsky and Seymour Papert.
27
It included a key theorem demonstrating that the most common (and simplest) type of neural net used at the time (called a Perceptron, pioneered by Cornell’s Frank Rosenblatt) was unable to solve the simple problem of determining whether or not a line drawing was fully connected.
28
The neural-net movement had a resurgence in the 1980s using a method called “backpropagation,” in which the strength of each simulated synapse was determined using a learning algorithm that adjusted the weight (the strength of the output) of each artificial neuron after each training trial so the network could “learn” to more correctly match the right answer.
However, backpropagation is not a feasible model of training synaptic weights in an actual biological neural network, because backward connections to actually adjust the strength of the synaptic connections do not appear to exist in mammalian brains. In computers, however, this type of self-organizing system can solve a wide range of pattern-recognition problems, and the power of this simple model of self-organizing interconnected neurons has been demonstrated.
Less well known is Hebb’s second form of learning: a hypothesized loop in which the excitation of a neuron would feed back on itself (possibly through other layers), causing a reverberation (a continued reexcitation of the neurons in the loop). Hebb theorized that this type of reverberation could be the source of short-term learning. He also suggested that this
short-term reverberation could lead to long-term memories: “Let us assume then that the persistence or repetition of a reverberatory activity (or ‘trace’) tends to induce lasting cellular changes that add to its stability. The assumption can be precisely stated as follows: When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A’s efficiency, as one of the cells firing B, is increased.”
Although Hebbian reverberatory memory is not as well established as Hebb’s synaptic learning, instances have recently been discovered. For example, sets of excitatory neurons (ones that stimulate a synapse) and inhibitory neurons (ones that block a stimulus) begin an oscillation when certain visual patterns are presented.
29
And researchers at MIT and Lucent Technologies’ Bell Labs have created an electronic integrated circuit, composed of transistors, that simulates the action of sixteen excitatory neurons and one inhibitory neuron to mimic the biological circuitry of the cerebral cortex.
30
These early models of neurons and neural information processing, although overly simplified and inaccurate in some respects, were remarkable, given the lack of data and tools when these theories were developed.