Introduction to Neural Networks

"Energy" and neural networks
Discrete Hopfield network
Constraint satisfaction problems

Constraint satisfaction

Hand-wiring the constraints in a neural net.

How does one "sculpt the energy landscape"? One can use a form of Hebbian learning to dig holes (i.e. stable points or attractors) in the energy landscape indicating things to be remembered. Alternatively, one can study the nature of the problem to be solved and hand-wire the network to represent the constraints (i.e. reason out, or make an educated guess as to what the weights should be). 
We will follow the second approach to solve the correspondence problem. This problem crops up in a number of domains in pattern theory and recognition, and occurs whenever the features in two patterns need to be matched up, but one pattern is an unknown distortion of the other. Imagine, for example, the image of a face in memory, and you want to test whether an incoming stimulus is from the same person. Both patterns have more or less the same features, but they don't superimpose because they are from different views or expressions.  So you want to try to morph one on to the other, but before you do that, you might want to decide which features go with which--i.e. pupils to pupils, nostrils to nostrils, etc.. Establishing correspondences between two similar but not quite identical patterns has also been a central, and challenging problem in both stereo vision and motion processing.
In the next few sections, we will show how the weights in a network can be set up to represent constraints. Then we will look at three ways of letting the network evolve: asynchronous, synchronous, and partially asychronous  updating. The first case exactly satisfies the assumptions required for Hopfield's energy function.

Establishing correspondences: An example of constraint satisfaction

Introduction to stereo and the correspondence problem

If you cross your eyes so that you can perceptually fuse the two random patterns below, you may be able to see a small square floating in front of the random background. Crossing your eyes means that the left image goes to the right eye, and the right to the left. (Some people are better at looking at the left image with the left eye, and right with the right eye. For this type of human, the images below should be exchanged.)
This is an example of a random dot stereogram originally developed by Bela Julesz in the 1960's.

It is made by taking a small square sub-patch in one image, shifting it by a pixel or two, to make the second image. Since the subpatch is shifted, it leaves a sub-column of pixels unspecified. These get assigned new random values.  
The stereogram mimics what happens in the real world when you look at an object that stands out in depth from a background--the left eye's view is slightly different than the right eye's view. The pixels for an object in front are shifted with respect to the background pixels in one eye's view compared to the other. There is a disparity between the two eyes. The distances between two points in the left eye and the distance of the images of the same two points in the right eye are, in general different, and depend on the relative depth of the two points in the world.
To see depth in a random dot stereogram, the human visual system effectively solves a correspondence problem. The fundamental problem is to figure out which of the pixels in the left eye belong to which ones in the right. This is a non-trivial computational problem when so may of the features (i.e. the pixel intensities) look the same--there is considerable potential for false matches.  A small minority don't have matching pairs (i.e. the ones that got filled in the vertical slot left after shifting the sub-patch). We'll get to this in a moment, but first let's make a stereogram using Mathematica.
Human perception solves the stereo correspondence, so let us see if we can devise a neural network style algorithm to solve it.

Making a random dot stereogram using the Map operation

Review: Getting and putting pieces of a matrix

There are lots of ways to place a patch at two different locations in two images. We'll define a function to put submatrices in. Mathematica lets you pull submatrices out of a matrix. For example, the following shows three commands for pulling out a section consisting of rows 1 and 2, and columns 2 and 3:

mm = {{1,2,3},{3,4,5},{6,7,8}} 
mm[[ Range[1,2], Range[2,3] ]]
mm[[ 1 ;; 2, 2 ;; 3 ]]
Take[mm, {1, 2}, {2, 3}]

Recall that Range[] is a function for making lists spanning a range:

Range[2,4]

There are several ways to insert a submatrix into a larger matrix (e.g. using ReplacePart[]). To put a small matrix into a larger matrix:

mm={{1,2,3},{3,4,5},{6,7,8}}
mm[[1;;3,2;;3]]={{a,b},{c,d},{e,f}}
mm

Or you can use:

mm={{1,2,3},{3,4,5},{6,7,8}}
mm[[Range[1, 3], Range[2, 3]]]={{a,b},{c,d},{e,f}}
mm

We''ll insert a patch of pixels into the left and right eye using MapAt[function, expression, positions]. MapAt will apply a function to the twoDlefteye matrix that replaces values at the specified positions with the values from patch. Although you could easily write a simple routine that looped through the large matrix inserting the patch, it is particularly easy to modify the function below to insert bits at arbitrary locations, just by setting up the list of coordinates appropriately.

Initialize

Let's make a simple random dot stereogram in which a flat patch is displaced by disparity pixels to the left in the left eye's image. il and jl are the lower left positions of the lower left corner of patch in the twoDlefteye. Similarily, ir and jr are the lower left insert positions in the twoDrighteye matrix.
 In order to help reduce the correspondence problem later, we can increase the number of gray-levels, or keep things difficult with just a few--below we use four gray-levels.

size = 32; patchsize = size/2;

twoDlefteye=RandomInteger[3,{size,size}];twoDrighteye=twoDlefteye;patch=RandomInteger[3,{patchsize,patchsize}];

Left eye

The left eye's view will have the patch matrix displaced one pixel to the left.

disparity = 1;
il = size/2-patchsize/2 + 1; jl = il  - disparity;

i=1; 
label[x_] := Flatten[patch][[i++]];
twoDlefteye = MapAt[label, twoDlefteye, 
	Flatten[Table[{i,j},
		{i,il,il + Dimensions[patch][[1]]-1},
	{j,jl,jl + Dimensions[patch][[2]]-1}],1] ];

Right eye

The right eye's view will have the patch matrix centered.

ir = size/2-patchsize/2 + 1; jr = ir;

i=1; 
label[x_] := Flatten[patch][[i++]];
twoDrighteye = MapAt[label, twoDrighteye, 
	Flatten[Table[{i,j},
	{i,ir,ir + Dimensions[patch][[1]]-1},
	{j,jr,jr + Dimensions[patch][[2]]-1}],1] ];

Display a pair of images

It is not easy to fuse the left and right image without a stereo device (it requires placing the images side by side and crossing your eyes. We can check out our images another way.
The visual system also solves a correspondence problem over time. We can illustrate this using ListAnimate[]. By slowing it down, you can find a speed in which the central patch almost magically appears to oscillate and float above the the background. If you stop the animation (click on ||), the central square patch disappears again into the camouflage.

gl=ArrayPlot[twoDlefteye,Mesh->False,Frame->False,Axes->False,ImageSize->Tiny];
gr=ArrayPlot[twoDrighteye,Mesh->False,Frame->False,Axes->False,ImageSize->Tiny];ListAnimate[{gl,gr,gl,gr},4]

GraphicsRow[{gl,gr}]

Two-state neural network  implementation using Marr and Poggio (1976) constraints

Simplify the problem to one dimension

We will apply the constraints proposed by Marr and Poggio (1976) to try to solve the correspondence problem for just the middle rows of twoDlefteye and twoDrighteye:

lefteye=twoDlefteye[[(size/2)]];righteye=twoDrighteye[[(size/2)]];ArrayPlot[{lefteye,righteye},AspectRatio->(1/8),Mesh->True]

We've displayed the left eye's view above the right eye's, so you can try to spot which pixels have been shifted. Because the patch was shifted horizontally, we haven't lost the essence of the correspondence problem by reducing it to one dimension. You can see the ambiguity in the correspondence problem.
Following Marr and Poggio, we will try to solve the correspondence (i.e. which pairs of pixels in the two images belong together) using three constraints: 
	compatibility, 
	uniqueness, and 
	smoothness. We will see what these constraints mean as we move along. The challenge is to design a network that  enforces these constraints.

The compatibility constraint

The compatibility constraint says that similar features in each eye should match, in other words like pixels (same graylevel) in each image should match. We can represent compatibility by constructing a "compatibility matrix" which has a 1 at each location where there is a possible match (i.e. in graylevel color), and zeros elsewhere.

compatibility = Table[If[
		lefteye[[i]]==righteye[[j]],1,0],
		{i,size},{j,size}];

Here's the plot of the compatibility of the middle row (size/2) for the left and right eye's stereo pictures, where black represents 1 (compatible), and white 0 (not compatible).

ArrayPlot[compatibility,DataReversed->True,Mesh->True,Frame->True,FrameLabel->{"x, other eye","x, one eye"}]

The uniqueness and smoothness constraints

But many of these possible matches seem unlikely. A given point in one eye shouldn't have lots of matches in the other eye--a point on a surface in the world typically maps to at most one point in each eye. We would like to find a line through the compatibility matrix to indicate unique matches. So we have to discourage more than one unit from being on in any given row or column (i.e. enforce a uniqueness constraint).
Also when we think about how features along surfaces typically change as a function of position, we realize that surfaces to a first approximation are smooth--thus, nearby points on a surface project to nearby points in the two eyes. For convenience, let's assume an even stronger constraint in which nearby points have constant disparities. (Recall that disparity refers to the relative shift of corresponding points in the two images.)
This means to we should encourage elements that have nearest neighbor support along the diagonals to be on (to encourage regions that have constant disparity). This latter constraint is called a "surface smoothness constraint", which refers to the underlying assumption that changes in depth usually change gradually--sudden changes in disparity are rare.

We will follow Marr and Poggio and set up a threshold logic unit at each location of the compatibility matrix, and wire each up to reflect the above constraints. We are going to have to worry about the boundaries. There are several ways of doing this. One is to have a "free" boundary in which the connection weights at the boundaries are actually different (to compensate for a lack of neighbors in the other directions). Although not biologically realistic, a second way is to use a toroidal geometry, restricting indices by the following modulus function: myMod[x_] := Mod[x-1,size]+1. This option is computationally convenient and makes it possible to comply with the restriction of symmetric connections everywhere.

To encourage uniqueness, let's make the vertical and horizontal connection weights all equal and negative with a weight inhib. To encourage smoothness, we'll make the diagonal support positive with mutally excitatory weights excit. The network will have biases for each unit (which are equivalent to appropriate thresholds for each unit) proportional to the original compatibility matrix. These biases correspond to the Ui's in the Hopfield net and serve to prevent the network from loosing this strong constraint from the data as the iterations progress.

Note that we won't  set up the network calculations using a matrix and matrix multiplications as you will do in the Hopfield memory examples. This is because most of our connections are zero and we don't want to waste time multiplying zero elements and adding them up. So the code below will look messier.

Hopfield Net: Asynchronous updating--all sites visited randomly, at random times

To do  random asynchronous updating, you simply pick a site at random and update it, and then pick another, and so on.

V = compatibility; V1 = V;

excit = 2; inhib = -1; k =6; theta = 13;
threshold[xx_] := N[If[xx>theta,1,0]];

myMod[x_] := Mod[x-1,size]+1;

The neighborhood size along the diagonal, horizontal, and vertical directions are each 8. We use constant weighting. Although the code looks messy, it is doing something pretty simple. Most of the lines of code are just specifying which neighboring elements of the compatibility matrix should excite or inhibit the unit being updated.

DocumentNotebook[Dynamic[ArrayPlot[V1,DataReversed->True,Mesh->True,ImageSize->Small]]]

Timing[numiterations=10000;For[iter=1,iter≤numiterations,iter++,
If[Mod[iter,10]==1,ArrayPlot[V1,Mesh->True]];
i=RandomInteger[size-1]+1;j=RandomInteger[size-1]+1;V1[[i,j]]=threshold[inhib (V1[[myMod[i+1],myMod[j]]]+V1[[myMod[i-1],myMod[j]]]+V1[[myMod[i],myMod[j-1]]]+V1[[myMod[i],myMod[j+1]]] V1[[myMod[i+2],myMod[j]]]+V1[[myMod[i-2],myMod[j]]]+V1[[myMod[i],myMod[j-2]]]+V1[[myMod[i],myMod[j+2]]] V1[[myMod[i+3],myMod[j]]]+V1[[myMod[i-3],myMod[j]]]+V1[[myMod[i],myMod[j-3]]]+V1[[myMod[i],myMod[j+3]]] V1[[myMod[i+4],myMod[j]]]+V1[[myMod[i-4],myMod[j]]]+V1[[myMod[i],myMod[j-4]]]+V1[[myMod[i],myMod[j+4]]])+excit (V1[[myMod[i+1],myMod[j+1]]]+V1[[myMod[i-1],myMod[j-1]]]+V1[[myMod[i+2],myMod[j+2]]]+V1[[myMod[i-2],myMod[j-2]]]+V1[[myMod[i+3],myMod[j+3]]]+V1[[myMod[i-3],myMod[j-3]]]+V1[[myMod[i+4],myMod[j+4]]]+V1[[myMod[i-4],myMod[j-4]]])+k compatibility[[i,j]]];];]

Energy and constraint satisfaction

What use is energy?
We haven't computed the Hopfield energy for each iteration. But we could, and in fact this calculation can be useful. 
In our toy example of a random dot stereogram, we know the answer (we made the stereogram), so computation of the energy of the final stable state can be compared with the energy of the right answer. If the right answer has a lower energy than the stable state reached, we know we got stuck in a local minimum. If the right answer has a higher energy, then the constraints must have been set up incorrectly. Energy can be a useful number to calculate during network model development.

References

Hertz, J., Krogh, A., & Palmer, R. G. (1991). Introduction to the theory of neural computation (Santa Fe Institute Studies in the Sciences of Complexity ed.). Reading, MA:      Addison-Wesley Publishing Company. 
Hopfield, J. J. (1982). Neural networks and physical systems with  emergent collective computational abilities. Proc. Natl. Acad.  Sci. USA, 79, 2554-2558.
Hopfield, J. J. (1984). Neurons with graded response have  collective computational properties like those of two-state  neurons. Proc. Natl. Acad. Sci. USA, 81, 3088-3092.
Hopfield, J. J. (1994). Neurons, Dynamics and Computation. Physics Today, February.
Marr, D., & Poggio, T. (1976). Cooperative computation of stereo disparity. Science, 194, 283-287.
Marr, D., Palm, G., & Poggio, T. (1978). Analysis of a cooperative stereo algorithm. Biol. Cybernetics, 28, 223-239.
Mayhew, J. E. W., & Frisby, J. P. (1981). Psychophysical and Computational Studies Towards a Theory of Human Stereopsis. Artificial Intelligence, 17, 349-385.
Yuille, A. L., Geiger, D., & Bülthoff, H. H. (1991). Stereo integration, mean field theory and psychophysics. Network, 2, 423-442.