Introduction to Neural Networks
U. Minn. Psy 5038

Problem Set 4

Exercise 1 - Oja's rule for weight growth

Recall that there is nothing in the outer product version of the Hebb learning rule to prevent the weights from growing without bound.  Assume a linear model of a neuron with n inputs:
 y = w1*x1 + w2*x2 +...+wn*xn.
Show that Oja's modification of the Hebb rule tends to normalize the weights according to the following rule:
Sum[w1^2+w2^2+...+wn^2] = 1

Your proof needn't use Mathematica.

Exercise 2 (easy) - Hopfield network

Notes on continuous response Hopfield network (from Lecture notes)

Definitions and functions that you will find useful for the exercises below in this problem set are reproduced from Lecture notes in this section.

Definitions

We will let the resistances and capacitances all be one, and the current input Ii be zero. Define the sigmoid function, g[] and its inverse, inverseg[]:

a1 := (2/Pi); b1 := Pi 1.4 / 2;
g[x_] := N[a1 ArcTan[b1 x]]; 
inverseg[x_] := N[(1/b1) Tan[x/a1]];

Initialization of starting values

The initialization section sets the starting output values of the two neurons 
V = {0.2, -0.5}, and the internal values u = inverseg[V], the step size, dt=0.3, and the 2x2 weight matrix, Tm such that the synaptic weights between neurons are both 1. The synaptic weight between each neuron and itself is zero.

dt = 0.3;
Tm = {{0,1},{1,0}};
V = {0.2,-0.5};
u = inverseg[V];
result = {};

Main Program

The following function computes the output of the ith neuron for the network with a list of neural values uu, and a weight matrix Tm.

Hopfield[uu_,i_] := uu[[i]] + dt ((Tm.g[uu])[[i]] - uu[[i]]);

Let's accumulate some results for a series of 80 iterations. Then we will plot the pairs of activities of the two neurons over the 80 iterations.

result=Table[{k=RandomInteger[{1,2}],u[[k]]=Hopfield[u,k],u},{80}];

result=Transpose[result][[3]];gresults=ListPlot[g[result],Joined->False,AxesOrigin->{0,0},PlotRange->{{-1,1},{-1,1}},Frame->True,AspectRatio->1,Ticks->None]

Energy landscape

Here is a contour plot of the energy landscape.

inig[x_] := -N[(a1*Log[Cos[x/a1]])/b1];
energy[Vv_] := -0.5 (Tm.Vv).Vv +
	Sum[inig[Vv][[i]], {i,Length[Vv]}];

gcontour=ContourPlot[energy[{x,y}],{x,-1,1},{y,-1,1},AxesOrigin->{0,0},ContourShading->False,PlotRange->{-0.1`,0.8`},Contours->32,PlotPoints->30]

Show[gresults,gcontour]

Run the above simulation (in "Notes on continuous...") of a 2-neuron continous response Hopfield net with a starting value of V that will allow the state vector to be drawn to the other attractor basin. Plot the trajectory superimposed on the energy contour plot as shown above.

Exercise 3 - Synchronous updating for Hopfield network

Define a function syncHopfield[uu_] to do deterministic synchronous updating rather than the asynchronous updating rule specified in the above Hopfield network.Use results = NestList[syncHopfield,*,*]  with 40 iterations to list the state space trajectory. Plot the results as shown below.  
Use the same starting values used in Lecture notes (see also above).
Now use  asynchronous iterations (as in the above insert from Lecture notes) to compute 20 trajectories all from the same starting point. Compute the average trajectory, and plot both this average trajectory and the deterministic synchronous trajectory together on the same contour plot to  compare the descent with the asynchronous case.
Which one appears to be closest to a steepest descent?