Introduction to Neural Networks
U. Minn. Psy 5038

Problem Set 4

Exercise 1 - Oja's rule for weight growth

Recall that there is nothing in the outer product version of the Hebb learning rule to prevent the weights from growing without bound.  Assume a linear model of a neuron with n inputs:
 y = w1*x1 + w2*x2 +...+wn*xn.
Show that Oja's modification of the Hebb rule tends to normalize the weights according to the following rule:
Sum[w1^2+w2^2+...+wn^2] = 1

Your proof needn't use Mathematica.

Exercise 2 --  Capacity of the graded response Hopfield network.

If a Hopfield network has N model neurons, how many memories can it store? One way of assessing this is to randomly generate num stimuli of dimension dim = N, where num< dim, and store them using the outerproduct form of the Hebbian rule. stimuli[] can be used to generate the samples variable below. Then we use the same stimuli in samples as inputs to the network. If a samples vector stays put, i.e. is a stationary point, then we score that network run as a "success", i.e. a correct output. Another way of putting it is that if the vector samples[[i]] is a stationary point, then results[[i]] ≃  samples[[i]], where results[[i]] represents the output vector pattern of activities of the Hopfield simulation. If none of the stimuli move in state space, then we know we can store at least num patterns. But if num gets too big, then we expect to see increasing number of cases for which samples[[i]] is far from results[[i]]. What we decide is "close enough" is somewhat arbitrary. Below, we use HammingDistance[] to score memory recall performance. The overall idea is like asking yourself if pattern X is familiar, and if  your memory wanders off in some other direction, the answer is no.
The goal of this exercise is to find out how the capacity changes with N.  You will do a simulation to estimate the fraction of memories that can be stored relative to N (=dim). 
The code supplied below will help you get started. Adapt/improve the code to: 
	1) Pick the first value of dim, dim ϵ {12, 24, 48, 96, 192}. 
	2) For that value of dim,  and  for each value of num, where num ≃   1/12, 2/12, 4/12, 6/12, or 8/12 fractions of dim, 
	run at least 10 repeats, each with a new set of input samples. From these repeats, calculate the proportion correct.
	3) Plot up proportion correct vs. fraction of dim (i.e. vs. 1/12, 2/12, 4/12, 6/12, or 8/12)
	4) Repeat the above for the next value of dim.
	5) Plot up the five curves, one for each value of dim. E.g. it might look something like:

The following function,  stimuli[],  generates "num" random stimuli of dimension "dim"

stimuli[num_,dim_]:=Table[RandomChoice[{-1.0,1.0},dim],{num}];

Try out the function, so you understand it.

Define g[] and its inverse.

The following functions and parameters come from Lecture 19.

a1 := (2/Pi); b1 := Pi 1.4 / 2;
g[x_] := N[a1 ArcTan[b1 x]]; 
inverseg[x_] := N[(1/b1) Tan[x/a1]];

Starter Program

You can hand-set values of num and dim in the next cell, and then execute the rest the cells to find out how well the num patterns were stored in a dim-dimensional network, in terms of the number correctly remembered.

num=50;
dim=200;

samples=stimuli[num,dim];
weights = Sum[Outer[Times,samples[[i]],samples[[i]]],{i,1,Length[samples]}];
For[i=1,i<=Length[weights], i++, weights[[i,i]] = 0];

dt = 0.03;
Hopfield[uu_] := uu + dt (weights.g[uu] - uu);

results={};
For[i=1,i≤Length[samples],i++,
input=samples[[i]];
remembering=g[Nest[Hopfield,input,30]];
results=Append[results,remembering];
];
results=N[Sign[results]]; 
(*For convenience we force the results to +/- 1.0 for comparison with input samples *)

To count as "correct", we'll require that at least 5% (.1 x N) of the elements of the result vector pattern match the input sample vector. By calculating a confusion matrix, we can also keep track of how often a stimulus pattern has migrated sufficiently close to another of the input patterns to be counted as a false match. But we are mainly interested in the diagonal--the proportion of the stimulus patterns that are stationary points, i.e. correct matches.
We could make the HammingDistance criterion more stringent or lax. (Because of the symmetry in the network, we can have an exact match to the negative vector. We saw this in Lecture 19, where sometimes the negative image of one of the letters, P, I, T was recalled.  For simplicity, we will ignore that possibility.)

confusionmatrix=Table[If[HammingDistance[results[[k]],samples[[j]]]<.05*dim,1,0],{j,1,num},{k,1,num}];

A black cell in the array plot of the confusion matrix means that the kth output pattern (column k) matched the ith samples pattern that was intended to be stored. Perfect recall for all num patterns would result in all black along the diagonal, and white off the diagonal.

gconfusion=ArrayPlot[confusionmatrix,ImageSize->Tiny]
score = Total[Diagonal[confusionmatrix]];
percent =N[100.0*score/num,2];

Print["number correct:",score,", number of stimuli: ",num", percent correct: ",percent, "%"]