Introduction to Neural Networks
U. Minn. Psy 5038

Problem Set 3

Exercise 1 - Orthogonality of large random vectors

The assumption of orthogonality for the input patterns for the linear associator would seem to make it useless as a memory advice for arbitrary patterns. However, if the dimensionality of the input space is large, the odds are pretty good that the cosine of the angle between any two random vectors is close to zero. Make three histograms showing the distributions of the cosines of random vectors for dimensions 10, 50 , and 250. The histograms should put the cosines in bins from -0.9 to 0.9 in steps of 0.2. Sample the cosines one hundred times for each histogram. 
You can use the  Histogram[] function, or  BinCounts[] and BarChart[]. Here is an example to show the histogram for a uniform distribution between -0.9 and 0.9:

data[size_]:=BinCounts[Table[1.8 RandomReal[]-0.9,{100}],{-0.9,0.9,0.2}];

BarChart[data[10],ChartLabels->{-0.8,-0.6,-0.4,-0.2,0,0.2,0.4,0.6,0.8},PlotRange->{-10,100}]

Exercise 2 - Regression and Widrow-Hoff learning

Make a function: rline[slope_,intercept_] to generate pairs of random numbers {x,y} where x ranges between 0 and 10, and whose y coordinate is a straight line with slope, slope_ and intercept, intercept_  but perturbed by additive uniform random noise over the range -2 to 2. (See Lecture 11)

Generate a data set from rline with 200 samples with slope 11 and intercept 0.

Use the function Fit[] to find the slope and intercept of this data set. Here is an example of how Fit[] works:

Fit[{{1,3},{2,5},{3,7.1}}, {1,x}, x]

If you wanted to fit a line through the origin, you would write: Fit[{{1,3},{2,5},{3,7.1}}, {x}, x], and a quadratic fit as: Fit[{{1,3},{2,5},{3,7.1}}, {1,x,x^2}, x].

Now implement a Widrow-Hoff algorithm to find the slope of the data. Assume the data goes through the origin, and initialize the slope to zero for the first iteration.  Use ListPlot to show the values of the slope as a function of the iterations from 1 to 200. What is a suitable range for the learning constant?

Exercise 3 - Use backprop to learn to classify letters independent of orientation

Define two classes of patterns, T's and C's. Each class has four members for rotations of 0,90,180, and 270 degrees. Train a non-linear feedforward net with one layer of hidden units to correctly classify the eight patterns as T or C. To get you started, here are exemplars for T and C on a 3x3 grid:

T1 = {{0.9,0.9,0.9},{0.1,0.9,0.1},{0.1,0.9,0.1}};
C1 = {{0.9,0.9,0.9},{0.9,0.1,0.1},{0.9,0.9,0.9}};

ArrayPlot[C1,Mesh->True,ColorFunction->GrayLevel,AspectRatio->1,Frame->False]

Fix the learning constant at 1. What is the minimum numbers of hidden units you can find that will still converge in under 600 iterations?
For this number of hidden units, what is the maximum value of the learning constant (eta) that still allows convergence? You can use Backpropagation.m  for this exercise.

Exercise 4 - Efficient coding: Principal Components of Image Data Set

An important theory of neural receptive field organization says that the receptive field structure of sensory neurons, such as the simple cells of V1 are organized to efficiently representing incoming image information (Barlow). One way to efficiently code a large dimensional data set is to project the data onto axes that capture the most variance. PCA is one way to do this.

Read in GrayLonesome256x256.jpg using:

granite=Import[SystemDialogInput["FileOpen"],"Data"];
width = Dimensions[granite][[1]]

Use the following to break this large image into a series of subimages to use as samples

The idea is that each  subimage will be used as a statistical sample. We compute the outer product of each, and then average all 16 to get an estimate of the autocovariance matrix.

nregions = 16;swidth = width/nregions

subface=Table[Take[granite,{i*swidth+1,i*swidth+swidth},{j*swidth+1,j*swidth+swidth}],{i,0,nregions-1},{j,0,nregions-1}];

subfacelist=Table[0.0,{256}];Table[subfacelist[[i+16*(j-1)]]=N[Flatten[subface[[i,j]]]],{i,1,16},{j,1,16}];

Now we subtract off the mean.

subfacelist2 = Table[subfacelist[[i]] - Mean[subfacelist[[i]]], {i,1,256}];

Now calculate the autocovariance matrix for the resulting collection of images,  subfacelist2[i], i=1 to 256, and plot the autocovariance matrix with ArrayPlot[]. (Recall the use of the Flatten[] function.)

Using Mathematica's built-in functions, calculate the eigenvectors and eigenvalues of the autocovariance matrix and use ListPlot to plot the eigenvalues.

Display the first 32 eigenvectors as "eigenpictures". (Recall the use of the Partition[] function. Use appropriate  options to keep the display compact.) Note how the eigenvectors are ordred with increasingly fine spatial scale.

Describe briefly how PCA can be implemented in a neural network using hebbian synapses.