Introduction to Neural Networks

Problem Set 3

This exercise is an extension of the one in an earlier problem set where we assumed spike emission at a point is a Bernoulli trial, and the probability of eliciting a spike was proportional to intensity. The idea was to visualize a spatial pattern of spikes at an instant where there were no spatial interactions. Here is one way of solving it:

origimage=Import["ExampleData/ocelot.jpg"];Image[Partition[RandomVariate[BernoulliDistribution[#]]& /@ Flatten[ImageData[origimage]],200]]

Our goal in the current exercise is to extend this visualization model by simulating a “snap-shot” of what an on-center, lateral inhibitory filtering of the input coming out of the retina en route to LGN might look like. This is more representative of daytime, called “photopic” vision.  Imagine we could view activity across a retinotopic slice across the optic nerve after viewing the picture for an instant. There are many ways to do this and with varying complexity, for example taking into account Poisson statistics of spike rates over time (where we could be explicit about what an instant is, e.g. 100 msec), including both temporal and spatial filtering, sigmoidal non-linearities, and inhomogeneous retinal sampling. But we’ll keep it fairly simple to emphasize concepts and tools.

We’ll add two additional components to the above code: 1) convolution with a linear center-surround spatial filter (as in stage 1 of the generic model); and 2) a threshold operation that simulates the response of an “on-center” cell (a particular form of stage 2; however, we won’t explicitly model threshold and saturation).

You can construct your own program as you wish, but it may help to try to use high-level image processing built-in functions in Mathematica as suggested in the hints in the following parts of this exercise:

a) Use LaplacianGaussianFilter[ ] on origimage. This will create a center-surround filter and do the convolution in one step. Let the kernel radius = 3.0. Short-hand for this filtering operation is LoG. Use ImageAdjust[ ] to normalize the filtered output to range between 0 and 1, and display the result. Call it ifiltered. It should look something like:

b) To simulate the output of an on-response unit, run the above filtered output through ImageClip[<out>,{0.5,1.0}], 
and again normalize the output, again using ImageAdjust[].  The idea is to set values of  ifiltered that are below a threshold of 0.5 to 0.0, and then use ImageAdjust[] to get output values to go from 0 to 1 again.

(The threshold of 0.5 is arbitrary and takes advantage of the  ImageAdjust[] operation in part 1 above, which “centered” the response range, in the sense of stretching the gray levels to be between 0 and 1. One could be more sophisticated and have an adaptive component to set the threshold, e.g. based on a local average. Later on we’ll see how to model a sigmoidal threshold for an integrate-and-fire neuron probabilistically.)

Call your filtered clipped response iclipped. It should look like:

c) Now adapt the same solution from problem 5 in problem set 2 (see above), but applied to the iclipped image to produce the “snap-shot” of the spiking activity:

Image[Partition[RandomVariate[BernoulliDistribution[#]]& /@ Flatten[ImageData[iclipped]],200]]

Call the image ispike. 

Use GraphicsRow[ ] to compare a) the original image b) the LoG filtered image c) the on-center response and d) the the “snap-shot” of the spiking activity. It should look like:

GraphicsRow[{origimage,ifiltered,iclipped,ispike}]

Note that the ispike picture is a “spike” rather than “rate” view of retinal output. It illustrates the high degree of sparseness in the spatial pattern of spikes at an instant. (There is another class of cells--the “off” cells--that signals the negative going filtered response.) Sparse responses raise the question of how the richness of perceptual experience emerges from such an apparently drastic reduction in information.

Execute the invisible cell below to initialize the matrix basiset. 
You can see what is in the cell by selecting the menu item Cell:Cell Properties:Open

basisset={{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(2 (Sqrt[2]))),0,0,(1/(2 (Sqrt[2]))),0,(1/(2 (Sqrt[2]))),(1/(2 (Sqrt[2]))),0,(1/(2 (Sqrt[2]))),0,0,(1/(2 (Sqrt[2]))),0,(1/(2 (Sqrt[2]))),(1/(2 (Sqrt[2]))),0,-(1/(2 (Sqrt[2]))),0,0,-(1/(2 (Sqrt[2]))),0,-(1/(2 (Sqrt[2]))),-(1/(2 (Sqrt[2]))),0,-(1/(2 (Sqrt[2]))),0,0,-(1/(2 (Sqrt[2]))),0,-(1/(2 (Sqrt[2]))),-(1/(2 (Sqrt[2]))),0},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2])))},{(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2]))),(1/(4 (Sqrt[2]))),-(1/(4 (Sqrt[2])))}};

Hint: The solution is a one-linear that uses one Mathematica function.

Let {f1,g1} and {f2,g2} be two input/output pairs, where f and g represent the input and output vectors to a linear neural network.

f1={1,-1,1};
f2={1,1,0};
g1= {3,1,2};
g2 = {-1,-1,2};

Calculate the outer product of g1 with f1 (i.e., (Subscript[g, 1]) ((Subscript[f, 1])^T)). Let this outer product equal W.

Two vectors can be defined to have the same “pattern” if they point in the same direction, regardless of their vector lengths. This interpretation of  pattern reflects the assumption that it is the activity of the neurons relative to each other that encode an event. Changes in the overall activity (length of the vector) may represent the changes in the confidence that the event is indeed the case.

Test to see whether W recalls the “pattern” of g1 if f1 is fed in as the input.  Hint: You can use CosineDistance[ ].

Now add the outer product of g2 with f2 to W, i.e. W = (Subscript[g, 1]) ((Subscript[f, 1])^T) + (Subscript[g, 2]) ((Subscript[f, 2])^T)

If we mainly care about the vector direction of the outputs (i.e. their “patterns”), show that we still have good recall for both f1 and f2 inputs.

The assumption of orthogonality for the input patterns for the linear associator would seem to make it completely impractical as a memory advice for arbitrary patterns. However, if the dimensionality of the input space is large, the odds get better that the cosine of the angle between any two independently sampled random vectors is close to zero, i.e. that the angle between them is close to 90 deg. 

The exercise here is to show how that the histograms for the distributions of the angles between two random vectors for dimensions 10, 100, and 1000 get progressively narrower and concentrated around 90 degrees. To sample, use RandomVariate[UniformDistribution[{-1,1}], n], where n is the dimensionality of each vector in a pair (i.e. n=10,100, or 1000). 

Define a function sample[n] that returns the angles between 1000 pairs of random vectors of dimensionality = n. This will give you 1000 points for each histogram.

To calculate the angle between vectors, you can use:

angle[v1_,v2_]:= ArcCos[v1.v2/(Norm[v1]*Norm[v2])]*(180./Pi)

Now use Histogram[{sample[10],sample[50],sample[2500]},ChartLegends->{“10”,”100”,”1000”}] to display your results which should look something like this:

(Legended[, (SwatchLegend["10", "100", "1000"])])

This exercise shows how the the supervised learning via the Widrow-Hoff training rule relates to statistical regression. (See Lecture 10.)

Make a function: rline[slope_,intercept_] to generate pairs of random numbers {x,y} where x ranges between 0 and 10, and whose y coordinate is a straight line with slope, slope_ and intercept, intercept_  but perturbed by additive uniform random noise over the range -2 to 2.

Generate a data set from rline with 200 samples with slope 11 and intercept 0.

Use the function Fit[] to find the slope and intercept of this data set. Here is an example of how Fit[] works:

Clear[x]

Fit[{{1,3},{2,5},{3,7.1}}, {1,x}, x]

If you wanted to fit a line through the origin, you would write: Fit[{{1,3},{2,5},{3,7.1}}, {x}, x], and a quadratic fit as: Fit[{{1,3},{2,5},{3,7.1}}, {1,x,x^2}, x].

Now implement a Widrow-Hoff algorithm to find the slope of the data. Assume the data goes through the origin, and initialize the slope to zero for the first iteration.  Use ListPlot to show the values of the slope as a function of the iterations from 1 to 200. What is a suitable range for the learning constant?

This exercise illustrates the use of the classic “backprop” model of learning in multiple-layer perceptrons. It also shows how learning can drive outputs to threshold and saturation regions (interpreted as “false” and “true”), which allows one to do classification via a feedforward non-linear network. The application is to a “toy” version of the rotation invariance problem. The backprop algorithm doesn’t scale up well to complicated “real world” invariance problems. For that, other methods such as deep belief networks, are needed.

You can use Backpropagation.m  for this exercise. 

Define two classes of patterns, T's and C's. Each class has four members for rotations of 0,90,180, and 270 degrees. Train a non-linear feedforward net with one layer of hidden units to correctly classify the eight patterns as T or C. To get you started, here are exemplars for T and C on a 3x3 grid:

T1 = {{0.9,0.9,0.9},{0.1,0.9,0.1},{0.1,0.9,0.1}};
C1 = {{0.9,0.9,0.9},{0.9,0.1,0.1},{0.9,0.9,0.9}};

ArrayPlot[C1,Mesh->True,ColorFunction->GrayLevel,AspectRatio->1,Frame->False]

Fix the learning constant at 1. What is the minimum numbers of hidden units you can find that will still converge in under 600 iterations?

For this number of hidden units, what is the maximum value of the learning constant (eta) that still allows convergence?