Introduction to Neural Networks

Daniel Kersten

Problem Set 2

Let  w be a weight vector representing a template pattern. You can think of w as representing the strengths of a set of synapses to a neuron. Let {s} be a collection of input pattern vectors all of unit length (norm).  The values of the elements of s represent the strengths of the inputs to the neuron. Show theoretically that the dot product of  s with w gives maximum response to the pattern s’ that matches the "form" of the template pattern. This is the basis for thinking of a neuron with receptive field w as a “feature detector” for inputs of the same form.

a) Define a semi-linear  "squashing" function, thresh[s], that equals:  
				                        0 for s < 0; 
				                        s for 1 >=s >= 0; 
				                        1 for s> 1. 
(Hint: There are lots of ways to do this "from scratch", by building up rules, by using Which[]...but take a look at Piecewise[]).
Plot thresh[s] from s = -2 to 2.

b) There are neurons in the primary visual cortex of mammals called "simple cells". One model for these cells is a linear filter (the dot product of the previous exercise) followed by a thresholding non-linearity (e.g. the half-wave rectification of a diode).  The receptive field weights of this filter can have a center-surround organization, which for one class of subset of cells can be characterized by an excitatory center, and an inhibitory surround. This kind of spatial filtering is sometimes referred to as lateral inhibition. In one dimension, a simple model weight vector could look like this:

w = {-2,-1,6,-1,-2};

Use the above weight vector w, and your thresh[] function to calculate the response of the model simple cell. Calculate the response to the three inputs, s :

s1 = {-1, -.5, 3, -.5, -1}/25.;
s2 = {2, 1, 0, 1, 2} /10.;
s3 = {0, -7, 0, 7, 0}/25.;

A response can be zero either because of the threshold, or because the input pattern is a poor match to the pattern of receptive field weights. Explain the responses to s2 and s3 in terms of these two factors.

This exercise will give you some more practice with the symbolic processing in Mathematica. Later on, when we study multiple layer perceptrons we will  need to use the derivative of a non-linear squashing function in our derivation of the back-propagation learning rule. For this reason, it is useful to have a squashing function that is differentiable and has a closed form solution for the derivative. 

a) Use Mathematica’s D[ ] function to define the derivative of the logistic function, 1/(1 + Exp[ - 4 r]).

squash[r_] := 1/(1 + Exp[-r/.25]);

Call the derivative  dsquash[]. 
Plot dsquash from r = -2 to 2.

b) Mathematica has a built-in function: LogisticSigmoid[x]. Again use D[ ], but this time compute the derivative of LogisticSigmoid[x]. This exercise illustrates a particularly nice property of the logistic function.

Given certain assumptions, the response of single model neuron in a population can be viewed as "the projection" of the input onto the basis vector represented by that neuron's receptive field weights. This exercise gives you some practice in manipulating matrices and vectors, and understanding how vectors can be re-represented in a change of basis set; in this case from the original cartesian to a Walsh basis.

This exercise uses the Walsh basis set from Lecture 4.

a) Given the vector:

h = {1,2,3,4,5,6,7,8};

use ListPlot to graph the "spectrum" of h with respect to the 8-dimensional orthonormal Walsh basis vectors given in the lecture. (You can just copy and paste them in your answer Notebook.). Recall that the spectrum consists of the magnitudes of the projections of h onto the Walsh functions. 

b) Verify your answer by reconstructing h from the projections, i.e. as a linear sum of the basis vectors.

This exercise will give you practice in using random sampling,  basic functional programming using Map[] to visualize a dynamic “neural image” of spiking activity.

Suppose you had an instrument that could show you the spatio-temporal pattern of action potentials over a population of neurons in response to a sensory input. What might it look like?

At very low light levels, local retinal response is roughly proportional to the average light level over a localized region. We will model the spiking output using a very simple model in which the probability of a spike at a given time and location is proportional to the light intensity at that location. Assume spike emission at a point is a Bernoulli trial (i.e. the flip of biased coin). Use the following image as input:

image=Import["ExampleData/ocelot.jpg"];
idata=ImageData[image];

1) What are the dimensions of idata? What are the maximum and minimum values of idata?

2) Using the following functions: Map[], RandomVariate[],  BernoulliDistribution[], Flatten[], Partition[] and Image[], make a random binary image which is a random sample of retinal output. Your answer should be one line of code and produce an output that looks something like this:

3) Now use the above expression as the first argument in Table[] to generate 10 independent samples of a binary image response to idata. Store these in a list:
	 movie = Table[?, {10}];

4) Use ListAnimate[movie] to play your movie. It should look something like:

This exercise is easiest if you copy and modify bits of code from Lecture 5 used to illustrate Mach bands. Make a graylevel  plot of a 30x30 pixel step function (rather than the ramp function used in Lecture 5). As with the ramp in Lecture 5 illustrating Mach bands, the pattern should only change in the x-direction. 
Your plot should look like this:

a) Look carefully at the bright side of the transition from dark to light. Do you perceive a bigger Mach band for the step than for the ramp?

b) Now, using the same parameters as were used for the ramp input in Lecture 5, find the response produced by the recurrent lateral inhibition equations to the step. Graph it.

c) Using the same parameters as were used for the ramp input in Lecture 5, find the response produced by the recurrent lateral inhibition equations to the ramp. Graph it.

d) Consider the model predictions. Does the model predict a bigger, smaller, or the same size Mach band (at the bright knee) for the step than for the ramp?

Compare the response of the network to the step function with and without self-inhibition.  Do this using Manipulate[< your code here>, {{iterations, 5},1, 20, 1}] to overlay two graphs using the following initialization parameters:

size = 30;
spaceconstant =5;
maxstrength = 0.05;
iterations = 200; 
epsilon = .3;

iterations is the variable in the Manipulate’s “slider”. Your output should look something like for iterations = 3.

How many iterations are required before the network stabilizes with self-inhibition? Without self-inhibition?

a) Given the above ramp input and weight matrix W, find the steady-state solution (i.e. matrix W’) to the limulus equation, by solving a set of simultaneous linear equations using the Inverse[] function that computes the inverse of a matrix. Use the matrix, W’, to find and plot the solution along with the input pattern corresponding to a ramp, and a step. Recall that there is a Mathematica  function, IdentityMatrix[], you might find useful. Your final plot should look like this:

b) This steady state solution can be viewed as a simple linear feedforward network. Use ListPlot[] to show that the effective receptive field of neurons in this feedforward network has a center-surround organization. You only need to show the plot for effective weights of neuron 14, i.e. the 14th row of W’.

Convolution is perhaps the simplest example of a neural network architecture in which the same operation gets repeated over space and/or time. It is a fundamental operation in models of image processing and recognition, as in “convolutional neural networks” for recognition. In terms of matrix construction, each subsequent row is equal to the previous row shifted right by one column width.

Let Wprime be the vector corresponding to the 14th row of W’ (from the previous exercise). Use ListConvolve[] and ListPlot[] to show that you get the same output to the ramp stimulus as you obtained in the previous exercise. 

HINTS: i) Align the kernel (i.e. Wprime) at position 14; ii) use 0 for the padding value in ListConvolve[]; iii) You only need one line of code -- just ListConvolve[ ] (with the correct variable settings) and ListPlot[].

Let the weights to a single neuron be represented by the first row of matrix W which we model as a “gabor function”:

weights = Table[Cos[8. x] Exp[-(x*4.)^2],{x,-2,2,.01}];
ListPlot[weights,Joined->True,PlotRange->All]

And then each subsequent row of W is shifted right by one. This captures the idea that each neuron prefers the same feature, but is looking for that feature at a different location.

To simplify dealing with how to introduce new elements on the left with each shift, we’ll wrap the right part of each gabor vector around to the left. ToeplitzMatrix[] does this automatically for us:

W = ToeplitzMatrix[weights];
MatrixPlot[W]

Use ListPlot[,Joined->True] to plot the fourth eigenvector  of W. Note that the eigenvector has a sinusoidal form. This is the reason that sinusoids are particularly useful as test basis sets in linear systems analysis, especially when the network has a convolutional architecture.

Multiply the fourth eigenvector by W and then use CosineDistance[ ]  to verify that the resulting output vector points in the same direction as the input. Note that Cosine Distances between vectors that point in the same direction and orthogonal directions are 0 and  1, respectively. You may not get the exact answer you expect, but you should be close.