Introduction to Neural Networks

Problem Set 2

Exercises

Exercise 1: Comparing lateral inhibition responses to step and ramp images

This exercise is easiest if you copy and modify bits of code from Lecture 5 used to illustrate Mach bands. Make a graylevel  plot of a 30x30 pixel step function (rather than the ramp function used in Lecture 5). As with the ramp in Lecture 5 illustrating Mach bands, the pattern should only change in the x-direction. 
Your plot should look like this:

a) Look carefully at the bright side of the transition from dark to light. Do you perceive a bigger Mach band for the step than for the ramp?

b) Now, using the same parameters as were used for the ramp input in Lecture 5, find the response produced by the recurrent lateral inhibition equations to the step. Graph it.

c) Using the same parameters as were used for the ramp input in Lecture 5, find the response produced by the recurrent lateral inhibition equations to the ramp. Graph it.

d) Consider the model predictions. Does the model predict a bigger, smaller, or the same size Mach band (at the bright knee) for the step than for the ramp?

Exercise 2: Compare presence and absence of self-inhibition

Compare the response of the network to the step function with and without self-inhibition.  Do this using Manipulate[< your code here>, {{iterations, 5},1, 20, 1}] to overlay two graphs using the following initialization parameters:

size = 30;
spaceconstant =5;
maxstrength = 0.05;
iterations = 200; 
epsilon = .3;

iterations is the variable in the Manipulate’s “slider”. Your output should look something like for iterations = 3.

How many iterations are required before the network stabilizes with self-inhibition? Without self-inhibition?

Exercise 3: Feedforward steady-state solution

a) Given the above ramp input and weight matrix W, find the steady-state solution (i.e. matrix W’) to the limulus equation, by solving a set of simultaneous linear equations using the Inverse[] function that computes the inverse of a matrix. Use the matrix, W’, to find and plot the solution along with the input pattern corresponding to a ramp, and a step. Recall that there is a Mathematica  function, IdentityMatrix[], you might find useful. Your final plot should look like this:

b) This steady state solution can be viewed as a simple linear feedforward network. Use ListPlot[] to show that the effective receptive field of neurons in this feedforward network has a center-surround organization. You only need to show the plot for effective weights of neuron 14, i.e. the 14th row of W’.

Exercise 4: Convolution using ListConvolve[ ]

Let Wprime be the vector corresponding to the 14th row of W’ (from the previous exercise). Use ListConvolve[] and ListPlot[] to show that you get the same output to the ramp stimulus as you obtained in the previous exercise. 

HINTS: i) Align the kernel (i.e. Wprime) at position 14; ii) use 0 for the padding value in ListConvolve[]; iii) You only need one line of code -- just ListConvolve[ ] (with the correct variable settings) and ListPlot[].

Exercise 5 - Visualize the spatial pattern of spikes from an on-center cell

This exercise is an extension of the last one in your first problem set where we assumed spike emission at a point is a Bernoulli trial, and the probability of eliciting a spike was proportional to intensity. The idea was to visualize a spatial pattern of spikes at an instant where there were no spatial interactions. Here is one way of solving it:

origimage=Import["ExampleData/ocelot.jpg"];Image[Partition[RandomVariate[BernoulliDistribution[#]]& /@ Flatten[ImageData[origimage]],200]]

Our goal in the current exercise is to extend this visualization model by simulating a “snap-shot” of what an on-center, lateral inhibitory filtering of the input coming out of the retina en route to LGN might look like. This is more representative of daytime, called “photopic” vision.  Imagine we could view activity across a retinotopic slice across the optic nerve after viewing the picture for an instant. There are many ways to do this and with varying complexity, for example taking into account Poisson statistics of spike rates over time (where we could be explicit about what an instant is, e.g. 100 msec), including both temporal and spatial filtering, sigmoidal non-linearities, and inhomogeneous retinal sampling. But we’ll keep it fairly simple to emphasize concepts and tools.

We’ll add two additional components to the above code: 1) convolution with a linear center-surround spatial filter (as in stage 1 of the generic model); and 2) a threshold operation that simulates the response of an “on-center” cell (a particular form of stage 2; however, we won’t explicitly model threshold and saturation).

You can construct your own program as you wish, but it may help to try to use high-level image processing built-in functions in Mathematica as suggested in the hints in the following parts of this exercise:

a) Use LaplacianGaussianFilter[ ] on origimage. This will create a center-surround filter and do the convolution in one step. Let the kernel radius = 3.0. Short-hand for this filtering operation is LoG. Use ImageAdjust[ ] to normalize the filtered output to range between 0 and 1, and display the result. Call it ifiltered. It should look something like:

b) To simulate the output of an on-response unit, run the above filtered output through ImageClip[<out>,{0.5,1.0}], 
and again normalize the output, again using ImageAdjust[].  The idea is to set values of  ifiltered that are below a threshold of 0.5 to 0.0, and then use ImageAdjust[] to get output values to go from 0 to 1 again.

(The threshold of 0.5 is arbitrary and takes advantage of the  ImageAdjust[] operation in part 1 above, which “centered” the response range. One could be more sophisticated and have an adaptive component to set the threshold, e.g. based on a local average. Later on we’ll see how to model a sigmoidal threshold for an integrate-and-fire neuron probabilistically.)

Call your filtered clipped response iclipped. It should look like:

c) Now adapt the code from the above answer to problem 5 in the first problem set:
	Image[Partition[RandomVariate[BernoulliDistribution[#]]& /@ Flatten[ImageData[image]],200]]
to produce the “snap-shot” of the spiking activity. Call the image ispike. 

Use GraphicsRow[ ] to compare a) the original image b) the LoG filtered image c) the on-center response and d) the the “snap-shot” of the spiking activity. It should look like:

GraphicsRow[{origimage,ifiltered,iclipped,ispike}]

Note that the ispike picture is a “spike” rather than “rate” view of retinal output. It illustrates the high degree of sparseness in the spatial pattern of spikes at an instant. (There is another class of cells--the “off” cells--that signals the negative going filtered response.) Sparse responses raise the question of how the richness of perceptual experience emerges from such an apparently drastic reduction in information. This is why the visual problem is sometimes broken up into image encoding and decoding stages. This is a demo of an encoding stage. Keep the problems of encoding and decoding in mind for later in the course. The distinction isn’t hard and fast.

Exercise 6 - Regression and Widrow-Hoff learning

This exercise shows how the the supervised learning via the Widrow-Hoff training rule relates to statistical regression. (See Lecture 10.)

Make a function: rline[slope_,intercept_] to generate pairs of random numbers {x,y} where x ranges between 0 and 10, and whose y coordinate is a straight line with slope, slope_ and intercept, intercept_  but perturbed by additive uniform random noise over the range -2 to 2.

Generate a data set from rline with 200 samples with slope 11 and intercept 0.

Use the function Fit[] to find the slope and intercept of this data set. Here is an example of how Fit[] works:

Clear[x]

Fit[{{1,3},{2,5},{3,7.1}}, {1,x}, x]

If you wanted to fit a line through the origin, you would write: Fit[{{1,3},{2,5},{3,7.1}}, {x}, x], and a quadratic fit as: Fit[{{1,3},{2,5},{3,7.1}}, {1,x,x^2}, x].

Now implement a Widrow-Hoff algorithm to find the slope of the data. Assume the data goes through the origin, and initialize the slope to zero for the first iteration.  Use ListPlot to show the values of the slope as a function of the iterations from 1 to 200. What is a suitable range for the learning constant?

Exercise 7 - Use error backpropagation to learn to classify letters independent of orientation

This exercise illustrates the use of the classic “backprop” model of learning in multiple-layer perceptrons. It also shows how learning can drive outputs to threshold and saturation regions (interpreted as “false” and “true”), which allows one to do classification via a feedforward non-linear network. The application is to a “toy” version of the rotation invariance problem. The backprop algorithm doesn’t scale up well to complicated “real world” invariance problems. For that, other methods such as deep belief networks, are needed.

You can use Backpropagation.m  for this exercise. 

Define two classes of patterns, T's and C's. Each class has four members for rotations of 0,90,180, and 270 degrees. Train a non-linear feedforward net with one layer of hidden units to correctly classify the eight patterns as T or C. To get you started, here are exemplars for T and C on a 3x3 grid:

T1 = {{0.9,0.9,0.9},{0.1,0.9,0.1},{0.1,0.9,0.1}};
C1 = {{0.9,0.9,0.9},{0.9,0.1,0.1},{0.9,0.9,0.9}};

ArrayPlot[C1,Mesh->True,ColorFunction->GrayLevel,AspectRatio->1,Frame->False]

Fix the learning constant at 1. What is the minimum numbers of hidden units you can find that will still converge in under 600 iterations?

For this number of hidden units, what is the maximum value of the learning constant (eta) that still allows convergence?