Introduction to Neural Networks
U. Minn. Psy 5038
Spring, 1999
Daniel Kersten

Problem Set 2
(see Lecture 5 - Lateral inhibition)

Exercises

Exercise 1

This exercise is easiest if you copy and modify bits of code from Lecture 5 used to illustrate Mach bands. Make a DensityPlot of a 30x30 pixel step function (rather than the ramp function used in Lecture 5). As with the ramp in Lecture 5 illustrating Mach bands, the pattern should only change in the x-direction. Now, using the same parameters as were used for the ramp input in Lecture 5, find the response produced by the recurrent lateral inhibition equations. 
Look at the brighter knee of the density plot. Do you perceive a bigger Mach band for the step than for the ramp?
Consider the model predictions. Does the model predict a bigger Mach band (at the bright knee) for the step than for the ramp?

\(\(Clear[step]; 
step[x_] := \ 0.2\  /; \ x <= 15; 

    step[x_]\  := \ 0.8\  /; x >= 16; 
size\  = 30; 

    e := \ Table[step[i], {i, 1, size}]; 
e1 = e; 

    picture = Table[e1, {i, 1, 30}]; 

\)\)

\(
Clear[ramp]; 
low\  = \ 0.2; \ hi\  = \ 0.8; 

    ramp[x_]\  := \ low\  /; \ x < 10\)ramp[x_]  := 
      ((hi - low)/10) x  +  (low - (hi - low))  /;  
        x >= 10  &&  x < 20ramp[x_]  :=  hi  /;  x >= 20\(rampE := \ Table[ramp[i], {i, 1, size}]; 
e2 = rampE; 

    picture2 = Table[e2, {i, 1, 30}]; \)

\(\(
ListDensityPlot[picture, \ Frame -> False, Mesh -> False, 
      PlotRange -> {0, 1}]; 


    ListDensityPlot[picture2, \ Frame -> False, Mesh -> False, 
      PlotRange -> {0, 1}]; 
\)\)

\(\(
size = 30; 
spaceconstant = 5; 
maxstrength = 0.05; 

    iterations = 15; 
epsilon = 0.3; \)\)

\(\(
\(g0 = 
      ListPlot[e, PlotRange -> {{0, 30}, {\(-0.5\), 1.0}}, 
        DisplayFunction -> Identity]; \)\)\)

\(\(
f = Table[Random[], {size}]; 

    W = Table[
        N[\(-maxstrength\)\ Exp[\(-Abs[i - j]\)/spaceconstant], 1], {i, 
          size}, {j, size}]; 
T[f_] := f + epsilon \((e + W . f - f)\); 
    \)\)

\(\(\(r0 = 
      ListPlot[rampE, PlotRange -> {{0, 30}, {\(-0.5\), 1.0}}, 
        DisplayFunction -> Identity]; \)
\)\)

\(\(
f2 = Table[Random[], {size}]; 

    W = Table[
        N[\(-maxstrength\)\ Exp[\(-Abs[i - j]\)/spaceconstant], 1], {i, 
          size}, {j, size}]; 
T2[f_] := f + epsilon \((rampE + W . f - f)\); 
    
\)\)

\(\(
r1 = 
      ListPlot[Nest[T2, f2, iterations], PlotJoined -> True, 
        PlotRange -> {{0, 30}, {0, 1.0}}, 
		DisplayFunction -> Identity]; 
    
Show[r0, r1, 
      Graphics[Text[iterations\ "iterations", {size/2, \(-0.4\)}]], 
	
      DisplayFunction -> $DisplayFunction]; 
\)\)

\(\(
g1 = 
      ListPlot[Nest[T, f, iterations], PlotJoined -> True, 
        PlotRange -> {{0, 30}, {0, 1.0}}, 
		DisplayFunction -> Identity]; 
    
Show[g0, g1, 
      Graphics[Text[iterations\ "iterations", {size/2, \(-0.4\)}]], 
	
      DisplayFunction -> $DisplayFunction]; \)\)

Exercise 2

Use the function NestList[] (instead of the Nest[] used in Exercise 1) to produce a 12x30 matrix, called responses whose rows are the responses or state vectors for each of 12 sequence of iterations or states in Exercise 1. Although we can't view the dynamical trajectory in state space of a 30 dimensional state vector (30 neurons), you can extract the activities at each of the twelve states of just 2 neurons at locations 15, and 16. Do this using the matrix extraction rule which produces submatrix from matrix m:

submatrix = m[[ Range[rowi, rowj], Range[columni,columnj] ]];

Then use ListPlot to plot the twelve points. Do not use PlotJoined->True, but do use PlotRange, and AxesOrigin to present the state vector evolution clearly.

responses = NestList[T, f, 12];

\(

submatrix = responses[\([Range[1, 12], Range[15, 16]]\)]\)\(\(ListPlot[submatrix, PlotRange -> {{0.0, 0.45}, {0.4, 0.65}}, 
      AxesOrigin -> True]; \)		\)

Exercise 3

Make two graphs to compare the response of the network to the step function with and without self-inhibition using the following initialization parameters:

size = 30;
spaceconstant =5;
maxstrength = 0.5;
iterations= 10; 
epsilon = .3;

How many iterations are required before the network stabilizes with self-inhibition? Without self-inhibition?

\(e2 = Table[\(-maxstrength\), {i, size}]; 
w0 = DiagonalMatrix[e2]; 

    W2 = W - w0; \)

\(\(\(T2[f_] := f + epsilon \((e + W2 . f - f)\); \)
\)\)

\(\(\(For[iterations = 5, iterations < 15, 

      g2 = ListPlot[Nest[T, f, iterations], PlotJoined -> True, 
          PlotRange -> {{0, 30}, {0, 1.0}}, 
		
          DisplayFunction -> Identity]; 

      g3 = ListPlot[Nest[T2, f2, iterations], 
          PlotStyle -> {RGBColor[1, 0, 0]}, PlotJoined -> True, 
          PlotRange -> {{0, 30}, {0, 1.0}}, 
		
          DisplayFunction -> Identity]; 

      Show[g0, g2, g3, 
        Graphics[Text[iterations\ "iterations", {size/2, \(-0.4\)}]], 
	
        DisplayFunction -> $DisplayFunction]; \(iterations++\)]; \)
\)\)

Exercise 4

Given the above ramp input and weight matrix, find the steady-state solution to the limulus equation, by solving a set of simultaneous linear equations using the Inverse[] function that computes the inverse of a matrix. Plot the solution along with the input pattern e, as we did in the lecture. By the way, there is a Mathematica  function, IdentityMatrix[], you might find useful.
This steady state solution can be viewed as a simple linear feedforward network. Use ListPlot to show that the effective receptive field of neurons in this feedforward network have a center-surround organization. Just show the plot for effective weights of neuron 15.

\(\(
size\  = \ 30; 
spaceconstant\  = 5; 
maxstrength\  = \ 0.05; 
    \)\)

\(\(
\(W\ 	 = \ 
      Table[N[\(-maxstrength\)\ Exp[\(-Abs[i - j]\)/spaceconstant], 1], 

        		{i, size}, {j, size}]; \)\)\)

\(\(
e\ 	 = \ 
      Join[Table[0, {i, N[size/3]}], 
        Table[i/N[size/3], 
		{i, N[size/3]}], \ 
        Table[1, {i, N[size/3]}]]; 
id\  = \ IdentityMatrix[30]; 

    g0\ 	 = \ 
      ListPlot[e, \ PlotRange\  -> \ {{0, 30}, {\(-0.5\), 1.0}}, 
		
        DisplayFunction\  -> \ Identity]; \)\)

\(\(W2\  = \ id\  - \ W\ ; 
invW\  = \ Inverse[W2]; 

    response\  = \ invW . e; 
\)\)

\(\(
g3\  = \ 
      ListPlot[response, PlotJoined -> True, 
		
        PlotRange\  -> \ {{0, 30}, {\(-0.5\), 1.0}}, 
		
        DisplayFunction\  -> \ Identity]; 

    Show[g0, g3, DisplayFunction\  -> \ $DisplayFunction]
\)\)

\(\(ListPlot[invW[\([15]\)], PlotJoined -> True, \ 
      PlotRange -> {\(- .1\), 1}\ ]
\)\)