Introduction to Neural Networks
U. Minn. Psy 5038
Spring, 1998
Daniel Kersten

Problem Set 1

Problem Set 1

Exercise 1

A. Let  w be a weight vector representing a template pattern. Let {x} be a collection of pattern vectors all of unit length.  Show theoretically that the cross-correlator gives maximum response to the pattern which matches the form of the template pattern. 

The cross correlator takes the dot product between the input vetor and the vector(s) in the weight matrix. The dot product can be written as:
a.b = |a| |b| cos theta
If the input vector and the vector(s) in the weight martix are all normalized (aka all have vector length of "1"), then the dot product can be written as:
a.b = 1 * 1 cos theta
Since we know the largest value of cos theta is "1" when theta (the angle between the vetors) is "0", then the cross correlator will give the maximum response "1" when the pattern matches the form of the corresponding template pattern.

B. Now write a Mathematica  program to demonstrate this property of cross-correlators. Use the Table function to make fill a 32x32 matrix R with random numbers. Use Random[].  Then define a function normalize[x] that takes as input a vector x, and returns a normalized version of x. Use the Table function again to turn R into a matrix R2 whose rows are normalized to unit length. Calculate the matrix product of R2 with the 8th row of R2. Use ListPlot to show that the maximum of the product occurs at element 8.  Make several more plots using other rows of R2, and show the maximum always occurs at the row that matches the input vector.

\(\(
R = Table[Random[], {x, 1, 32}, {y, 1, 32}]; 

    normalize[x_]\  := \ N[x/Sqrt[x . x]]; 

    R2\  = \ Table[normalize[R[\([x]\)]], {x, 1, 32}]; \)\)

\(\(
\(ListPlot[R2 . R2[\([8]\)], \ PlotJoined -> True, \ 
      PlotRange -> {0.5, 1}]; \)\)\)

\(\(


    \(ListPlot[R2 . R2[\([10]\)], \ PlotJoined -> True, \ 
      PlotRange -> {0.5, 1}]; \)\)\)

\(\(
\(ListPlot[R2 . R2[\([20]\)], \ PlotJoined -> True, \ 
      PlotRange -> {0.5, 1}]; \)\)\)

Exercise 2

Use a set of rules to define a semi-linear  "squashing" function, limit[x], which is:  
				                        -1 for x < -1; 
				                        x for 1 >=x >= -1; 
				                        1 for x> 1. 
Plot limit[x] from x = -2 to 2.

\(
Clear[limit]\)limit[x_] := -1 /; x < -1limit[x_] := x /; 1 >= x >= -1\(limit[x_] := 1 /; x > 1
\)Plot[limit[x], {x, -2, 2}]

Exercise 3

Define a function  dsquash[] to be equal to the derivative of the logistic function:

squash[r_] := 1/(1 + Exp[-r]);

Clear[dsquash]dsquash[r_]  =  D[1/(1  +  Exp[-r]), r]  /. r -> xPlot[dsquash[x],  {x, -2, 2}] ;

Plot dsquash from r = -2 to 2. 
(Hint: you may have to use the rule for replacing a variable with a value in an expression). Later on, we will  need to use the derivative of the non-linear squashing function in our neural networks. For this reason, it is useful to have a squashing function that has a closed form solution for the derivative.

Exercise 4

There are neurons in the primary visual cortex of mammals called "simple cells". One model for these cells is a linear cross-corelator followed by a thresholding non-linearity (e.g. the half-wave rectification of a diode).  The receptive field weights of the cross-correlator typically show a "center-surround" organization. In one dimension, a model weight vector could look like this:

w = {-2,-1,6,-1,-2};

Define a threshold function thresh[s] that is zero for s less than zero, and equal to s for values of s greater than or equal to 0. 

Use the above weight vector w, and your thresh[] function to model the response of a simple cell. What is the response of your cell to an input x:

a) x = {-1,-.5,3,-.5,-1}

or

b) x = {2, 1, 0, 1, 2} ?

\(
Clear[thresh]\)thresh[s_]  :=  0  /;  s < 0thresh[s_]  :=  s  /;  s >= 0\(x = {\(-1\), \(- .5\), 3, \(- .5\), \(-1\)}; 
x2 = {2, 1, 0, 1, 2}; 

    w = {\(-2\), \(-1\), 6, \(-1\), \(-2\)}; 
R = thresh[w . x]\)R2 = thresh[w . x2]

Exercise 5

Express the vector:

h = {1,2,3,4,5,6,7,8} ;

as a linear sum of normalized Walsh vectors (see Lecture 4). Plot the "spectrum" of h. In particular use ListPlot to show the spectrum, which consists of the eight values of the projections of h onto the 8 Walsh functions. Verify your answer by reconstructing h from the projections.

\(\(
v1\  = \ {1, \ 1, \ 1, \ 1, \ 1, \ 1, \ 1, \ 1}; 

    v2\  = \ {1, \(-1\), \(-1\), \ 1, \ 1, \(-1\), \(-1\), \ 1}; 

    v3\  = \ {1, \ 1, \(-1\), \(-1\), \(-1\), \(-1\), \ 1, \ 1}; 

    v4\  = \ {1, \(-1\), \ 1, \(-1\), \(-1\), \ 1, \(-1\), \ 1}; 

    v5\  = \ {1, \ 1, \ 1, \ 1, \(-1\), \(-1\), \(-1\), \(-1\)}; 

    v6\  = \ {1, \(-1\), \(-1\), \ 1, \(-1\), \ 1, \ 1, \(-1\)}; 

    v7\  = \ {1, \ 1, \(-1\), \(-1\), \ 1, \ 1, \(-1\), \(-1\)}; 

    v8\  = \ {1, \(-1\), \ 1, \(-1\), \ 1, \(-1\), \ 1, \(-1\)}; 
\)\)

\(normalize[x_]\  := \ N[x/Sqrt[x . x]]; 
w1 = normalize[v1]; 

    w2 = normalize[v2]; 
w3 = normalize[v3]; 
w4 = normalize[v4]; 

    w5 = normalize[v5]; 
w6 = normalize[v6]; 
w7 = normalize[v7]; 

    w8 = normalize[v8]; 
normWalsh = {w1, w2, w3, w4, w5, w6, w7, w8}; \)

\(\(
\(spectrumh = {h . w1, h . w2, h . w3, h . w4, h . w5, h . w6, 
        h . w7, h . w8}; \)\)\)

\(\(
\(ListPlot[spectrumh, PlotStyle -> {RGBColor[1, 0, 0]}, 
      PlotJoined -> True]; \)\)\)

Reconstruction from the projections

spectrumh . normWalshh == spectrumh . normWalsh

Optional Exercises

Exercise 1

Write a Mathematica  function that computes the McCulloch-Pitts Inclusive OR and AND functions.

McCullochPitts[a_,  b_,  t_]  :=  
      If[ a  +  b  >=  t,  1,  0]

McCulloch-Pitts Inclusive OR

\(\(
\ 
    Flatten[Table[{a, b, McCullochPitts[a, \ b, 1]}, \ {a, \ 0, 1}, 
          \ {b, \ 0, \ 1}], \ 1]\  // \ TraditionalForm\)\)

McCulloch-Pitts AND

\(\(
\ 
    Flatten[Table[{a, b, McCullochPitts[a, \ b, 2]}, \ {a, \ 0, 1}, 
          \ {b, \ 0, \ 1}], \ 1]\  // \ TraditionalForm\)\)

Exercise 2

Write a Mathematica  function that includes the noise term in the generic connectionist model.

<< Statistics`ContinuousDistributions`

\(ndist\  = \ NormalDistribution[0.5,  .1]; 

    squash[r_]\  := \ 1/\((1\  + \ Exp[\(-r\)])\); \)

\(w\  = \ {\(-2\), \(-1\), 6, \(-1\), \(-2\)}; 

    x = {\(-1\), \(- .5\), 3, \(- .5\), \(-1\)}; \)

Function for generic connectionist model with noise.

genericNeuron[x_, w_, dist_] := N[squash[x . w] + Random[dist]];

\(\(
genericNeuron[x, w, ndist]\)\)