Introduction to Neural Networks
U. Minn. Psy 5038

Daniel Kersten

Problem Set 1

Problem Set 1

Exercise 1

Let  w be a weight vector representing a template pattern. You can think of w as representing the strengths of a set of synapses to a neuron. Let {s} be a collection of input pattern vectors all of unit length (norm).  The values of the elements of s represent the strengths of the inputs to the neuron. Show theoretically that the cross-correlator gives maximum response to the pattern that matches the "form" of the template pattern. Recall that the response of the cross-correlator is determined by the dot-product of the input vector with the weight vector.

Exercise 2

a) Define a semi-linear  "squashing" function, thresh[s], that satisfies:  
				                        0 for s < 0; 
				                        s for 1 >=s >= 0; 
				                        1 for s> 1. 
(Hint: There are lots of ways to do this "from scratch", by building up rules, by using Which[]...but take a look at Piecewise[]).
Plot thresh[s] from s = -2 to 2.
b) There are neurons in the primary visual cortex of mammals called "simple cells". One model for these cells is a linear cross-corelator followed by a thresholding non-linearity (e.g. the half-wave rectification of a diode).  Analogous to the receptive field properties of an ommatidium of the horseshoe crab, the receptive field weights of this cross-correlator typically show a lateral inhibitory organization. In one dimension, a much reduced model weight vector could look like this:

w = {-2,-1,6,-1,-2};

Use the above weight vector w, and your thresh[] function to calculate the response of the model simple cell. Calculate the response to the three inputs, s :

s1 = {-1, -.5, 3, -.5, -1}/25.;
s2 = {2, 1, 0, 1, 2} /10.;
s3 = {0, -7, 0, 7, 0}/25.;

A response can be zero either because of the threshold, or because the input pattern is a poor match to the pattern of receptive field weights. 
Explain the responses to s2 and s3 in terms of these two factors.

Exercise 4

Later on, when we study back-propagation we will  need to use the derivative of a non-linear squashing function in our derivation of a learning rule for neural networks. For this reason, it is useful to have a squashing function that has a closed form solution for the derivative. 
Use Mathematica to find the derivative of the function, 1/(1 + Exp[-r]). Call the derivative  dsquash[].

squash[r_] := 1/(1 + Exp[-r/.25]);

Plot dsquash from r = -2 to 2.

Mathematica Hint: You can't directly define and plot the function dsquash[x_]:=D[squash, etc..]. But you can:
1)  Use Evaluate[ ] to force evaluation when dsquash is an argument of Plot[]. 
2) You may wish to use the Mathematica rule for replacing a variable with a value in an expression. This would also enable you to define the derivative function all on one line. 
3) ...but the easiest way is to use the prime notation, e.g. f'(x) when defining the derivative of f(x).

Exercise 4

(Uses Walsh basis set from Lecture  4)

a) Given the vector:

h = {1,2,3,4,5,6,7,8};

use ListPlot to graph the "spectrum" of h with respect to the 8-dimensional orthonormal Walsh basis vectors given in the lecture. (You can just copy and paste them in your answer Notebook.). Recall that the spectrum consists of the  magnitudes of the projections of h onto the Walsh functions. 

b) Verify your answer by reconstructing h from the projections, i.e. as a linear sum of the basis vectors.