Introduction to Neural Networks
U. Minn. Psy 5038
Spring, 1999

Lecture 8
The Linear associator, continued

Initialization

Off[SetDelayed::write]
Off[General::spell1]

Review of linear association model

Outer product

Outer product, g(f^T)models the increase in weight strength when input f is associated with an output g:

Outer[Times,Array[g,3],Array[f,3]]//MatrixForm

Learning & recall

1. Learning
Let {fn,gn} be a set of input/output activity pairs. Memories are stored by superimposing new weight changes on old ones. Information from many associations is present in each connection strength.

2. Recall
Let f be an input possibly associated with output pattern g.  For recall, the neuron acts as a linear summer:

3. Condition for perfect recall
If {fn} are orthonormal, the system shows perfect recall:

Exercise

Try this either with paper and pencil, or with Mathematica.
Let f1 and f2 be two orthogonal, normalized input patterns:

and g1, g2 two output patterns:

Form the outer product:

Test for "recall" by feeding f1 as input to W. Stimulate W with f2. What happens? Add the outer product:

to the previous W matrix. Now test for recall on stimulation with f1, and f2. What do you find?

Heteroassociation

In this and the next section, we will use Mathematica to simulate a process of association between image representations of the letters, T, I, and P. You will learn more about how to manipulate lists in Mathematica. And you will learn some of the limitations of linear recall. There are several simple exercises/questions you should try to answer.

Simulation of heteroassociative learning - Learning "IT"

Stimuli

If after seeing I, the letter T follows, you might expect that T would become associated with I. The letter I might later act as a stimulus that should elicit T as a response.

Imatrix = {
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}};

Tmatrix = {
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 1, 0, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 1, 0, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 1, 0, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 1, 0, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 1, 0, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 1, 0, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 1, 0, 0, 0, 0, 0}, 
  {0, 1, 1, 1, 1, 1, 1, 1, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}};

Pmatrix = {
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 
  {0, 1, 0, 0, 0, 0, 0, 0, 0, 0}, 
  {0, 1, 0, 0, 0, 0, 0, 0, 0, 0}, 
  {0, 1, 0, 0, 0, 0, 0, 0, 0, 0}, 
  {0, 1, 1, 1, 1, 1, 0, 0, 0, 0}, 
  {0, 1, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 1, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 1, 0, 0, 0, 0, 1, 0, 0, 0}, 
  {0, 1, 1, 1, 1, 1, 0, 0, 0, 0}, 
  {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}};

normalize[x_] := N[x/Sqrt[x.x]];
Tv = normalize[Flatten[Tmatrix]];
Iv = normalize[Flatten[Imatrix]];
Pv = normalize[Flatten[Pmatrix]];

Sidenote: Making images into vectors: Flatten[ ] and Partition[ ]

Flatten[] takes a list of lists and turns it into a list of elements, that is, it removes all of the inner braces:

Flatten[{{a,b},{c,d}}]

Partition[] is sort of like an inverse for Flatten[] and takes a list of elements and structures it back into a list of lists:

Partition[%,2]

For our purposes, Flatten[] turns a matrix representing a 2D picture (e.g. the letter I, or T) into a vector that we can store in a weight matrix memory. Later, we use Partition[] to turn whatever the matrix remembers into a 2D picture for comparison with the input picture originally learned.

ListDensityPlot[Tmatrix, PlotRange -> {0, 1}];

ListDensityPlot[Partition[Tv,10],PlotRange->{0,.3}];

Learning an association  I -> T

Learning

Let's use the outer product to represent the change in the synaptic weights caused by the simultaneous activity of T and I which assuming a Hebb-type rule, is proportional to the product of the activities:

Weights = Outer[Times,Tv,Iv];

Now if sometime later, the Weights matrix is "stimulated" with the letter I, it produces as a response the letter T:

Recall: Remembering T from I

response = Weights.Iv;

ListDensityPlot[Partition[response,10]];

Exercise

What if Tv was the input? What if a random vector was the input? What response would you expect?

Add the association T -> P to the mix

Learning P from T, storing it with the association between I and T

Weights = Weights + Outer[Times,Pv,Tv];

Recall: Stimulate with T: Response? P, I, or a mixture?

response = Weights.Tv;

ListDensityPlot[Partition[response,10]];

Exercise

If you look carefully, you can see some evidence for interference. Why might you expect this based on the two inputs Iv and Tv? Try comparing the dot products of the various inputs.
Can you think of a network for recall that might help to reduce the interference?

Exercise

The DensityPlot functions don't always give the best way of seeing variations in a function or list. Try ListPlot[response]. What do you notice?

Exercise

The plot function automatically scales the plot range so that white corresponds to the maximum value in the list. Use Max[Tv] to find the peak value in a list.

Add the association I -> P to the mix

Learning P & I, store it with the associations between I & T, P & T

Weights = Weights + Outer[Times,Iv,Pv];

Recall: Stimulate with P: Response? T, I, or a mixture?

response = Weights.Pv;

ListDensityPlot[Partition[response,10]];

Autoassociation

If f=g, then we have an autoassociative system. There is only one set of units, and each element potentially connects to each other element.  Later we will see how this architecture is used in non-linear networks. Autoassociation stores information about the relationships between the elements or features of a stimulus. We can show how this kind of knowledge can be used later to predict or reconstruct missing information. Neural networks of this sort build internal models of the statistical structure of the ensemble they are exposed to. Practical examples are systems of networks that can learn about their own special environment (e.g. an expert on single-lane country roads, another on two-lane highways, and another on four-lane divided highways. when given a new environment, these experts can "compare notes" to see how well this new environment fits their internal models. Which ever expert "knows" the new environment the best, gets to have its expert driver  drive the car!)

Reconstructive property

Autoassociation can reconstruct missing parts of a stimulus.

x=(({{(Subscript[f, 1])}, {(Subscript[f, 2])}, {.}, {.}, {(Subscript[f, m])}, {(Subscript[g, 1])}, {.}, {.}, {.}, {(Subscript[g, n])}}))

Suppose a whole pattern x, consists of two parts {f1,f2,f3}, and {g1,g2,g3,g4}, and the association of vector x with itself is represented by the outer product in matrix W:

f={f1,f2,f3,0,0,0,0};
g={0,0,0,g1,g2,g3,g4};
x=f+g;
W=Outer[Times,x,x];

x

x = f + g

Sometime later, we input a version of x, but with "missing" elements--i.e. x with some elements set to zero--namely, vector f. What do we get in response?

Simplify[W.f]

So if f is normalized, each sum of squared elements of f is equal to 1, and W.f is equal to x. 
The matrix W restores f to the full pattern x.

Exercise

What does it mean to have a "missing" part of a pattern?

Autoassociation includes heteroassociation

At first it may seem that an autoassociative system is a more restrictive type of association than heteroassociation. But if we form a new vector f' in which we stack f on top of g, then autoassociation outer product matrix contains within it, the heteroassociation between f and g.

Question:

What can you say about the eigenvectors of the weight matrix from autoassociative learning?

Autoassociative example with TIP pictures

Learn about T, learn about I, and store the associations together by superimposing their weight matrices

Clear[Weights];
Weights = Outer[Times,Tv,Tv] + Outer[Times,Iv,Iv];

Sometime later, stimulate the network with an impoverished T, missing some bits

forgettingT =
Join[Take[Tv,(Dimensions[Tv][[1]]-30)],Table[0, {30}]];

Take[m,n] returns a list containing the first n elements of m. Take[m,-n] returns a list containing the last n elments of m.

ListDensityPlot[Partition[forgettingT,10]];

Recall of the original T, from the missing bits

rememberingT = Weights.forgettingT;

ListDensityPlot[Partition[rememberingT,10]];

Interference: Corrupt T again, this time with some other random bits missing

Let's do something a little more drastic to T. We'll randomly "delete" pixels of the picture:

pepper = Table[Random[Integer,1],{Dimensions[Tv][[1]]}];
peppermatrix = DiagonalMatrix[pepper];
forgettingT = peppermatrix.Tv;

ListDensityPlot[Partition[forgettingT,10]];

rememberingT = Weights.forgettingT;
ListDensityPlot[Partition[rememberingT,10]];

ListPlot[Partition[rememberingT,10][[5]],
	PlotRange->{0,.5},AxesOrigin->{0,-.25}];

Note: You can get information about the options as well as the functions with a ?? query:

??ListPlot
??AxesOrigin

Interference: Corrupt T, with added noise

forgettingT = Tv;
forgettingT[[59]] = .27;
ListDensityPlot[Partition[forgettingT,10]];

rememberingT = Weights.forgettingT;
ListDensityPlot[Partition[rememberingT,10]];

Although the memory looks pretty good, it is not perfect because although Tv and Iv were almost orthogonal, with a cosine of about .09, they were not perfectly orthogonal. In fact, we can get a measure of how close rememberingT is to Tv  in terms of the cosine of the angle between them:

Tv.normalize[rememberingT]

Interference with more autoassociations: I, T, and now P too

If we have the connection matrix, Weights store another letter, P, then we will begin to get even more interference when we try to recall T from a fragment of T:

Weights = Weights + 
			Outer[Times,Pv,Pv];

rememberingT = Weights.forgettingT;

ListDensityPlot[Partition[rememberingT,10]];

This is because the patterns we've stored are not mutually orthogonal, and in particular, P is too close to I and T:

{Iv.Pv, Tv.Pv, Tv.Iv}

Exercise - include a threshold & making functions listable

Define a non-linear threshold, step[x_], which when applied to rememberingT removes the interference. A critical paramter is the threshold. How could you make the threshold adaptive?
Note: When you define a new function, it is not necessarily "Listable". This means that to apply step[] to remembertingT, you would have to use the Map[] function:

The Map[] function is used often enough, that Mathematica  has a short-hand:

Map[f,{a,b,c}]

f /@ {a,b,c}

Map[step,rememberingT]

The following is equivalent:

step /@ rememberingT

Alternatively, you can define your function to be listable with

SetAttributes[step,Listable]

Then you can apply step directly to the list rememberingT, and then the function will be applied successively to each element of the list:

step[rememberingT]