Introduction to Neural Networks

Problem Set 5

A. Write down a factorization of the joint distribution for the directed graph
B. Which variables are the parents of C?
C. Which variables are conditionally independent given A?

This exercise illustrates conditional dependence or simple “explaining away”. We have two random “causes”, the values of flips of coin1 (x= 1 or 0) and coin2 (y = 1 or 0), which together influence a third variable z, also taking on values 1 or 0.

Generative model: Priors

pcoin1=BernoulliDistribution[.75];
pcoin2=BernoulliDistribution[.4];

Samples from priors

c1:=RandomVariate[pcoin1] (* x *)
c2:=RandomVariate[pcoin2] (* y *)

Generative model for the data (or likelihood):

myXnor[x_,y_]:=Which[x==0&&y==0,1,x==0&&y==1,0,x==1&&y==0,0,x==1&&y==1,1];

z[xc1_,yc2_] := myXnor[xc1,yc2]; (* z *)

Now make lots of data, stored in jointsample. Assume there we have sufficient data so jointsample is representative of the population joint distribution:

jointsample = Table[{x=c1,y=c2,z[x,y]},{10000}];

To answer the questions below, you can get by with just a few functions: Total[ ], Length[ ], and Count[list,pattern], with appropriate patterns. N[ ] could also come in handy.

A. Calculate the marginals of x,y, and z

B. Assuming that jointsample is representative of the population joint distribution, and without actually calculating probabilities, but using the data in jointsample, answer the questions:

If z = 1, what is the probability of the event {x=1,y=0}?
If z = 1, what is the probability of the event {x=1,y=1}?

Suppose we only have an algorithm to sample from univariate Normal distributions, but would like to draw samples a bivariate normal distribution, say with mean vector: {(Subscript[μ, 1]),(Subscript[μ, 2])}, and covariance matrix: {{1,ρ},{ρ,1}}.

mndist = MultinormalDistribution[{(Subscript[μ, 1]),(Subscript[μ, 2])},{{1,ρ},{ρ,1}}];

If we know the conditional distributions of mndist, then we can use the Gibbs sampler to draw samples  alternatively from the two conditional distributions.

With a little effort, one can show that the conditional distribution of p(x1 | x2) is normally distributed, 

p(x1 | x2) = 𝒩(μ,(σ^2)) →  𝒩((Subscript[μ, 1]) + ρ *(x2 -(Subscript[μ, 2])), 1- (ρ^2))

so  x1 ∼ 𝒩((Subscript[μ, 1]) + ρ *(x2 -(Subscript[μ, 2])), 1- (ρ^2))
	
and similarly,

p(x2|x1) = 𝒩(μ,(σ^2)) →  𝒩((Subscript[μ, 2]) + ρ *(x1 - (Subscript[μ, 1])),  1 - (ρ^2))

with x2 ∼ 𝒩((Subscript[μ, 1]) + ρ *(x2 -(Subscript[μ, 2])), 1- (ρ^2)).

Set up and run a Gibbs Sampler with  ρ = 0.9 and  {(Subscript[μ, 1]),(Subscript[μ, 2])} = {2,1}. Choose your own starting sample.
Run it for n = 1,000 iterations.

Using Histogram[], plot estimates of the marginal distributions of x1 and x2.

Using ListPlot[], display a scatter plot of the 1000 samples.

Plot the scatter plot on top of a contour plot of the theoretical bivariate distribution, gcontour:

gcontour =ContourPlot[PDF[ndist,{x,y}]^.05,{x,-8,8},{y,-8,8},ColorFunction->”BlueGreenYellow”,ImageSize->Small]

Plot the sequence of the first 100 traces  on top of gcontour. It should look something like, depending on your initial values:

HINT: Feel free to adapt the Gibbs sampling code in Lecture 20 .

In this exercise, we revisit the T vs. C classification problem, but using Mathematica’s Classify[ ] function. Critically, we will test how well two different classification methods, each trained on just four rotations, generalize to rotated versions of the letters T and C.

Note that all of the Classify methods are “discriminative”--they have no built-in knowledge of the underlying generative structure, i.e. that training instances result from rotations in the plane.

Classify[] will accept image formatted data, which makes it easy to visualize the stimuli and results. The default measure of distance between images is EuclideanDistance[].

This exercise will illustrate pros and cons of using  high-level classifier program---lots under the hood. 
The exercise also illustrates how different methods generalize differently as a consequence of the underlying data structure, and the range of training instances.

Stimulus generation

The following code generates training stimuli. There are four rotations for each letter spaced 90 degrees apart. Each image has 64x64 pixels.

Δangle = 90;
iTlist=Table[ImageRotate[,deg Degree, Full],{deg,0,360-Δangle,Δangle}];
iClist=Table[ImageRotate[,deg Degree, Full],{deg,0,360-Δangle,Δangle}];

Define the training set using rules

Here’s the syntax:
	Thread[{data1, data2, data3} -> {label1, label2, label3}]
	which outputs:
	
	{data1 -> label1, data2 -> label2, data3 -> label3}
	
	Use ConstantArray[] to fill the lists with the labels. And Join[] to concatenate both sets of training pairs.

trainingset =Flatten[Join[{Thread[iTlist->ConstantArray["T",Dimensions[iTlist][[1]]]],
Thread[iClist->ConstantArray["C",Dimensions[iClist][[1]]]]}]];

Ask Mathematica to set up a linear support vector machine classifier

csvm=Classify[trainingset,Method->{"SupportVectorMachine","KernelType"->"Linear"}];

Check setup, and look at Mathematica’s default choices

Use ClassifierInformation[csvm]. 
(You can get other kinds of information using:
ClassifierInformation[csvm, “MethodDescription”];
ClassifierInformation[csvm,”Properties”];)

ClassifierInformation[csvm]

Train and visually inspect how well the model generalizes to rotations in the plane

For simplicity, we will only test at rotations spaced by 45 degrees.

Manipulate[{ImageRotate[iTlist[[1]],deg Degree, Full],cneuralnet[ImageRotate[Image[Partition[#,64]]&/@iTlist[[1]],deg Degree, Full]],ImageRotate[iClist[[1]],deg Degree, Full],csvm[ImageRotate[iClist[[1]],deg Degree, Full]]}, {deg,0,360,45}]

How many wrong classifications out of the 8 rotations? You can just visually count them.

Try a linear kernel function with an expanded training set, where instances are now 10 degrees apart (rather than 90 degrees)

csvm=Classify[trainingset,Method->{"SupportVectorMachine","KernelType"->"Linear"}];
ClassifierInformation[csvm]

How many wrong classifications out of the 8 tested rotations?

Try a polynomial kernel function with the original training set containing only four instances of each letter

Use a polynomial of degree 3.

csvm=Classify[trainingset,Method->{"SupportVectorMachine","KernelType"->"Polynomial","PolynomialDegree"->3}];
ClassifierInformation[csvm]

Now how many wrong classifications out of the 8 rotations?

Try a nearest neighbors classifier with the original training set containing only four instances of each letter

csvm=Classify[trainingset,Method->{"NearestNeighbors"}];
ClassifierInformation[csvm]

How many wrong classifications out of the 8 rotations?

Which, if any, method(s) generalize perfectly?

Which method(s) generalize best, i.e. with the fewest number of training pairs?