Introduction to Neural Networks

Problem Set 3

Exercise 1 - Efficient coding: Principal Components of Image Data Set

An important theory of neural receptive field organization says that the receptive field structure of sensory neurons, such as the simple cells of V1 are organized to efficiently representing incoming image information (Barlow). One way to efficiently code a large dimensional data set is to project the data onto axes that capture the most variance. PCA is one way to do this.

Convert (GrayLonesome256x256.jpg) to image data:

granite = ImageData[];

width = Dimensions[granite][[1]]

Use the following to break this large image into a series of subimages to use as samples

The idea is that each  subimage will be used as a statistical sample. We compute the outer product of each, and then average all 16 to get an estimate of the autocovariance matrix.

nregions = 16;swidth = width/nregions

subface=Table[Take[granite,{i*swidth+1,i*swidth+swidth},{j*swidth+1,j*swidth+swidth}],{i,0,nregions-1},{j,0,nregions-1}];

subfacelist=Table[0.0,{256}];Table[subfacelist[[i+16*(j-1)]]=N[Flatten[subface[[i,j]]]],{i,1,16},{j,1,16}];

Now we subtract off the mean.

subfacelist2 = Table[subfacelist[[i]] - Mean[subfacelist[[i]]], {i,1,256}];

Now calculate the autocovariance matrix for the resulting collection of images,  subfacelist2[i], i=1 to 256, and plot the autocovariance matrix with ArrayPlot[]. (Recall the use of the Flatten[] function.)

Using Mathematica's built-in functions, calculate the eigenvectors and eigenvalues of the autocovariance matrix and use ListPlot to plot the eigenvalues.

Display the first 32 eigenvectors as "eigenpictures". (Recall the use of the Partition[] function. Use appropriate  options to keep the display compact.) Note how the eigenvectors are ordred with increasingly fine spatial scale.

Exercise 2 - Calculate and plot the marginals for a mixture distribution

Multivariate gaussian distributions are often inadequate to model real-life problems, that for example might involve more than one mode. One solution is to approximate more general distributions by a sum or mixture of gaussians.

Clear[mix];

r1=0.4*{{1,.6},{.6,1}};
r2=0.4*{{1,-.6},{-.6,1}};
m1 = {1,.5}; m2 = {-1,-.5};
ndist1 = MultinormalDistribution[m1, r1];
ndist2 = MultinormalDistribution[m2, r2];

mix[x_] := 0.4 PDF[ndist1, x] + .6 PDF[ndist2, x];

gg1=ContourPlot[mix[{x1,x2}],{x1,-2,2},{x2,-2,2},PlotRange->Full,ImageSize->Small]

Marginals for mixture

Write Mathematica code to calculate and then plot the two marginal distributions for the above mixture distribution. Your plot should look like:

By inspecting your plot, estimate the modes for each of the marginal distributions. How do they compare with the mode of the joint distribution?

Exercise 3 - Derive the optimal rule for integrating two noisy measurements to estimate the mean

A recurring theme in probabilistic computations is to combine information based on knowledge of how the variables are distributed probablistically. We’ve seen this in the context of combining data with priors, for example to update the prior during learning, and also when passing messages. Work in theoretical neuroscience by Ma et al. (2006) showed conditions under which neural populations could achieve optimal estimates by summing spikes. Ma, “Bayesian inference with probabilistic population codes. Nature Neuroscience, 9, 1423-8”. Their model was motivated in part by psychophysical observations showing that humans often combine sensory cues optimally.

Suppose there are two cues ((Subscript[x, 1]), (Subscript[x, 2])) to the depth, μ,  of an object. The generative model can be described by: (Subscript[x, i]) = μ + (Subscript[n, i]), where (Subscript[n, i]) ~ N[0, (Subscript[σ, i])], for i = 1, 2. In other words, assume the Gaussian case, with conditional independence (as represented in graph below). 

Assume you’ve separately measured the means,  (Subscript[μ, i]) and standard deviations, (Subscript[σ, i]) of your depth estimates for each cue. Show that the maximum a posteriori estimate of the depth using the combined cues is:

μ = ((Subscript[r, 1])/((Subscript[r, 1]) + (Subscript[r, 2])))(Subscript[μ, 1]) +  ((Subscript[r, 2])/((Subscript[r, 1]) + (Subscript[r, 2])))(Subscript[μ, 2])

where (Subscript[r, i]) = 1/(Subscript[σ, i]^2), and (Subscript[μ, 1]) and (Subscript[μ, 2]) are the estimates of the means obtained separately. HINT: See the methods used in the last part of Lecture 16 on computing the MAP estimate of the mean given several measurements.