Introduction to Neural Networks

Psy 5038W, Fall 2009, 3 credits
Psychology Department , University of Minnesota

Place: S150 Elliott Hall
Time: 9:45-11:00 MW


Course home pages:
courses.kersten.org

Instructor: Daniel Kersten, Office: S212 Elliott Hall, Phone: 625-2589, email: kersten@umn.edu
Office hours: Mondays 11:00 to 12:00 and by appointment.

TA: Michael Blank, Office: email: blan0138@umn.edu
Office hours: 11:00-12:00 on Tuesdays, in Elliott Hall S501, and by appointment.

Course description. Introduction to large scale parallel distributed processing models in neural and cognitive science. Topics include: linear models, statistical pattern theory, Hebbian rules, self-organization, non-linear models, information optimization, and representation of neural information. Applications to sensory processing, perception, learning, and memory.

General Readings and Software

Grade Requirements

There will be a mid-term, final examination, programming assignments, as well as a final project. The grade weights are:

 


Outline & Lecture Notes

(NOTE: Updated links to lecture material below will be revised and posted on the day of the lecture
--if you want a preview, check out lectures from 2007)


All lecture notes are in Mathematica Notebook and pdf format. You can download the Mathematica notebook files below to view with Mathematica or MathPlayer (which is free).

 

Date

Lecture

Additional Readings & supplementary material

Assignments
due

I.

1

Sep 9

Introduction (pdf file)|Mathematica notebook

Instructions for accessing Mathematica

Mathematica screencast
Neuroscience tutorial (Clinical, Wash. U.)
Top 100 Brain Structures

fMRI4Newbies

 

2

Sep 14

The neuron (pdf file)| Mathematica notebook

Hodgkin-Huxley.nb
Koch & Segev, 2000 (pdf)
Meunier & Segev, 2002 (pdf)

 

3

Sep 16

Neural Models, McCulloch-Pitt (pdf file)| Mathematica notebook

Koch, C., & Segev, I. (Eds.). (1998) (pdf)

 

4

Sep 21

Generic neuron model (pdf file)| Mathematica notebook    

II.

5

Sep 23

Lateral inhibition (pdf file)| Mathematica notebook

Hartline (1972) (pdf)

 

 

6

Sep 28

Matrices (pdf file)| Mathematica notebook   PS 1. Introduction to Mathematica , vectors, cross-correlation

7

Sep 30 Linear systems, learning & Memory (pdf file)| Mathematica notebook    
III.

8

Oct
5

Linear Associator (pdf file)| Mathematica notebook

einstein64x64.jpg
shannon64x64.jpg

 

9

Oct 7

Sampling, Summed vector memory (pdf file)| Mathematica notebook ProbabilityOverviewNN.nb  

10

Oct 12

Non-linear networks, Perceptron (pdf file)| Mathematica notebook

LeNet-5

 

PS 2. Lateral inhibition

11

Oct 14

Regression, Widrow-Hoff (pdf file)| Mathematica notebook    

12

Oct 19

Multilayer feedforward nets, Backpropagation (pdf file)| Mathematica notebook

Backpropagation.m

Poirazi,Brannon & Mel (2003) (pdf)

Williams (1992) (pdf)

 
IV.

13

Oct 21

Science writing (pdf)
(Mathematica notebook)

Gopen & Swan, 1990 (pdf)
Hopfield (1982)(pdf)

 

 

14

Oct 26

MID-TERM

MID-TERM STUDY GUIDE

MID-TERM (16%)

15

Oct 28

Networks and Visual Representation (pdf file)| Mathematica notebook Carrandini, Heeger, Movshon (1996)(pdf)  

16

Nov 2

Neural Representation and coding (pdf file) Mathematica notebook

Sanger (2003) (pdf)

Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C., & Fried, I. (2005).(pdf)

 

17

Nov 4

Self-organization, Principal Components Analysis (pdf file)| Mathematica notebook Supplement: ContingentAdaptation.nb  

18

Nov
9

Discrete Hopfield network (pdf file)| Mathematica notebook

Hopfield (1982) (pdf)
Marr & Poggio (1976) (pdf)

PS 3. Learning

GrayLonesome256x256.jpg

19

Nov 11

Graded response Hopfield network (pdf file)| Mathematica notebook

Hopfield (1984) (pdf)
Durstewitz et al. (2000) (pdf)

PROJECT IDEAS
Sample abstracts from the past

For demonstration style projects, see the Wolfram Demonstration site.

A specific example.

20

Nov 16

Boltzmann machine (pdf file)| Mathematica notebook

Sculpting the energy function, interpolation (pdf file)| Mathematica notebook)

21

Nov 18

Adaptive maps (pdf file)| Mathematica notebook
smallRetinaCortexMap.nb
GraylefteyeDan.jpg

 

 

Final project title & paragraph outline (2%)

22 Nov 23 Probability
(pdf file)| Mathematica notebook

Griffiths and Yuille (2006) (pdf)

Jordan, M. I. and Bishop. C. MIT Artificial Intelligence Lab Memo 1562, March 1996. Neural networks.

PS 4 Hopfield network
V.
23 Nov 25 More on neural coding, Generative models,Bayes nets and inference
(pdf file)
Mathematica notebook
Knill & Pouget (2004) (pdf)
Pouget et al. (2006) (pdf)
 
 24 Nov 30 Belief Propagation
(pdf)
Mathematica notebook

Pattern Recognition and Machine Learning, Chapter 8.
Christopher M. Bishop (pdf)

Weiss Y. (pdf)

 
25 Dec 2 EM
(pdf)
Mathematica notebook
   
26 Dec 7 Bayes decision theory (pdf)
Mathematica notebook
Fisher's linear discriminant notes (pdf) Mathematica notebook

REVISED DATES in RED

Complete Draft of Final Project (5%) Due December 7

27 Dec 9

Kalman filter, fisher discriminant (pdf)
Mathematica notebook

Kalman notes (pdf)

Rao & Ballard 1999 (pdf)
Wolpert et al (1995) (pdf)

Peer comments on Final Project (5%:) Due December 14
28 Dec 14

SVMs, Bias/Variance, Wrap-up & Review
(pdf)
Mathematica notebook

Bias/Variance notes (pdf)
SVMS:

Jäkel et al., (2009)

Jäkel et al., (2007)

Mathematica SVMs: Nilsson, Björkegren & Tegnér (2006)
MathSVMv7.nb

Drafts returned with Instructor comments December 16
    Dec 16 EXAM (offical last instruction day) FINAL STUDY GUIDE FINAL EXAM (16%)
  Dec 23 (last day of fall semester)   Submit Final Revised Draft of Project (28%)
         

 


Final Project Assignment.

This course teaches you how to understand cognitive and perceptual aspects of brain processing in terms of computation. Writing a computer program encourages you to think clearly about the assumptions underlying a given theory. Getting a program to work, however, tests just one level of clear thinking. By writing about your work, you will learn to think through the broader implications of your final project, and to effectively communicate the rationale and results in words.

Your final project will involve: 1) a computer simulation and; 2) a 2000-3000 word final paper describing your simulation. For your computer project, you will do one of the following: 1) Devise a novel application for a neural network model studied in the course; 2) Write a program to simulate a model from the neural network literature ; 3) Design and program a method for solving some problem in perception, cognition or motor control. The results of your final project should be written up in the form of a short scientific paper, describing the motivation, methods, results, and interpretation. Your paper will be critiqued and returned for you to revise and resubmit in final form. You should write for an audience consisting of your class peers. You may elect to have your final paper published in the course's web-based electronic journal.

Completing the final paper involves 3 steps:

  1. Outline. You will submit a working title and paragraph outline by the deadline noted in the syllabus. These outlines will be critiqued in order to help you find an appropriate focus for your papers. (2% of grade). (Consult with the instructor or TA for ideas well ahead of time).
  2. Complete draft. You will then submit a complete draft of your paper (2000-3000 words). Papers must include the following sections: Abstract, Introduction, Methods, Results, Discussion, and Bibliography. Use citations to motivate your problem and to justify your claims. Figures should be numbered and have figure captions. Cite authors by name and date, e.g. (Marr & Poggio, 1979). Use a standard citation format, such as APA. Papers must be typed, with a page number on each page.Each paper will be reviewed with specific recommendations for improvement. (5% of grade)
  3. Peer commentary. Each student will submit a paragraph on an anonymous paired project draft (5% of grade)
  4. Final draft. You will submit a final revision for grading. (28% of grade). The final draft must be turned in by the date noted on the syllabus. Students who wish to submit their final papers to be published in the class electronic journal should turn in both paper and electronic copies of their reports.

If you choose to write your program in Mathematica, your paper and program can be combined can be formated as a Mathematica notebook. See: Books and Tutorials on Notebooks.

Your paper will be critiqued and returned for you to revise and resubmit in final form. You should write for an audience consisting of your class peers.

Some Resources:

Student Writing Support: Center for Writing, 306b Lind Hall and satellite locations (612.625.1893) http://writing.umn.edu.
Online Writing Center:http://www.owc.umn.edu

NOTE: Plagiarism, a form of scholastic dishonesty and a disciplinaryoffense, is described by the Regents as follows: Scholasticdishonesty means plagiarizing; cheating on assignments or examinations;engaging in unauthorized collaboration on academic work; taking,acquiring, or using test materials without faculty permission; submittingfalse or incomplete records of academic achievement; acting alone or incooperation with another to falsify records or to obtain dishonestlygrades, honors, awards, or professional endorsement; or altering,forging, or misusing a University academic record; or fabricating orfalsifying of data, research procedures, or data analysis.http://www1.umn.edu/regents/policies/academic/StudentConductCode.html

© 2003, 2005, 2007 Computational Vision Lab, University of Minnesota, Department of Psychology.