Instructor: Daniel Kersten. Office: S212
Elliott Hall. Phone: 612 625-2589 email: kersten@umn.edu
Office hours: Mondays 9:30-10:30 am or by
appointment.
The visual perception of what is in the world is accomplished continually, instantaneously, and usually without conscious thought. The very effortlessness of perception disguises the underlying richness of the problem. We can gain insight into the processes and functions of human vision by studying the relationship between neural mechanisms and visual behavior through computer analysis and simulation. Students will learn about the anatomy and neurophysiology of vision and how they relate to the phenomona of perception. An underlying theme will be to treat vision as a process of statistical inference. There will be in-class programming exercises using the language Mathematica. No prior programming experience is required; however, some familiarity with probability, vector calculus and linear algebra is helpful.
(EV) Early Vision. Yuille and Kersten. In From Neuron to Cognition via Computational Neuroscience, M.A. Arbib, James J. Bonaiuto Editors, Cambridge MA: The MIT Press, in 2016 (preprint pdf)
Understanding Vision: Theory, Models, and Data. Li Zhaoping. 2014.(publisher page) (author's web outline)
(FV) Foundations of Vision. Wandell (web)
(NVN) The New Visual Neurosciences. John S. Werner and Leo M. Chalupa, edts. 2014. (Table of Contents pdf)
Mathematica is the primary programming environment for this course. Students who have registered for the course will have Google Docs access through the Psychology Department's site license.
Alternatives: Mathematica is available in several labs on campus, go to http://www.oit.umn.edu/computer-labs/software/index.htm
You may wish to purchase Mathematica for
Students see
http://www.wolfram.com/products/student/mathforstudents/index.html.
You can also access Mathematica on the CLA
servers:
If you never programmed before go here. If you have programming experience, go here.
For user help on using Mathematica, see: http://mathematica.stackexchange.com
http://ipython.org
http://jupyter-notebook-beginner-guide.readthedocs.org/en/latest/index.html
http://www.scipy.org
For an online course in using Python and PsychoPy for research in human vision see:
http://nbviewer.ipython.org/github/gestaltrevision/python_for_visres/blob/master/index.ipynb
Gopen, G. D., & Swan, J. A., 1990. The Science of Scientific Writing. American Scientist, 78, 550-558. (pdf)
Grade Requirements
There will be programming assignments and a final project.
The grade weights are:
Exercise/programming assignments: 55%
Final project presentations: 5 %
Final project : 40% (four parts: 2%+5%+5%+28%)
The programming assignments will use
the Mathematica programming environment. No prior
experience with Mathematica is necessary.
Assignment due By the midnight on the day due.
Late Policy:
Assignments turned in within 24 hours following the due
date will have 15% deducted from the assignment score.
Assignments turned in between 24 and 48 hours following
the due date will have 30% deducted from the score.
Assignments more than 48 hours late will receive a score
of zero.
Check this section before each class for recent additions and revisions.
(5036W Course material from 2015)
Lecture notes
are in Mathematica Notebook
and pdf format. You can
download the Mathematica
notebook files below to view with Mathematica
or Wolfram CDF Player
(which is free).
| University Calendar | Date | Lecture | Main Readings | Supplementary Material | Assignments due |
|
I. Introduction
|
Sep 6
|
1. Introduction to Computational Vision |
1.IntroToComputationalVision.nb Olshausen, B. A. (2013). Perception as an Inference Problem. In M. Gazzaniga (Ed.), The New Cognitive Neurosciences, 5th Edition (pp. 1–22). MIT Press. (pp. 1–18). MIT Press. (pdf)
|
Screencast: http://www.wolfram.com/broadcast/screencasts/handsonstart/
(WITH AUDIO) Kersten, D., & Yuille, A. (2003). Bayesian models of object perception. Current Opinion in Neurobiology, 13(2), 1-9. (pdf) |
|
|
Sep 11
|
2.Limits to Vision | 2.LimitsToVision.nb (pdf) Hecht, S., Shlaer, S., & Pirenne, M. H. (1942). Energy, quanta, and vision. Journal of General Physiology, 25, 819-840. (pdf) |
Barlow, H. B. (1981). Critical Limiting Factors in the Design of the Eye and Visual Cortex. Proc. Roy. Soc. Lond. B, 212, 1-34. (pdf) Baylor, D. A., Lamb, T. D., & Yau, K. W. (1979). Responses of retinal rods to single photons. Journal of Physiology, Lond., 288, 613-634. (pdf) Tinsley, J. N., Molodtsov, M. I., Prevedel, R., Wartmann, D., Pons, J. E. E., Lauwers, M., & Vaziri, A. (2016). Direct detection of a single photon by humans. Nature Communications, 7, 1–9. http://doi.org/10.1038/ncomms12172 (pdf)
|
||
|
Sep 13
|
3. The Ideal Observer | 3.TheIdealObserver.nb (pdf) |
Griffiths, T. L., & Yuille, A. (2008). A primer on probabilistic inference. In M. Oaksford and N. Chater (Eds.). The probabilistic mind: Prospects for rational models of cognition. Oxford: Oxford University Press (pdf). Try your luck against an ideal discriminator of dot density YesNoDotDiscriminationDemo.nb |
Upload Assignment #1 to Moodle |
|
|
Sep 18
|
4. Ideal observer analysis: Humans vs. ideals. Neurons vs. ideals |
4.IdealObserverAnalysis.nb (pdf)
|
Kersten and Mamassian (2008), Ideal observer theory. The New Encyclopedia of Neuroscience, Squire et al., editors (pdf). Geisler, W. S. (2011). Contributions of ideal observer theory to vision research. Vision Research, 51(7), 771–781.(pdf) Burgess, A. E., Wagner, R. F., Jennings, R. J., & Barlow, H. B. (1981). Efficiency of human visual signal discrimination. Science, 214(4516), 93-94. (pdf) Deneve, S., Latham, P. E., & Pouget, A. (1999). Reading population codes: a neural implementation of ideal observers. Nature Neuroscience, 2(8), 740–745. (pdf) Measure your absolute efficiency to
discriminate dot density using a 2AFC task 2AFCDotDiscriminationDemo.nb |
||
|
II. Image
formation,
pattern synthesis |
Sep 20
|
5.Psychophysics: tools & techniques |
SKEDetection2AFCInLineDisplay.nb Farell, B. & Pelli, D. G. (1999) Psychophysical methods, or how to measure a threshold and why. In R. H. S. Carpenter & J. G. Robson (Eds.), Vision Research: A Practical Guide to Laboratory Methods, New York: Oxford University (pdf) Press.http://psych.nyu.edu/pelli/ Morgenstern, Y., & Elder, J. H. (2012).
Local Visual Energy Mechanisms Revealed by Detection of
Global Patterns. Journal of Neuroscience, 32(11),
3679–3696. For a free Python psychophysics package, see: http://www.psychopy.org |
||
|
Sep 25
|
6. Bayesian decision theory & perception |
6.BayesDecisionTheory.nb Geisler, W. S., & Kersten, D. (2002). Illusions, perception and Bayes. Nat Neurosci, 5(6), 508-510. (pdf) |
|
||
|
Sep 27
|
7. Limits to spatial resolution, image modeling, introduction to linear systems |
7.ImageModelLinearSystems.nb Campbell, F. W., & Green, D. (1965). Optical and retinal factors affecting visual resolution. Journal of Physiology (Lond.), 181, 576-593. (pdf) |
Williams, D. R. (1986). Seeing through the photoreceptor mosaic. 9(5), 193-197. (pdf) |
Upload Assignment #2 to Moodle (correction 10/4/17) |
|
|
III. Early visual
coding
|
Oct 2
|
8. Linear systems analysis | 8.LinearSystemsOptics.nb (pdf) |
Tutorials: |
|
|
Oct 4
|
9. Features and filters. Spatial filter models of early human vision |
Campbell, F. W., & Robson, J. R. (1968). Application of Fourier Analysis to the Visibility of Gratings. Journal of Physiology 197, 551-566. (pdf) De Valois, R. L., Albrecht, D. G., & Thorell, L. G. (1982). Spatial frequency selectivity of cells in macaque visual cortex. Vision Res, 22(5), 545-559. (pdf) Watson, A. B. (1987). Efficiency of a model human image code. J Opt Soc Am A, 4(12), 2401-2417. (pdf) IPython demo of gabor filtering Steerable pyramids: http://www.cns.nyu.edu/~eero/steerpyr/ |
|||
|
Oct 9
|
10. Features and filters. Local processing & image analysis | 10.ImageProcessing.nb (pdf) Gollisch, T., & Meister, M. (2010). Eye Smarter than Scientists Believed: Neural Computations in Circuits of the Retina. Neuron, 65(2), 150–164. (pdf) |
Albrecht, D. G., De Valois, R. L., &
Thorell, L. G. (1980). Visual cortical neurons: are bars
or gratings the optimal stimuli? Science, 207(4426),
88-90.(pdf)
Adelson, E. H., & Bergen, J. R. (1991). The plenoptic function and the elements of early vision. In M. S. Landy & J. A. Movshon (Eds.), Computational Models of Visual Processing. Cambridge, MA: The MIT Press: A Bradford Book.(pdf) ClassificationImage demo (ReverseCorrelation.nb) Ahumada, A. J., Jr. (2002). Classification image weights and internal noise level estimation. J Vis, 2(1), 121-131. (pdf) |
Upload Assignment 3 to Moodle |
|
|
Oct 11
|
11. Coding efficiency: Retina |
Geisler, W. S. (2008). Visual perception and
the statistical properties of natural scenes. Annu Rev
Psychol, 59, 167-192. (pdf)
|
Laughlin, S. (1981). A simple coding
procedure enhances a neuron's information capacity. Z
Naturforsch [C], 36(9-10), 910-912.(pdf) |
||
|
Oct 16
|
12. Coding efficiency: Cortex |
12.SpatialCodingEfficiency.nb Simoncelli, E. P., & Olshausen, B. A. (2001). Natural image statistics and neural representation. Annu Rev Neurosci, 24, 1193-1216.(pdf) |
Laughlin, S. B., de Ruyter van Steveninck, R.
R., & Anderson, J. C. (1998). The metabolic cost of
neural information. Nat Neurosci, 1(1), 36-41.(pdf) |
||
|
IV.
Intermediate-level vision,
integration, grouping |
Oct 18
|
13. Edge detection | 13.EdgeDetection.nb (pdf) |
Hubel, D. H., & Wiesel, T. N. (1977). Ferrier lecture. Functional architecture of macaque monkey visual cortex. Proc R Soc Lond B Biol Sci, 198(1130), 1-59. (pdf) |
|
|
Oct 23
|
14. Objects and scenes from images. The visual cortical pathways and hierarchy. | 14.ScenesfromImages.nb Kersten, D. J., & Yuille, A. L. (2014). Inferential Models of the Visual Cortical Hierarchy. In M. S. Gazzaniga & G. R. Mangun (Eds.), The New Cognitive Neurosciences, 5th Edition (pp. 1–22). MIT Press. (pdf) |
Zhou H, Friedman HS, von der Heydt R (2000) Coding of border ownership in monkey visual cortex. J Neuroscience 20: 6594-6611. (pdf) | ||
|
Oct 25
|
15. Scene-based generative models | 15.SurfaceGeometryDepth.nb Kersten, D., Mamassian, P., & Yuille, A. (2004). Object perception as Bayesian Inference. Annual Review of Psychology, 55, 271-304. (pdf) |
|
|
|
|
Oct 30
|
16. Shape-from-X | 16.ShapeFromX.nb (pdf) |
Reflectance map: Shape from shading: Horn BKP (1986) Robot Vision. Cambridge MA: MIT Press. Ch 11 (pdf). Barron, J. T., & Malik, J. (2015). Shape, Illumination, and Reflectance from Shading. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(8), 1670–1687. http://doi.org/10.1109/TPAMI.2014.2377712 (pdf) Belhumeur, P. N., Kriegman, D. J., &
Yuille, A. (1997). The Bas-Relief Ambiguity. (pdf)
Johnson, M. K., & Adelson, E. H. (2011). Shape
Estimation in Natural Illumination. Computer Vision and
Pattern Recognition (CVPR), 2553–2560. |
Upload Assignment #4 to Moodle bluradaptationdemo (Webster et al. pdf) motion-induced-blindness demo (Bonneh et al. pdf) |
|
|
Nov 1
|
17. Shape from shading |
17.
Shape from shading.nb Lect_17Intro_Python.ipynb (source) (pdf)
|
Anaconda python installation recommended. We will use Juypter/IPython, a browser-based notebook interface for python.
See here for illustrations of IPython cell types, and here for a collection of sample notebooks. Look here for some good tips on installation, as well as the parent directory for excellent ipython-based course material on scientific computing using Monte Carlo methods. For a quick start to scientific programming, see: http://nbviewer.ipython.org/gist/rpmuller/5920182 For a comphrensive coverage of scientific python see:https://scipy-lectures.github.io And for a ground-up set of tutorials on python see: http://learnpythonthehardway.org/book/ Switching from matlab to python? http://wiki.scipy.org/NumPy_for_Matlab_User |
|
|
|
Nov 6
|
18. Motion: optic flow |
OpenCV python demo: OpticFlowSparse.ipynb |
Horn, B. K. P., & Schunck, B. G. (1981). Determining Optical Flow. Artificial Intelligence, 17, 185-203. (pdf) Optic Flow (2013) Florian Raudies, Scholarpedia, 8(7):30724. doi:10.4249/scholarpedia.30724 (link) (with available matlab code) Optic flow matlab code from Michael Black's lab. (link) Borst, A. (2007). Correlation versus gradient
type motion detectors: the pros and cons. Philos Trans R
Soc Lond B Biol Sci, 362(1479), 369-374. pdf) http://web.mit.edu/persci/people/adelson/illusions_demos.html EV: Section 2.4 FV: Chapter 10 |
|
|
|
Nov 8
|
19. Motion: biological, human perception | 19.MotionHumanPerception.nb Weiss, Y., Simoncelli, E. P., &
Adelson, E. H. (2002). Motion illusions as optimal
percepts. Nat Neurosci, 5(6), 598-604. |
Heeger, D. J., Simoncelli, E. P., &
Movshon, J. A. (1996). Computational models of cortical
visual processing. Proc Natl Acad Sci U S A, 93(2),
623-627. (pdf) EV: Section 4.4 |
||
|
Nov 13
|
20. Material perception |
V1 and lightness (pdf) Doerschner, K., Fleming, R. W., Yilmaz, O., Schrater, P. R., Hartung, B., & Kersten, D. (2011). Visual motion and the perception of surface material. Current Biology, 21(23), 2010–2016. (pdf) |
Fleming, R. W., Dror, R. O., & Adelson, E.
H. (2003). Real-world illumination and the perception of
surface reflectance properties. J Vis, 3(5), 347-368. (link) Adelson, E. H. (1993). Perceptual organization and the judgment of brightness. Science, 262, 2042-2044 (pdf) Boyaci, H., Fang, F., Murray, S. O., & Kersten, D. (2007). Responses to lightness variations in early human visual cortex. Curr Biol, 17(11), 989-993 (pdf)http://www.bilkent.edu.tr/~hboyaci/Vision/ http://web.mit.edu/persci/people/adelson/checkershadow_illusion.html http://gandalf.psych.umn.edu/users/kersten/kersten-lab/demos/transparency.html http://gandalf.psych.umn.edu/~kersten/kersten-lab/demos/MatteOrShiny.html |
Upload Assignment 5 to Moodle texture_classification_plot_gabor.ipynb
Upload Final project title & paragraph outline to Moodle |
|
|
Nov 15
|
21. Texture. |
Freeman, J., & Simoncelli, E. P. (2011). Metamers of the ventral stream. Nature Publishing Group, 14(9), 1195-1201. http://doi.org/10.1038/nn.2889 (pdf) |
Heeger DJ and Bergen JR, Pyramid Based Texture Analysis/Synthesis, Computer Graphics Proceedings, p. 229-238, 1995. (pdf). From: https://github.com/rbaravalle/efros A sample: out2.png |
||
|
Nov 20
|
22.Science writing (Thanksgiving week) | 22.ScienceWriting.nb (pdf) |
Gopen & Swan, 1990 (pdf) |
||
|
Nov 22
|
23.Perceptual integration | 23.PerceptualIntegration.nb (pdf) |
McDermott, J., Weiss, Y., & Adelson, E. H.
(2001). Beyond junctions: nonlocal form constraints on
motion interpretation. Perception, 30(8), 905-923. (pdf) Hillis, J. M., Ernst, M. O., Banks, M. S., & Landy, M. S. (2002). Combining sensory information: mandatory fusion within, but not between, senses. Science, 298(5598), 1627-1630.(pdf) Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415(6870), 429-433. (pdf) Stocker, A. A., & Simoncelli, E. (2008). A Bayesian model of conditioned perception. Advances in Neural Information Processing Systems, 20, 1409-1416. (pdf) IPython demo of ideal integration EV: Section 5 |
||
|
V. High-level
vision
|
Nov 27
|
24. Object recognition I |
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012). How does the brain solve visual object recognition? Neuron, 73(3), 415–434. (pdf) |
Liu, Z., Knill, D. C., & Kersten, D.
(1995). Object Classification for Human and Ideal
Observers. Vision Research, 35(4), 549-568. (pdf) Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences of the United States of America, 111(23), 8619-8624. (pdf) |
|
|
Nov 29
|
25. Object recognition II feeforward architectures | 25_Bidirectional_I.key.pdf (pdf) Ullman, S., Vidal-Naquet, M., & Sali, E. (2002). Visual features of intermediate complexity and their use in classification. Nat Neurosci, 5(7), 682-687. (pdf) |
Grill-Spector, K. (2003). The neural basis of object perception. Curr Opin Neurobiol, 13(2), 159-166.(pdf) Rao, R. P., & Ballard, D. H. (1999). Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci, 2(1), 79-87. (pdf) Bullier, J. (2001). Integrated model of visual processing. Brain Res Brain Res Rev, 36(2-3), 96-107. (pdf) Tenenbaum JB: Bayesian modeling of human concept learning. In Advances in Neural Information Processing Systems. Edited by Kearns MSS, Solla A, Cohn DA: Cambridge, MA: MIT Press: 1999.(pdf) |
Upload Assignment 6 to Moodle |
|
|
Dec 4
|
26. Object recognition III feedback architectures | 26_BidirectionalFeedback.key.pdf (pdf)
|
Torralba, A., Oliva, A., Castelhano, M. S., & Henderson, J. M. (2006). Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychol Rev, 113(4), 766-786. (pdf) Chikkerur, S., Serre, T., Tan, C., & Poggio, T. (2010). What and where: A Bayesian inference theory of attention. Vision Research, 50(22), 2233–2247.
|
||
|
Dec 6
|
27. Empirical evidence for bidirectional computations |
27.EmpiricalEvidenceBidirectionalProcessing(pdf)
|
Longuet-Higgins, H. C., & Prazdny, K.
(1980). The Interpretation of a Moving Retinal Image.
Proceedings of the Royal Society of London B, 208,
385-397. (pdf)
|
Upload a complete DRAFT of FINAL PROJECT to Moodle by Wednesday December 6th, 5 PM. |
|
|
Dec 11
|
28. Vision for action, spatial layout, heading. Homegeneous coordinates. | 28.SpatialLayoutScenes.nb Kalman filter notes (pdf) |
Upload your peer comments to Moodle by Monday Dec 11th | ||
|
Dec 13
(Last day of class) |
In Class Project Presentations |
Drafts returned to you with Instructor comments |
|||
|
Dec 21
|
Upload Final Revised Draft of Project to Moodle |
Goal: This course integrates the behavioral, neural and computational principles of perception. Students often find the interdisciplinary integration to be the most challenging aspect of the course. Through writing, you will learn to synthesize results from diverse and typically isolated disciplines. By writing about your project work, you will learn to think through the broader implications of your project, and to effectively communicate the rationale and results of your contribution in words. You will do a final page research report in which you will describe, in the form of a scientific paper, the results of an original computer program on a topic in computational vision.
Your final project will involve: 1) a computer program and; 2) a 2000-3000 word final paper describing your project. For your computer project, you will do one of the following: 1) Write a program to simulate a model from the computer vision literature ; 2) Design and program a method for solving some problem in perception. 3) Design and program a psychophysical experiment to study an aspect of human visual perception. The results of your final project should be written up in the form of a short scientific paper or Mathematica Notebook, describing the motivation, methods, results, and interpretation.
If you choose to write your program in Mathematica, your paper and program can be combined can be formated as a Mathematica notebook. See: Books and Tutorials on Notebooks. If you do your final project using Python, you can turn your paper in as a Jupyter notebook.
Your paper will be critiqued and returned for you to revise and resubmit in final form. You should write for an audience consisting of your class peers.
Completing the final paper involves 4 steps. Each step requires that you email a document to the teaching assistant.
Outline (2% of grade). You will submit a working title and paragraph outline by the deadline noted in the syllabus. These outlines will be critiqued in order to help you find an appropriate focus for your papers. (Consult with the instructor or TA for ideas well ahead of time).
Complete draft (5% of grade). A double-spaced, complete draft of the paper must be turned in by the deadline noted in the syllabus. Papers should be between 2000 and 3000 words. In addition to the title, author and date lines, papers must include the following sections: Abstract, Introduction, Methods, Results, Discussion, and Bibliography. Use citations to motivate your problem and to justify your claims. Cite authors by name and date, e.g. (Marr & Poggio, 1979). Citations should be original sources, not wikipedia. Use a standard citation format, such as APA. (The UM library has information on style guides, and in particular APA style.) Papers must be typed, with a page number on each page. Figures should be numbered and have figure captions. This draft will be reviewed by your instructor and one of you class peers. The point break down for the total 5% is: 2 pts for completing Introduction, 2 pts for completing Methods, 1 pt for completing Discussion)
Peer commentary (5% of grade). You will submit a written commentary (200 to 500 words) on a complete draft of one of your class peers. The project drafts and commentaries will be anonymous. The commentary should provide feedback to improve the quality and clarity of the writing.
Final draft (20% of grade) and "Cover letter" (8% of grade). The final draft must be turned in by the date noted on the syllabus. The "Cover letter" should describe how your revision addressed comments from your peer evaluator and from your instructor. It should itemize key criticisms together with a brief description of the changes you made to your draft manuscript.
Some Resources:
Student Writing Support: Center for Writing, 306b
Lind Hall and satellite locations (612.625.1893) http://writing.umn.edu.
NOTE: Plagiarism, a form of scholastic dishonesty and a disciplinary offense, is described by the Regents as follows: Scholastic dishonesty means plagiarizing; cheating on assignments or examinations; engaging in unauthorized collaboration on academic work; taking, acquiring, or using test materials without faculty permission; submitting false or incomplete records of academic achievement; acting alone or in cooperation with another to falsify records or to obtain dishonestly grades, honors, awards, or professional endorsement; altering, forging, or misusing a University academic record; or fabricating or falsifying data, research procedures, or data analysis. http://www1.umn.edu/regents/policies/academic/Code_of_Conduct.html. See too: http://writing.umn.edu/tww/plagiarism/ andhttp://writing.umn.edu/tww/plagiarism/definitions.html
