Introduction to Neural Networks
U. Minn. Psy 5038

Daniel Kersten

Lecture 1-Introduction

Goal

Understand the functioning of the brain as a computational device. Tools to explain brain & behavior. Relation to connectionist, neuromorphic, computational neuroscience research, and cognitive science.

Relation to Cognitive Science

Cognitive Science: The interdisciplinary study of the acquisition, storage, retrieval and utilization of knowledge.
Problems: perception, learning, memory, planning, action
Often, we don't know how to solve a problem even in principle. For others, we have solutions, but they don't resemble how a biological system might solve the problem.
What kinds of problems can large interconnected systems of model neurons solve? What are the limitations? What are the strengths?
How do neural networks relate to the larger field of statistical pattern recognition?

Understanding the relation between brain and behavior requires

A multidisciplinary approach
	Multiple levels of explanation.

Multidisciplinary approach

Three primary areas or disciplines influence current neural network research:

Neuroscience, computational neuroscience

Understand the basic building blocks or "hardware" of the nervous system
	these are: nerve cells or neurons, and their connections, the synapses
Our emphasis is on: large scale neural networks. Requires great simplification in the model of the 
	neuron...in order to compute and theorize about what large numbers of them can do.
Compare with other areas of Computational Neuroscience that emphasize the biology. Here we emphasize "brain-style""
	computation. Often wrong in detail, but driven by a curiosity about how the complex processes of
	perception, and memory  work.
What can these large scale neural systems do? That is, what can they compute? And how?

Computational theory, mathematics, statistical pattern recognition

Statistical inference, engineering (information and communication theory), statistical physics and computer science.
Provide the tools and analogs to abstract and formalize for analysis and simulation.
One of the characteristics of this course is to try to relate the neural models to statistical methods of inference and regression in order to understand the computational principles and power behind a neural implementation.
What should these large scale neural systems compute? What are the ways in which information is represented? How can a system be designed to get from input to output representations?

Behavioral sciences, psychology, cognitive science and ethology

Understand what subsystems are supposed to do as a functioning organism in the environment.
Psychology & Computational theory =>The brain is NOT a general purpose computer.

Multiple levels of explanation

Functional/Behavioral level

Psychology/Cognitive Science/Ethology tells us what is actually solved by functioning behaving organisms. Descriptions of behavior.

Statistical Inference level

Theories of pattern recognition, inference &  estimation.
Functionalities supported by neural network computing provide a useful way of categorizing models in terms of the computational tasks required:
1. Learning input/ouput mappings from examples (learning as regression, classification boundaries)
	associative memory (->neural networks: Hopfield net, back-prop, local minima can be useful.)
2. Inferring outputs from inputs (continuous estimation, discrete classification)
	memory recall, perceptual inference
	optimization or constraint satisfaction (->neural networks: Hopfield net, Boltzmann machine,  global minimum is desired, local minima  are problems)
3. Modeling data (learning as probability density estimation)
	self-organization of sensory data into useful representations or classes (e.g principal components analysis, clustering)
	(Can view 1. Learning input/output mappings as a special case)

Neural network level: Algorithms, implementation

Algorithms: Mathematics of computation tells us what is computable and how. Practical limits. Parallel vs. serial.
Input and output representation, and algorithms for getting from input to output. Programming rules, data structures.
Implementation: Wetware, hardware.
Neuroscience, neurophysiology and anatomy tell us the adequacies and inadequacies of our modeling assumptions.

Emphasis in this course

Understand high-level functions such as vision, pattern recognition, learning, memory, inference and control. In the brain these functions involve large-scale systems each with many "modules" and 10s to 100s of thousands of neurons in each. Appropriate level tends towards more abstract where we can manage the complexity through mathematical models.
The interaction between levels of analysis considers a function (e.g. pattern recognition), the theory to understand the function (e.g. through statistical inference), and how the function may be realized in a neural system (neural networks):

Overview of the Brain

Before we look at models of neurons and their interactions, let us get an overview of the large scale context. 
Understanding function means understanding how an organism's information processing is determined by the structure of its environmental inputs (e.g. natural images, objects to be avoided, places to go), and the nature of its outputs (e.g. estimates of shapes of objects to be manipulated, terrain to walk on, movements, and decisions). The brain doesn't operate in isolation, these inputs and outputs are intimately tied to the sensory and motor neurons that make up the peripheral nervous system, as well as the physical make-up of the  body itself.

The brain has both surface and interior structures. Surface structures that are visible in the side view below are: frontal, temporal, parietal and occipital lobes, and the cerebellum. But apart from the cerebellum, it isn't totally obvious where one part stops and another begins. Landmarks are the sulci (valleys) and gyri (bumps). E.g. the lateral fissure (sulcus) is perhaps the easiest to spot. It separates the temporal lobe from the frontal and parietal lobes.
There are internal components too: Thalamus (sensory and motor relays), hypothalamus (control of endocrine activity, temperature, food intake, etc..), basal ganglia (relay for motor behavior), limbic system (expression of emotion), medulla (part of lower brain stem, breathing, heart rate).
We believe that much of what makes us interesting as humans, our thoughts, imaginations, words and actions, depends on having a large and complex cortex.

A common view shows the surface or cortex--the gray matter.

From: http://www.utdallas.edu/~kilgard/brain.jpg

Check out: http://psychology.uwo.ca/fmri4newbies/PrimeronCorticalSulci.html

The anatomy provides an important, but static view of the brain. Recent exciting progress in functional imaging provides dynamic pictures that illustrate relationships between human functions (seeing, imagining, etc.) are related to various cortical areas. (From our own lab, see: http://gandalf.psych.umn.edu/~kersten/kersten-lab/Perceptual.html

Levels of organization, scales

From: Churchland & Sejnowski.  (10,000 A to a micron)

Example: Visual system

Let's take a look from two points of view: 1) information flow through a specific system--the visual system; 2) levels of organization at successive stages.

At a very coarse spatial scale, we know that we have eyes and a portion of the brain that processes the incoming images. This system enables us to recognize and manipulate objects, and that help us to navigate.

The above picture ends at primary visual cortex (V1, area 17), but there are more than 30 other visual areas after that.

Pathways out of the eye

From:	Milner, D., & Goodale, M. (1995). The Visual Brain in Action. Oxford: Oxford University Press.

Visual cortex & maps

Higher visual areas in cortex. Maps.

From:	Churchland, P. S., & Sejnowski, T. J. (1992). The Computational Brain. Cambridge, MA: MIT Press.
Adapted from Van Essen and Anderson 1990. See too, Felleman and Van Essen, 1991.

Cortical layers & circuits

Structure within maps. 
Casual inspection shows that the brain has gross structure. What is not immediately apparent  is that structures  do not consist of randomly connected nerve cells. There is a medium-level organization into multiple functional groupings. 
The neocortex has 6 more or less distinguishable layers, there is a microorganization into vertical columns. In the primary visual cortical area (V1, see above figure), there are ocular dominance and orientation selectivity columns which are believed to form a functional unit called a hypercolumn (1 to 2 mm). Each hypercolumn takes into account local image intensities and colors to represent information or features for a single point of the visual field.

http://www.ualr.edu/~klwennstrom/hypercolumn.gif

Retina, circuits, neurons & molecules

Rods, cones. Rods & rhodopsin molecules. Synapses. Different types of neurons: Spike-generating neurons, the ganglion cells.
Networks of neurons that behave as "image filters". M & P pathways.

Some brain specs

"cortico-centric" cognitive neuroscience
The human brain is:: volume - 1.4 liters, Cortex 2 mm, volume 0.32 liters
Cortex: 1.6 x 10^10 neurons, with about 4000 synapses/neuron, about 6x10^13 connections.

Getting started with Mathematica

Mathematica vs. Matlab vs. neural network simulation packages.
Features: symbolic & numeric, notebook, entire manual is built-in.

Front-end and Notebooks: Organize, outline, document, program 
Kernel: Separate program does the calculation

Numerical Calculations. You can do arithmetic. For example, type 5+7 as shown in the cell below, and then hit the "enter" key. Try other operations, 5^3, 4*3 (note that 4 3,
where a space separates the digits is also interpreted as multiplication). Note that if you
try division, e.g. 2/3, you get the exact answer back. To get a decimal approximation,
type N[2/3].

5+7

N[2/3]

You can go back and select an expression by clicking on the brackets on the far right. These brackets are features of the Macintosh interface and serve to organize text and calculations into a Notebook with outlining features. You can group or ungroup cells for text, graphs, and expressions in various ways to present your calculations. Explore these options under Cell in the menu.  You can see the possible cell types under the Style menu.

Find the square root of: 12345678987654321

Built-in functions. Mathematica has a very large library of built-in functions. They all begin with an uppercase letter. You can get information about a function, e.g. for the exponential of a function, or for plotting graphs by selecting the function (e.g. Exp) and going to "Find Selected Function" in the Help menu. Or you can enter:

?Exp

?Plot

If you type two question marks before a function, ??Plot, you'll get more information. Try it. What does the Random function do?

Plot3D[Sin[x + y],{x,-2,2},{y,-2,2}]

Try rotating the above 3D plot.

Mathematica 6 has a very useful and easy-to-program Manipulate[] function. E.g.

Manipulate[DensityPlot[Sin[a*x + b*y],{x,-2,2},{y,-2,2}],{a,1,6},{b,1,6}]

Online tutorials

I haven't found a good beginning tutorial yet for Mathematica 6. However, we will cover some more basics in the next lecture. You can also try  intro.nb. This has been "auto-updated" for version 6, so there is new material that won't be in there, and old stuff mainly relevant to earlier versions.
Check out the Demonstrations center at the Wolfram Mathematica site for some cool examples, such as:

Manipulate[Module[{bluecol,colfunc},
bluecol[n_]:=Blend[{Black,Blue,White},n];
If[usecol, colfunc=bluecol, colfunc=GrayLevel];

Graphics[{colfunc[left],Rectangle[{0,0},{3,3}],colfunc[right],Rectangle[{3,0},{6,3}],colfunc[mid],Rectangle[{1,1},{2,2}],Rectangle[{4,1},{5,2}]}]],{{left,.35,"left square"},.2,.8},{{right,.75,"right square"},.2,.8},{{mid,.5,"middle squares"},.2,.8},{{usecol, True,""},{True->"blues", False->"grays"},ControlPlacement->Bottom}]

"The Simultaneous Contrast Effect" from The Wolfram Demonstrations Project http://demonstrations.wolfram.com/TheSimultaneousContrastEffect/

References

Bear, M. F., Connors, B. W., & Paradiso, M. A. (2006). Neuroscience : exploring the brain (3rd ed.). Baltimore, MD: Lippincott Williams & Wilkins.
Churchland, P. S., & Sejnowski, T. J. (1992). The Computational Brain . Cambridge, MA: MIT Press.
Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cereb Cortex, 1(1), 1-47.
Kandel, E. R., Schwartz, J. H., & Jessell, T. M. (2000). Principles of neural science (4th ed.). New York: McGraw-Hill, Health Professions Division.
Levy, I., Hasson, U., & Malach, R. (2004). One picture is worth at least a million neurons. Curr Biol, 14(11), 996-1001.
Milner, D., & Goodale, M. (1995). The Visual Brain in Action . Oxford: Oxford University Press.
Purves, D. (2007). Neuroscience (4th ed.). Sunderland, Mass.: Sinauer Associates, Publishers.
Rolls, E.T., & Treves, A. (1998). Neural networks and brain function. Oxford ; New York: Oxford University Press.
Zeki, S. (1993). A Vision of the Brain. Oxford: Blackwell Scientific Publications.

Links

http://www.med.harvard.edu/AANLIB/cases/caseM/case.html
http://thalamus.wustl.edu/course/