Introduction to Neural Networks
U. Minn. Psy 5038
Spring, 1999
Daniel Kersten

Lecture 1

Goal

Understand the functioning of the brain as a computational device. Explain brain & behavior. Relation to connectionist, neuromorphic, computational neuroscience research, and cognitive science.

Relation to Cognitive Science

Cognitive Science: The interdisciplinary study of the acquisition, storage, retrieval and utilization of knowledge.
Problems: perception, learning, memory, planning, action
Often, we don't know how to solve a problem even in principle. For others, we have solutions, but they don't resemble how a biological system might solve the problem.
What kinds of problems can large interconnected systems of model neurons solve? What are the limitations? What are the strengths?
How do neural networks relate to the larger scheme of statistical pattern recognition?

Understanding the relation between brain and behavior requires

A multidisciplinary point of view.
	Multiple levels of explanation.

Multidisciplinary point of view

Three primary areas or disciplines influence current neural network research:

Neuroscience, computational neuroscience

Understand the basic building blocks or "hardware" of the nervous system
	these are: nerve cells or neurons, and their connections, the synapses
Our emphasis is on: large scale neural networks. Requires great simplification in the model of the 
	neuron...in order to compute and theorize about what large numbers of them can do.
Compare with Computational Neuroscience I, II, III. Emphasizes the biology. Here we emphasize "brain-style""
	computation. Often wrong in detail, but driven by a curiosity about how the complex processes of
	perception, and memory  work.
What can these large scale neural systems do? That is, what can they compute? And how?

Computational theory, mathematics, statistical pattern recognition

Statistical inference, engineering (information and communication theory), statistical physics and computer science.
Provide the tools and analogs to abstract and formalize for analysis and simulation.
One of the characteristics of this course is to try to relate the neural models to statistical methods of inference and regression in order to understand the computational principles and power behind a neural implementation.
What should these large scale neural systems compute? What are the ways in which information is represented? How can a system be designed to get from input to output representations?

Behavioral sciences, psychology, cognitive science and ethology

Understand what subsystems are supposed to do as a functioning organism in the environment.
Psychology & Computational theory =>The brain is NOT a general purpose computer.

Multiple levels of explanation.

A useful set of distinctions to bear in mind (Marr).

Functional level

Psychology/Cognitive Science/Ethology tells us what is actually solved by functioning systems.

Computational level

Functionalities supported by neural network computing provide a useful way of categorizing models in terms of the computational tasks required:
1. Learning input/ouput mappings from examples (learning as regression, classification boundaries)
	associative memory (e.g. Hopfield net, back-prop, local minima can be useful.)
2. Inferring outputs from inputs (continuous estimation, discrete classification)
	memory recall, perceptual inference
	optimization or constraint satisfaction (e.g. Hopfield net, Boltzmann machine,  global minimum is desired, local minima  are problems)
3. Modeling data (learning as probability density estimation)
	self-organization of sensory data into useful representations or classes (e.g principal components analysis, clustering)
	(Can view 1. Learning input/output mappings as a special case)

Algorithmic

Mathematics of computation tells us what is computable and how. Practical limits. Parallel vs. serial.
Input and output representation, and algorithms for getting from input to output. Programming rules, data structures.

Implementation or hardware

Wetware, hardware.
Neuroscience, neurophysiology and anatomy tell us the adequacies and inadequacies of our modeling assumptions.

Overview of the Brain

Before we look at models of neurons and their interactions, let us get an overview of the brain and nervous  sytem.

Levels of organization, scales

From: Churchland & Sejnowski.  (10,000 A to a micron)

Example: Visual system

From:	Milner, D., & Goodale, M. (1995). The Visual Brain in Action. Oxford: Oxford University Press.

Visual cortex & maps

From:	Churchland, P. S., & Sejnowski, T. J. (1992). The Computational Brain. Cambridge, MA: MIT Press.
Adapted from Van Essen and Anderson 1990. See too, Felleman and Van Essen, 1991.

Cortical layers

Casual inspection shows that the brain has gross structure. What is not immediately apparent  is that structures  do not consist of randomly connected nerve cells. There is a medium-level organization into multiple functional groupings. 
The neocortex has 6 more or less distinguishable layers, there is a microorganization into vertical columns. In the primary visual cortical area (V1), there are ocular dominance and orientation selectivity columns which are believed to form a functional unit called a hypercolumn (1 to 2 mm).

Some brain specs

"cortico-centric" neuroscience
The human brain is:: volume - 1.4 liters, Cortex 2 mm, volume 0.32 liters
Cortex: 1.6 x 10^10 neurons, with about 4000 synapses/neuron, about 6x10^13 connections.

References

Churchland, P. S., & Sejnowski, T. J. (1992). The Computational Brain . Cambridge, MA: MIT Press.
Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cereb Cortex, 1(1), 1-47.
Kandel, E. R., Schwartz, J. H. & Jessell, T. M. (1995). Essentials of Neural Science and Behavior. Norwalk, Connecticut: Appleton & Lang.
Milner, D., & Goodale, M. (1995). The Visual Brain in Action . Oxford: Oxford University Press.
Zeki, S. (1993). A Vision of the Brain. Oxford: Blackwell Scientific Publications.

Links

http://www.med.harvard.edu/AANLIB/home.html
 http://rprcsgi.rprc.washington.edu/~atlas/

Notes on getting started with Mathematica

Numerical Calculations. You can do arithmetic. For example, type 5+7 as shown in the cell below, and then hit the "enter" key. Try other operations, 5^3, 4*3 (note that 4 3,
where a space separates the digits is also interpreted as multiplication). Note that if you
try division, e.g. 2/3, you get the exact answer back. To get a decimal approximation,
type N[2/3].

5+7

N[2/3]

You can go back and select an expression by clicking on the brackets on the far right. These brackets are features of the Macintosh interface and serve to organize text and calculations into a Notebook with outlining features. You can group or ungroup cells for text, graphs, and expressions in various ways to present your calculations. Explore these options under Cell in the menu.  You can see the possible cell types under the Style menu.

Built-in functions. Mathematica has a very large library of built-in functions. They all begin with an uppercase letter. You can get information about a function, e.g. for the exponential of a function, or for plotting graphs:

?Exp

?Plot

If you type two question marks before a function, ??Plot, you'll get more information. Try it. What does the Random function do?