\documentclass[12pt]{article}
\setlength{\oddsidemargin}{0.2in}
\setlength{\topmargin}{-0.5in}
\setlength{\headheight}{0.0in}
\setlength{\headsep}{0.5in}
\setlength{\textwidth}{6.0in}
\setlength{\textheight}{9.0in}
\title{Towards a theoretical framework for the thalamocortical system}
\author{Bruno A. Olshausen\\
Redwood Neuroscience Institute}
\begin{document}

\maketitle

\begin{abstract}
This memo is an attempt to outline what I believe are, or at least
what should be, our research goals at RNI.  It is motivated in part by
our many discussions over the past six months on how we should go
about modeling the brain.  I will describe here what I believe to be
the key questions and issues, as well as the relevant disciplines.  My
hope is that we can develop some consensus on these issues and that
this will serve as a starting point for discussion and perhaps
eventually a viewpoint paper co-authored by the group that we can post
on our website.
\end{abstract}

\section{Our mission}

Our principal goal at RNI is to develop a theoretical framework for
the thalamocortical system, at the level of perception and cognition.
We say this unabashedly, because we believe that there now exists
sufficient data, in addition to mathematical/computational formalisms,
to begin assembling the many bits and pieces together into a coherent
theory.  This does not mean we will have all the answers, but it
should go a long way towards unifying existing data and models, and
also in providing a framework for motivating and designing future
experiments.

Having a solid theoretical framework is important for any field of
scientific endeavor.  Einstein said that...   

Theories provide the big picture.  They determine what questions we
ask, how we interpret the data, and what makes an experiment
interesting.  However, most of systems and cognitive neuroscience has
been drifting along over the past two decades without any agreed upon
theoretical framework.  One often hears questions posed of the form,
``how does A affect B?'', ``what is the role of X in Y?'', or ``where
in the cortex does function X reside?''  To some extent this approach
has been appropriate, as sometimes---when you don't know anything
starting out---you have to just gather enough data before you can see
a trend or find something interesting.  But I think it is safe to say
that this approach has now reached its asymptote in terms of what is
being gained from experimental neuroscience.  Our understanding of
basic perceptual and cognitive functions at the neural level is only
slightly more advanced today than it was 20 years ago, despite
billions of dollars of funding and the dedicated efforts of tens of
thousands of highly educated scientists.

Thus, developing a theoretical framework for neocortical function is
not a luxury or some kind of optional philosophical endeavor.  It has
now gotten to the point where the absence of theory is the single-most
factor, besides perhaps new technologies for recording, limiting
progress.  Our mission at RNI is to fill this void.


\section{The big picture}

To get started, it is helpful to provide at least a coarse-grained
picture of what we think the brain is doing, and how to break it down
into elementary parts.  I have made one such attempt in figure xxx.

                *** overall function diagram *** 

Here I have depicted an overall function diagram of the brain, with
sensors at one end actuators at the other end.  The simplest sort of
nervous system would be one that connected sensors directly to
actuators, to bring about reflexive actions.  An example would be a
simple light-seeking organism.  All mammals have reflexes mediated by
a single-synapse---e.g., the quick motor reflexes mediated through the
dorsal root ganglion in the spinal cord.  

At any given moment, we are aware of only a limited amount of
information impinging upon our sensory receptors.  Presumably this is
because we have limited processing resources at higher levels, and so
a selection mechanism (attention) is needed to serialize the
computationally demanding aspects of processing sensory information.
At the motor end of things there is also an important element of
selection---e.g., you can't move your arm both up and down at the same
time, you have to choose (it's like winner-take-all).

Between the sensors and actuators and their respective selection
systems lies all the interesting stuff.  I have dubbed this the
``world model,'' and it includes the internal representations about
the world that are inferred from the sensory inputs, and our
predictions about what is going to happen next and the consequences of
our actions.  Importantly, this can not be done in the language of
sensors and actuators.  That is why we need to develop high-level
internal representations of the environment - so we can reason about
what is going on and what is going to happen in a fairly abstract way.
More on this in the section below.

In addition to the world model, I have included plans, goals, and
motivations.  I'm not convinced we need to talk about consciousness
(yet) in order to understand the workings of this system.

Our goal is to attack the box called ``world model,'' which believe is
largely taking place in the neocortex and its interactions with the
thalamus.


\section{Why the thalamocortical system?}

We have zeroed in on the neocortex, in addition to its connections
with thalamus, for several reasons:
\begin{itemize}
\item Uniform architecture

\item Has exploded in growth in recent history, along with increased
intelligent capacities.

\item Much worked out about sensory coding, motor representations, etc.
\end{itemize}

This should not be seen as lowering in importance the role of other
brain regions.  For example, the basal ganglia are known to be
important for selecting actions, and also even for certain aspects of
cognition.  One idea is that they play a role in ``selecting
thoughts'' as well.  It would be a mistake to be too
thalamocortical-centric, and one should be on the lookout for other
regions that are central to explaining cognitive and perceptual
abilities.  But for now we focus, for sake of keeping us sane, on the
thalamocortical system.


\section{What kind of stuff do we need to know?}

Perhaps the single biggest reason why no one has yet attempted to
formulate a theory of the brain is that it is a highly
interdisciplinary endeavor.  It draws upon expertise in at least three
core areas: psychology, neuroscience, and mathematics/engineering
(figure xxx).  I will attempt to summarize here some of the most
relevant concepts and types of data from each of these disciplines.

\subsection{Psychology}

Psychology is essentially the study of overt behavior, as well as our
own introspections. As such, it provides us with a functional
description of the system.  It tells us what the system can and cannot
do, and how well it does it.  Sometimes psychological investigations
can even suggest certain things about the neural mechanisms at work.
Perceptual psychology has a long history of doing this, going back to
the 1800's (e.g., Fechner, Weber, Helmholtz).  One of its early
successes was color vision.  The trichromacy theory of perception was
formulated, tested, and shown to be consistent with psychophysical
data long before the discovery of cones in the retina.  In a similar
vein, Nakayama and Shimojo's recent experiments on surface perception
provide compelling evidence that 3D surface layout, figure/ground
assignment, etc. are fundamental aspects of intermediate-level
representation in the visual cortex.  However, there are not yet any
specific theories for how this is neurally implemented.  

Cognitive psychology has been more of a latecomer since the 1950-60's
(Neisser, Miller, Sperling, Chomsky).  theories of language, memory,
attention, and attempts to test these theories.  reasoning about
internal representations.  didn't have fMRI scanners at their
disposal.  field is essentially now dead.  part of our mission is to
revive it in a more mathematically/computationally informed manner.

We do not experience the retinal image, but rather our interpretation
of the image.  That is, our visual experience of the world is largely
{\em inferred} from photoreceptors.  It cannot be deduced from them
because there is insufficient information to provide a unique
interpretation.  The same goes for audition and touch.  

Perception, cognition, ``memory''

Prediction:  MacKay - epistimology,  Plato

analysis by synthesis

invariance, analogy, language


\subsection{Neuroscience}

Neuroscience is the study of internal mechanisms that lead to overt
behavior.  It concerns itself with the cellular mechanisms of
signalling and information transmission - action potentials and
synapses - as well as neuroanatomy, which tells us about the
architecture of the system.  Knowledge of certain aspects of
neuroanatomy and physiology can constrain our theories about how the
system works, and some of them---such as topographic organization and
feedback loops---may be highly suggestive.  However, no amount of data
at this level alone can tell us about what the system is actually {\em
doing}.  For that we need psychology, as well as the proper
mathematical/computational framework for connecting neural mechanisms
with behavior.  Note for example that most experiments in systems and
cognitive neuroscience are motivated by findings and theories from
psychology, not the other way around.

Cortical architecture

Hierarchical representation

Recurrent circuitry, feedback loops

diagram


\subsection{Mathematics and engineering}

Mathematics and engineering provide the theoretical foundations for
understanding complex systems such as the brain that are composed of
many interacting elements.  Mathematics provides a formal reasoning
system for dealing with quantitative information and relationships
among variables.  Without it, we are reduced to reasoning with purely
verbal communication, which has limited descriptive power and can lead
to confusion when there are many interacting quantities keep track of.

Engineering draws upon mathematics in order to design and build
practical systems that work in the real world.  Signal processing,
circuit theory, filtering, dealing with noise, etc.  In the case of
the brain, evolution has essentially ``engineered'' the brain, and so
we draw upon the tools of engineering to essentially ``reverse
engineer'' the brain.  

Concepts from physics also play an important role in brain theory.
Energy functions, Hamiltonians, and statistical mechanics are powerful
tools for understanding the dynamics of systems composed of many
interacting elements.

In the late 1950's and early 60's, folks such as Frank Rosenblatt and
others began tinkering with simple neural network models, in an
attempt to elucidate how learning and computation could be performed
by neural circuits in the brain.  This movement was revived in the
mid-1980's, but quickly split into two different camps:  engineers
and computer scientists attempting to solve classification problems
(e.g., the NIPS crowd), and neuroscientists attempting to build
biologically 

 largely driven by engineers attempting solve
classification problems rather than modeling the brain.


An important branch of mathematics relevant to modeling aspects of
perception and cognition is probability theory.  This is because the
problems of inference, prediction, associative memory, and the like
all involve the accumulation and storage and manipulation of joint
probabilities.

Probabilistic models

Bayesian inference

Latent variable models, Graphical models

Objective functions, optimization



\section{Where do we go from here?}

multi-disciplinary, No one field suffices,  we need to learn alot

Attack on all three fronts.

Our team: Bruno, Tony, Fritz, Pentti + postdocs

\end{document}
