\documentclass[12pt]{article} \setlength{\oddsidemargin}{0.2in} \setlength{\topmargin}{-0.5in} \setlength{\headheight}{0.0in} \setlength{\headsep}{0.5in} \setlength{\textwidth}{6.0in} \setlength{\textheight}{9.0in} \title{Towards a theoretical framework for the thalamocortical system} \author{Bruno A. Olshausen\\ Redwood Neuroscience Institute} \begin{document} \maketitle \begin{abstract} This memo is an attempt to outline what I believe are, or at least what should be, our research goals at RNI. It is motivated in part by our many discussions over the past six months on how we should go about modeling the brain. I will describe here what I believe to be the key questions and issues, as well as the relevant disciplines. My hope is that we can develop some consensus on these issues and that this will serve as a starting point for discussion and perhaps eventually a viewpoint paper co-authored by the group that we can post on our website. \end{abstract} \section{Our mission} Our principal goal at RNI is to develop a theoretical framework for the thalamocortical system, at the level of perception and cognition. We say this unabashedly, because we believe that there now exists sufficient data, in addition to mathematical/computational formalisms, to begin assembling the many bits and pieces together into a coherent theory. This does not mean we will have all the answers, but it should go a long way towards unifying existing data and models, and also in providing a framework for motivating and designing future experiments. Having a solid theoretical framework is important for any field of scientific endeavor. Einstein said that... Theories provide the big picture. They determine what questions we ask, how we interpret the data, and what makes an experiment interesting. However, most of systems and cognitive neuroscience has been drifting along over the past two decades without any agreed upon theoretical framework. One often hears questions posed of the form, ``how does A affect B?'', ``what is the role of X in Y?'', or ``where in the cortex does function X reside?'' To some extent this approach has been appropriate, as sometimes---when you don't know anything starting out---you have to just gather enough data before you can see a trend or find something interesting. But I think it is safe to say that this approach has now reached its asymptote in terms of what is being gained from experimental neuroscience. Our understanding of basic perceptual and cognitive functions at the neural level is only slightly more advanced today than it was 20 years ago, despite billions of dollars of funding and the dedicated efforts of tens of thousands of highly educated scientists. Thus, developing a theoretical framework for neocortical function is not a luxury or some kind of optional philosophical endeavor. It has now gotten to the point where the absence of theory is the single-most factor, besides perhaps new technologies for recording, limiting progress. Our mission at RNI is to fill this void. \section{The big picture} To get started, it is helpful to provide at least a coarse-grained picture of what we think the brain is doing, and how to break it down into elementary parts. I have made one such attempt in figure xxx. *** overall function diagram *** Here I have depicted an overall function diagram of the brain, with sensors at one end actuators at the other end. The simplest sort of nervous system would be one that connected sensors directly to actuators, to bring about reflexive actions. An example would be a simple light-seeking organism. All mammals have reflexes mediated by a single-synapse---e.g., the quick motor reflexes mediated through the dorsal root ganglion in the spinal cord. At any given moment, we are aware of only a limited amount of information impinging upon our sensory receptors. Presumably this is because we have limited processing resources at higher levels, and so a selection mechanism (attention) is needed to serialize the computationally demanding aspects of processing sensory information. At the motor end of things there is also an important element of selection---e.g., you can't move your arm both up and down at the same time, you have to choose (it's like winner-take-all). Between the sensors and actuators and their respective selection systems lies all the interesting stuff. I have dubbed this the ``world model,'' and it includes the internal representations about the world that are inferred from the sensory inputs, and our predictions about what is going to happen next and the consequences of our actions. Importantly, this can not be done in the language of sensors and actuators. That is why we need to develop high-level internal representations of the environment - so we can reason about what is going on and what is going to happen in a fairly abstract way. More on this in the section below. In addition to the world model, I have included plans, goals, and motivations. I'm not convinced we need to talk about consciousness (yet) in order to understand the workings of this system. Our goal is to attack the box called ``world model,'' which believe is largely taking place in the neocortex and its interactions with the thalamus. \section{Why the thalamocortical system?} We have zeroed in on the neocortex, in addition to its connections with thalamus, for several reasons: \begin{itemize} \item Uniform architecture \item Has exploded in growth in recent history, along with increased intelligent capacities. \item Much worked out about sensory coding, motor representations, etc. \end{itemize} This should not be seen as lowering in importance the role of other brain regions. For example, the basal ganglia are known to be important for selecting actions, and also even for certain aspects of cognition. One idea is that they play a role in ``selecting thoughts'' as well. It would be a mistake to be too thalamocortical-centric, and one should be on the lookout for other regions that are central to explaining cognitive and perceptual abilities. But for now we focus, for sake of keeping us sane, on the thalamocortical system. \section{What kind of stuff do we need to know?} Perhaps the single biggest reason why no one has yet attempted to formulate a theory of the brain is that it is a highly interdisciplinary endeavor. It draws upon expertise in at least three core areas: psychology, neuroscience, and mathematics/engineering (figure xxx). I will attempt to summarize here some of the most relevant concepts and types of data from each of these disciplines. \subsection{Psychology} Psychology is essentially the study of overt behavior, as well as our own introspections. As such, it provides us with a functional description of the system. It tells us what the system can and cannot do, and how well it does it. Sometimes psychological investigations can even suggest certain things about the neural mechanisms at work. Perceptual psychology has a long history of doing this, going back to the 1800's (e.g., Fechner, Weber, Helmholtz). One of its early successes was color vision. The trichromacy theory of perception was formulated, tested, and shown to be consistent with psychophysical data long before the discovery of cones in the retina. In a similar vein, Nakayama and Shimojo's recent experiments on surface perception provide compelling evidence that 3D surface layout, figure/ground assignment, etc. are fundamental aspects of intermediate-level representation in the visual cortex. However, there are not yet any specific theories for how this is neurally implemented. Cognitive psychology has been more of a latecomer since the 1950-60's (Neisser, Miller, Sperling, Chomsky). theories of language, memory, attention, and attempts to test these theories. reasoning about internal representations. didn't have fMRI scanners at their disposal. field is essentially now dead. part of our mission is to revive it in a more mathematically/computationally informed manner. We do not experience the retinal image, but rather our interpretation of the image. That is, our visual experience of the world is largely {\em inferred} from photoreceptors. It cannot be deduced from them because there is insufficient information to provide a unique interpretation. The same goes for audition and touch. Perception, cognition, ``memory'' Prediction: MacKay - epistimology, Plato analysis by synthesis invariance, analogy, language \subsection{Neuroscience} Neuroscience is the study of internal mechanisms that lead to overt behavior. It concerns itself with the cellular mechanisms of signalling and information transmission - action potentials and synapses - as well as neuroanatomy, which tells us about the architecture of the system. Knowledge of certain aspects of neuroanatomy and physiology can constrain our theories about how the system works, and some of them---such as topographic organization and feedback loops---may be highly suggestive. However, no amount of data at this level alone can tell us about what the system is actually {\em doing}. For that we need psychology, as well as the proper mathematical/computational framework for connecting neural mechanisms with behavior. Note for example that most experiments in systems and cognitive neuroscience are motivated by findings and theories from psychology, not the other way around. Cortical architecture Hierarchical representation Recurrent circuitry, feedback loops diagram \subsection{Mathematics and engineering} Mathematics and engineering provide the theoretical foundations for understanding complex systems such as the brain that are composed of many interacting elements. Mathematics provides a formal reasoning system for dealing with quantitative information and relationships among variables. Without it, we are reduced to reasoning with purely verbal communication, which has limited descriptive power and can lead to confusion when there are many interacting quantities keep track of. Engineering draws upon mathematics in order to design and build practical systems that work in the real world. Signal processing, circuit theory, filtering, dealing with noise, etc. In the case of the brain, evolution has essentially ``engineered'' the brain, and so we draw upon the tools of engineering to essentially ``reverse engineer'' the brain. Concepts from physics also play an important role in brain theory. Energy functions, Hamiltonians, and statistical mechanics are powerful tools for understanding the dynamics of systems composed of many interacting elements. In the late 1950's and early 60's, folks such as Frank Rosenblatt and others began tinkering with simple neural network models, in an attempt to elucidate how learning and computation could be performed by neural circuits in the brain. This movement was revived in the mid-1980's, but quickly split into two different camps: engineers and computer scientists attempting to solve classification problems (e.g., the NIPS crowd), and neuroscientists attempting to build biologically largely driven by engineers attempting solve classification problems rather than modeling the brain. An important branch of mathematics relevant to modeling aspects of perception and cognition is probability theory. This is because the problems of inference, prediction, associative memory, and the like all involve the accumulation and storage and manipulation of joint probabilities. Probabilistic models Bayesian inference Latent variable models, Graphical models Objective functions, optimization \section{Where do we go from here?} multi-disciplinary, No one field suffices, we need to learn alot Attack on all three fronts. Our team: Bruno, Tony, Fritz, Pentti + postdocs \end{document}