 % ----------------------------------------------------------------
% AMS-LaTeX Paper ************************************************
% **** -----------------------------------------------------------
\documentclass[12pt]{amsart}
\usepackage{amssymb,amsmath}
\usepackage{times}
\usepackage{colordvi}
\usepackage[dvips]{graphicx}
%\usepackage[left=2.5cm,top=2cm,right=2.5cm,nohead,nofoot]{geometry}

% ----------------------------------------------------------------
\vfuzz2pt % Don't report over-full v-boxes if over-edge is small
\hfuzz2pt % Don't report over-full h-boxes if over-edge is small
% THEOREMS -------------------------------------------------------
\newtheorem{thm}{Theorem}[section]
\newtheorem{cor}[thm]{Corollary}
\newtheorem{conj}[thm]{Conjecture}
\newtheorem{ques}[thm]{Question}
\newtheorem{prob}[thm]{Problem}
\newtheorem{ex}[thm]{Example}
\newtheorem{lem}[thm]{Lemma}
\newtheorem{prop}[thm]{Proposition}
\newtheorem{defn}[thm]{Definition}
\newtheorem{rem}[thm]{Remark}
\numberwithin{equation}{section}
% MATH -----------------------------------------------------------

%%%%%%%%%%%%%%%%%%%%%%%%%%   Layouts %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\headheight=8pt
\topmargin=0.375truein
\topmargin=-.22truein
\textheight=8.9truein   \textwidth=6.3truein
\oddsidemargin=.1in \evensidemargin=.1in
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\def \i{{\boldsymbol{i}}}
\def \j{{\boldsymbol{j}}}
\def \u{{\boldsymbol{u}}}
\def \v{{\boldsymbol{v}}}
\def \Q { {\mathbb Q} }
\def \R { {\mathbb R} }
\def \C { {\mathbb C} }
\def \Z { {\mathbb Z} }

\pagestyle{plain}


\begin{document}
\title{Notes on Tyler's Amgen project}

%\author{Christopher J. Hillar}%
%\address{Mathematical Research Sciences Institute, 17 Gauss Way, Berkeley, CA 94120}
%\email{chillar@msri.org}

%\author{Friedrich T. Sommer}%
%\address{Redwood Center for Theoretical Neuroscience, Berkeley, CA 94720}
%\email{fsommer@berkeley.edu}


  %\date{}%
%\dedicatory{}%
%\commby{}%
% ----------------------------------------------------------------
\begin{abstract}
Here are some notes on the project so far
\end{abstract}
 \maketitle
% ----------------------------------------------------------------

\section{Basic Concepts}

The Kullback-Leibler Divergence can be seen as a measure of the distance between two probability distributions.  While it is not a true distance because it is not the same regardless of order, this is a good way to look at the measure.  The equation for the KL Divergence is:
\begin{equation}
D_{KL}(p(x) || q(x)) = \int p(x) \log \frac{p(x)}{q(x)}
\end{equation}

Another important quantity is the Fisher Information.  The Fisher Information represents a measure of the importance of a given parameter to a probability distribution.  For a parameter $\theta$, $P(x|\theta)$ represents the likelihood of a random variable $x$ conditioned on $\theta$.  The Fisher Information for this conditional probability distribution is:
\begin{equation}
F_{I} = \bigg\langle \left (\frac{\partial}{\partial\theta} \log P(x|\theta)\right )^{2}\bigg\rangle_{P(x|\theta)}
\end{equation}

For many parameters $\theta_{1},...,\theta_{n}$, a matrix J, known as the Fisher Information Matrix, can be created, such that:
\begin{equation}
J_{ij} = \bigg\langle \frac{\partial}{\partial\theta_{i}} \log P(x|\theta_{i}) \frac{\partial}{\partial\theta_{j}} log P(x|\theta_{j}) \bigg\rangle
\end{equation}

Using an input to a neural network as our functional parameters, we can write $P(x|\bold s)$, where $\bold s = (s_{1},...,s{n})$ is the input signal history and $s_{k}$ is the signal k timesteps in the past.  With this, we can write the Fisher Information Matrix as
\begin{equation}
J_{ij} = \bigg\langle \frac{\partial^{2}}{\partial s_{i}\partial s_{j}} \log P(x|\bold s)\bigg\rangle_{P(x|\bold s)}
\end{equation}

The Fisher Information Matrix represents the rate of change of the KL Divergence, as
\begin{equation}
D_{KL} \approx \delta s^T J \delta s
\end{equation}

For Gaussian systems, both the KL Divergence and the FIM can be written in simple forms:

\begin{equation}
D_{KL} = (\mu^1 - \mu^2)^T C^{-1} (\mu^1 - \mu^2)
\end{equation}
\begin{equation}
J_{ij} = \frac{\partial \mu^T}{\partial s_i}C^{-1}\frac{\partial \mu}{\partial s_j}
\end{equation}

If the mean is linearly dependent on the signal, the above approximation becomes exact.  This will be the case for all of our work.

\section{Memory in Kalman Filters}

Kalman Filter Equations:
\begin{equation}
x_{t} = Vx_{t-1} + \nu, \quad \nu \in \mathcal{N}(0,Q)
\end{equation}
\begin{equation}
y_{t} = Ux_{t} + \mu, \quad \mu \in \mathcal{N}(0,R)
\end{equation}
\begin{equation}
\hat{x}_{t}^{-} = V\hat{x}_{t-1}
\end{equation}
\begin{equation}
P_{t}^{-} = VP_{t-1}V^{T} + Q
\end{equation}
\begin{equation}
K_{t} = P_{t}^{-}U^{T}(UP_{t}^{-}U^{T} + R)^{-1}
\end{equation}
\begin{equation}
\hat{x}_{t} = \hat{x}_{t}^{-} + K_{t}(y_{t}-U\hat{x}_{t}^{-})
\end{equation}
\begin{equation}
P_{t} = (I - K_{t}U)P_t^{-}
\end{equation}

Iterated equations and their derivations:

$x_n$:

\begin{equation}
x_n = V^nx_0 + \sum_{k=0}^{n-1} V^k\nu_{n-k}
\end{equation}

$\hat{x}_n^-$:

\begin{equation*}
\begin{split}
\hat{x}_{n}^{-} = \ & V\hat{x}_{n-1} \\
= \ & V(\hat{x}_{n-1}^{-} + K_{n-1}(y_{n-1} - U\hat{x}_{n-1}^{-})) \\
= \ & (V - VK_{n-1}U)\hat{x}_{n- 1 }^{-} + VK_{n-1}y_{n-1} \\
= \ & (V - VK_{n-1}U)(V - VK_{n-2}U)\hat{x}_{n-2}^{-} + (V-VK_{n-1}U)VK_{n-2}y_{n-2} + VK_{n-1}y_{n-1} \\
= \ &  \left ( \prod_{k=1}^{n-1}(V - VK_{n-k}U) \right )Vx_0 + \sum_{i=1}^{n-1}\left ( \prod_{k=1}^{i-1}(V-VK_{n-k}U)\right )VK_{n-i}y_{n-i}
\end{split}
\end{equation*}


$P_n^-$:

\begin{equation}
P_n^- = \left ( \prod_{k=1}^{n-1}V(I - K_{n-k}U)\right ) P_1^-V^{(n-1)T} + \sum_{i=0}^{n-2}\left (\prod_{k=1}^{i}V(I - K_{n-k}U)\right )QV^{iT}
\end{equation}

$\hat{x}_n$:
\begin{equation*}
\begin{split}
\hat{x}_n = \ & \hat{x}_n^- + K_n(y_n - U\hat{x}_n^-) \\
= \ & V\hat{x}_{n-1} + K_ny_n - K_nUV\hat{x}_{n-1} \\
= \ & (V - K_nUV)\hat{x}_{n-1} + K_ny_n \\
= \ & (V - K_nUV)(V-K_{n-1}UV)\hat{x}_{n-2} + (V-K_nUV)K_{n-1}y_{n-1} + K_ny_n \\
= \ & \left (\prod_{k=0}^{n-1}(V - K_{n-k}UV)\right )x_0 + \sum_{i=0}^{n-1}\left (\prod_{k=0}^{i-1}(V - K_{n-k}UV)\right )K_{n-i}y_{n-i}
\end{split}
\end{equation*}


$P_n$:

\begin{equation}
P_n = \left (\prod_{k=0}^{n-2}(I - K_{n-k}U)V\right )P_1V^{(n-1)T} + \sum_{i=0}^{n-2}\left (\prod_{k=0}^{i}(I - K_{n-k}U)V\right )V^{-1}QV^{iT}
\end{equation}



Expectation and Covariance Calculations:

E($\hat{x}_n^-$):

\begin{equation}
E(\hat{x}_n^-) = \ \left ( \prod_{k=1}^{n-1} (V - VK_{n-k}U) \right ) Vx_0 + \sum_{i=1}^{n-1} \left ( \prod_{k=1}^{i-1}(V - VK_{n-k}U) \right )VK_{n-i}UV^{n-i}x_0
\end{equation}

E($\hat{x}_n$):

\begin{equation}
E(\hat{x}_n) = \ \left (\prod_{k=0}^{n-1}(V - K_{n-k}UV) \right )x_0 + \sum_{i=1}^{n-1} \left (\prod_{k=0}^{i-1}(V - K_{n-k}UV) \right )K_{n-i}UV^{n-i}x_0 + K_nUV^nx_0
\end{equation}

Cov($\hat{x}_n^-$):
\begin{equation}
\begin{split}
Cov(\hat{x}_n^-) = \ & \sum_{i=1}^{n-1}(B_iQB_i^T + A_iRA_i^T)\\
A_i = \ & \left ( \prod_{k=1}^{i-1}(V-VK_{n-k}U)\right )VK_{n-i}\\
B_i = \ & \sum_{j=1}^{i}A_jUV^{i-j}
\end{split}
\end{equation}

Cov($\hat{x}_n$)
\begin{equation}
\begin{split}
Cov(\hat{x}_n)  = \ & \sum_{i=0}^{n-1}(H_iQH_i^T + G_iRG_i^T)\\
G_i = \ & \left ( \prod_{k=0}^{i-1}(V-K_{n-k}UV)\right )K_{n-i}\\
H_i = \ & \sum_{j=0}^{i}G_jUV^{i-j}
\end{split}
\end{equation}

A few interesting things to note are that the expectation depends linearly on $x_n$ and that the covariance will increase with increasing $n$.  This second fact will become more important when looking at the FIM.

Fisher Information Matrices:

\begin{equation}
\begin{split}
J_{rs}^- = \ & \frac{\partial \mu^{-T}}{\partial y^{n-r}}C^{--1}\frac{\partial \mu^-}{\partial y^{n-s}}\\
\frac{\partial \mu^-}{\partial y^{n-r}} = \ & \left ( \prod_{k=1}^{r-1}(V-VK_{n-k}U)\right )VK_{n-r} = A_r\\
J_{rs}^- = \ & A_r^T\left ( \sum_{i=1}^{n-1}(B_iQB_i^T + A_iRA_i^T)\right )^{-1}A_s
\end{split}
\end{equation}

\begin{equation}
\begin{split}
J_{rs} = \ & \frac{\partial \mu^T}{\partial y_{n-r}}C^{-1}\frac{\partial \mu}{\partial y_{n-s}}\\
\frac{\partial \mu}{\partial y^{n-r}} = \ & \left ( \prod_{k=0}^{r-1}(V-K_{n-k}UV)\right )K_{n-r} = G_r\\
J_{rs} = \ & G_r^T\left ( \sum_{i=0}^{n-1}(H_iQH_i^T + G_iRG_i^T)\right )^{-1}G_s
\end{split}
\end{equation}

In the simple 1-D case, all variables $\in \mathbb{R}^{1x1}$, we can compute the decay of $J_{rr}^-$ and $J_{rr}$ as $r\rightarrow n$.
For $J_{rr}^-$,
\begin{equation}
\begin{split}
J_{rr}^- = \ & A_rC^{--1}A_r\\
= \ & \frac{A_r^2}{C^-}\\
= \ & \frac{\left (\left ( \prod_{k=1}^{r-1}(V-VK_{n-k}U)\right )VK_{n-r}\right )^2}{C^-}\\
\approx \ & \frac{((V-V\bold{K}U)^{r-1}\bold{K})^2}{C^-}\\
\approx \ & \alpha^- \cdot (\lambda^-)^{r-1}
\end{split}
\end{equation}

where $\bold{K}$ is the steady state approximation of $K_{n-i}$, $\alpha^- = \frac{V^2\bold{K}^2}{C^-}$ and $\lambda^- = (V-V\bold{K}U)^2$.

Similarly, $J_{rr}$ can be expressed as
\begin{equation}
J_{rr} \approx \alpha \cdot (\lambda)^r
\end{equation}

with $\alpha = \frac{\bold{K}^2}{C}$ and $\lambda = (V-\bold{K}UV)^2$.


Taking the sum of $J_{rr}^-$ from $1$ to $n-1$ yields
\begin{equation}
J_{tot}^- = \sum_{r=1}^{n-1}J_{rr}^- = \frac{\alpha^-}{\lambda^-}\frac{\lambda^- - (\lambda^-)^n}{1-\lambda^-}
\end{equation}

Similarly,
\begin{equation}
J_{tot} = \sum_{r=0}^{n-1}J_{rr} \approx \alpha \frac{1-\lambda^n}{1-\lambda}
\end{equation}




%\begin{figure}[htb]
%  \makebox[\linewidth]{\includegraphics[scale=0.6]{paperfitnessfinal3.eps}}
%  \caption{\label{fig1}
%  Fitness functions for a harmonic oscillator.  The points in blue come from using (\ref{SLfitness}) while those in green are from (\ref{HamiltonianFitness})}
%\end{figure}

\section{Desirable and Undesirable Memory}

The Kalman filter has built in it two types of memory, which we have termed desirable and undesirable.  The desirable memory can be seen as memory of the noise $\nu$ in the internal process, and the undesirable memory can be seen as memory of the noise $\mu$ in the observations.  We propose that the FMC of the desirable memory should decay slowly, and the FMC of the undesirable should decay quickly, perhaps exponentially. I will briefly go over the derivation of the FMC for both types of memory, which will be of the form of equation 1.7.

Firstly, $\hat{x}_n$ should be rewritten.

\begin{equation*}
\begin{split}
\hat{x}_n = \ & \left (\prod_{k=0}^{n-1}(V - K_{n-k}UV)\right )x_0 + \sum_{i=0}^{n-1}\left (\prod_{k=0}^{i-1}(V - K_{n-k}UV)\right )K_{n-i}y_{n-i}\\
= \ & Ax_0 + \sum_{i=0}^{n-1} B_iy_{n-i}\\
= \ & Ax_0 + \sum_{i=0}^{n-1} B_i(Ux_{n-i} + \mu_{n-i})\\
= \ & Ax_0 + \sum_{i=0}^{n-1} B_iU(V^{n-i}x_0 + \sum_{j=0}^{n-i-1} V^j\nu_{n-i-j}) + \sum_{i=0}^{n-1} B_i\mu_{n-i}\\
= \ & (A + \sum_{i=0}^{n-1} B_iUV^{n-i})x_0 + \sum_{i=0}^{n-1} B_iU\sum_{j=0}^{n-i-1} V^j\nu_{n-i-j} + \sum_{i=0}^{n-1} B_i\mu_{n-i}\\
= \ & Cx_0 + \sum_{i=0}^{n-1} \sum_{j=0}^i B_jUV^{i-j}\nu_{n-i} + \sum_{i=0}^{n-1}B_i \mu_{n-i}\\
\end{split}
\end{equation*}

Written out, this equation yields:
\begin{equation}
\begin{split}
\hat{x}_n = \ & \left(\left( \prod_{k=0}^{n-1}(V-K_{n-k}UV)\right) + \sum_{i=0}^{n-1}\left(\prod_{k=0}^{i-1}(V-K_{n-k}UV)\right)K_{n-i}UV^{i-j}\right)x_0 + \\ \ & \sum_{i=0}^{n-1}\sum_{j=0}^i\left( \prod_{k=0}^j(V-K_{n-k}UV)\right)K_{n-j}UV^{i-j}\nu_{n-i} + \sum_{i=0}^{n-1}\left(\prod_{k=0}^{i}(V-K_{n-k}UV)\right)K_{n-i}\mu_{n-i}
\end{split}
\end{equation}

Using this we can calculate our desired quantities.

It will help to specify the variable $B_i$ as
\begin{equation*}
B_i = \left(\prod_{k=0}^{i-1}(V-K_{n-k}UV)\right)K_{n-i}
\end{equation*}


\subsection{Desirable Memory}
$\\$

The expected value of $\hat{x}$ is derived as follows:

\begin{equation*}
\begin{split}
E(\hat{x}_n|\nu) = \ & E\left(Cx_0 + \sum_{i=0}^{n-1} \sum_{j=0}^i B_jUV^{i-j}\nu_{n-i} + \sum_{i=0}^{n-1}B_i \mu_{n-i}\right) \\
= \ & Cx_0 + \sum_{i=0}^{n-1} \sum_{j=0}^i B_jUV^{i-j}\nu_{n-i} + E\left(\sum_{i=0}^{n-1}B_i \mu_{n-i}\right)\\
= \ & Cx_0 + \sum_{i=0}^{n-1} \sum_{j=0}^i B_jUV^{i-j}\nu_{n-i}\\
\end{split}
\end{equation*}

For the FIM we are interested in the derivative of the expected value with respect an individual time point in the signal's history.  Thus,

\begin{equation*}
\frac{\partial E(\hat{x}_n|\nu)}{\partial \nu_{n-r}} = \sum_{j=0}^r B_jUV^{r-j}
\end{equation*}

The covariance of $\hat{x}_n$ is also derived as follows:

\begin{equation*}
\begin{split}
C(\hat{x}_n|\nu) = \ & Cov(\hat{x}_n|\nu) = Cov\left(Cx_0 + \sum_{i=0}^{n-1} \sum_{j=0}^i B_jUV^{i-j}\nu_{n-i} + \sum_{i=0}^{n-1}B_i \mu_{n-i}\right) \\
= \ & Cov(Cx_0) + Cov\left(\sum_{i=0}^{n-1} \sum_{j=0}^i B_jUV^{i-j}\nu_{n-i}\right) + Cov\left(\sum_{i=0}^{n-1}B_i \mu_{n-i}\right) \\
= \ & Cov\left(\sum_{i=0}^{n-1}B_i \mu_{n-i}\right) \\
= \ & \sum_{i=0}^{n-1}Cov(B_i \mu_{n-i}) \\
= \ & \sum_{i=0}^{n-1}B_i R B_i^T
\end{split}
\end{equation*}

Using the derivative of the expectation and the covariance calculations, we can write

\begin{equation}
J_{rs}(\hat{x}_n|\nu) = \left(\sum_{j=0}^r B_jUV^{r-j}\right)^T \left(\sum_{i=0}^{n-1}B_i R B_i^T\right)^{-1} \left(\sum_{j=0}^s B_jUV^{s-j}\right)
\end{equation}

\subsection{Undesirable Memory}
$\\$

The expected value is calculated similar to above, and yields

\begin{equation*}
E(\hat{x}_n|\mu) = Cx_0 + \sum_{i=0}^{n-1} B_i \mu_{n-i}
\end{equation*}

We are again interested in the derivative of this equation with respect to a previous $\mu$ term.

\begin{equation*}
\frac{\partial E(\hat{x}_n|\mu)}{\partial \mu_{n-r}} = B_r
\end{equation*}

The covariance can be derived in a similar fashion as before and shown to be

\begin{equation*}
C(\hat{x}_n|\mu) = \sum_{i=0}^{n-1}\left(\sum_{j=0}^i B_jUV^{i-j}\right) Q \left(\sum_{j=0}^i B_jUV^{i-j}\right)^T
\end{equation*}

Putting these equations together, we get an expression for the FIM.

\begin{equation}
J_{rs}(\hat{x}_n|\mu) = B_r^T \left(\sum_{i=0}^{n-1}\left(\sum_{j=0}^i B_jUV^{i-j}\right) Q \left(\sum_{j=0}^i B_jUV^{i-j}\right)^T\right)^{-1} B_r
\end{equation}


\section{Connection to Surya's Paper}

Surya's update equation was written as $x_t = Wx_{t-1} + vs_t + z_t$. We can make a library translating our variables from the Kalman filter to match Surya's equation. In our case, the update equation is $\hat{x}_t = (V-K_tUV)\hat{x}_{t-1} + K_tUx_t + K_t \mu_t$.

Thus, in our case, the network state is the updated prediction, and the variables can be related as follows:

$\\x_t \rightarrow \hat{x}_t \\$
$W \rightarrow (V-K_tUV) \\$
$v \rightarrow K_tU \\$
$s_t \rightarrow x_t \\$
$z_t \rightarrow K_t \mu_t \\$

A notable difference between the two equations is, however, that $W$ now becomes the time dependent $W_t$. The same occurs with $v$, becoming $v_t$.  Since the time dependence arises from the time dependence of the Kalman gain, we can approximate the Kalman gain with the steady-state Kalman gain to simplify the problem and allow for a more direct translation.

Using this simplification,the covariance of $\hat{x}_t$ can be calculated directly from Surya's equation for the covariance of $x_t$, namely

\begin{equation}
C_n = \epsilon \sum_{k=0}^\infty W^kW^{kT}.
\end{equation}

Plugging in for W and substituting $K cov(\mu_t) K^T$ for the covariance of $z_t$, we get

\begin{equation}
cov(\hat{x}_n) = \sum_{k = 0}^{n-1}(V-KUV)KRK^T(V-KUV)^T.
\end{equation}

Using equation (4) from Surya's supplemental paper, we can now calculate J(k).

\begin{equation}
J(k) = (KU)^T(V-KUV)^{kT}\left ( \sum_{j=0}^{n-1}(V-KUV)KRK^T(V-KUV)^T \right )^{-1} (V-KUV)(KU)
\end{equation}


\section{Stuff For Poster}

$P(\hat{x}_n|y+\delta y)\\$

$P(\hat{x}_n|y)\\$

$P(x_n)\\$

$x_n^1$     $x_n^2$     $\hat{x}_n^1$   $\hat{x}_n^2\\\\$

$D_{KL}(P(\hat{x}_n|y)||P(\hat{x}_n|y+\delta y)) = \frac{1}{2}\delta y^TJ\delta y$

\begin{equation}
J_{rs} = \left \langle \frac{\partial^2}{\partial y_{n-r}\partial y_{n-s}}\log P(\hat{x}_n|y) \right \rangle
\end{equation}
\begin{equation}
J_{rs} = \frac{\partial \mu^T}{\partial y_{n-r}}C^{-1}\frac{\partial \mu}{\partial y_{n-s}}
\end{equation}
\begin{equation}
J_{rs} = G_r^T\left ( \sum_{i=0}^{n-1}(H_iQH_i^T + G_iRG_i^T) \right )^{-1}G_s
\end{equation}

\begin{equation}
G_i = \left ( \prod_{k=0}^{i-1}(V-K_{n-k}UV)\right )K_{n-i}
\end{equation}

\begin{equation}
H_i = \sum_{j=0}^{i}G_iUV^{i-j}
\end{equation}

$x_{t+1}$   $Vx_t + \nu$    $Ux_{t-1} + \mu$    $Ux_{t+1} + \mu$

$\hat{x}_{t-1}^-$   $\hat{x}_{t+1}^-$   $y_{t-1}$   $y_{t+1}$

$\hat{x}_{t+1}$     $\hat{x}_{t-1}^{-}+K_{t-1}(y_{t-1}-U\hat{x}_{t-1}^{-})$

$\hat{x}_{t+1}^{-} + K_{t+1}(y_{t+1}-U\hat{x}_{t+1}^{-})\\$


In the standard Kalman filter, $x_t$ is a hidden variable that is of interest. In the discrete time case, an observation $y_t$ is generated at each timestep. The Kalman filter is designed such that the variable $\hat{x}_t$ provides the optimal estimate for $x_t$. $\hat{x}_t^-$ represents the predicted value of $x_t$ prior to the observation at time $t$. $\hat{x}_t$ is then generated by adjusting the predicted value by a weighted average of $y_t$ and $\hat{x}_t^-$. Since $y_t$ is corrupted by noise, this weighted average helps to denoise the signal to provide an accurate estimate for $x_t$.

The ability to accurately estimate $x_t$ is affected by $y$. An infinitesimally small perturbation in $y$ shifts $\hat{x}_t$ by an amount directly proportional to $J$,the FIM.



where $\mu$ is the expected value of $\hat{x}_n$ and $C$ is the covariance of $\hat{x}_n$. Following a good deal of manipulations, this can be expressed in terms of the Kalman filter parameters as







% ----------------------------------------------------------------
\begin{thebibliography}{1}


\bibitem{SL}
M. Schmidt, H. Lipson, \textit{Distilling free-form natural laws from experimental data}, Science, \textbf{324} (2009), 81--85.

\bibitem{SL2}
M. Schmidt, H. Lipson, \textit{Distilling free-form natural laws from experimental data}, Supplementary online materials.

\end{thebibliography}

\end{document}
% ----------------------------------------------------------------





