Extrap: TEA as an overdetermined quasi-Newton

Research/Extrapolation methods/notes.tex

@@ -357,6 +357,46 @@ Let $\vec{r}(\vec{x}_i) = \vec{x}_{i+1} - \vec{x}_i$ be the residual function.
 Then MPE may be expressed as equation (\ref{eq:uqn}) with $\fxi{i} = \vec{r}(\vec{x}_i)$ and $\vec{v}_i = \vec{r}(\vec{x}_{n+i-1})$.
 RRE and MMPE may also be expressed as such, using $\vec{v}_i = \vec{r}(\vec{x}_{n+i}) - \vec{r}(\vec{x}_{n+i-1})$ and $\vec{v}_i$ some fixed vector independent of $\vec{r}(\vec{x})$, respectively.
 
+TEA also has a connection to quasi-Newton methods, though in a more roundabout way.
+Recall the equation that forms the foundation of the quasi-Newton methods expressed in this section:
+\begin{equation}
+F_k = \hat{J} X_k.
+\end{equation}
+This equation assumes that our starting position is $\vec{x}_n$.
+If our starting position is instead $\vec{x}_{n+j}$ then the equation becomes
+\begin{equation}
+F_{n+j,k} = \hat{J}_{n+j} X_{n+j,k}.
+\end{equation}
+We can create any number of these equations as long as we have enough values of $\fxi{n+j}$ and $\vec{x}_{n+j}$.
+
+Using all of these equations to determine a direction would create an overdetermined system.
+One way to reduce the size of such a system is to take the inner product with some vector $\vec{q}$:
+\begin{equation*}
+\vec{q}^\top F_{n+j,k} = \vec{q}^\top \hat{J}_{n+j} X_{n+j,k}.
+\end{equation*}
+This forms one row of a larger linear equation.
+
+Recall that we sought $\vec{u}_j$ such that $F_{n+j,k} \vec{u}_j = \fxi{n+j}$, which led to the Newton step $\vec{x}_{n+j} - X_{n+j,k} \vec{u}_j$.
+If we assume that $\vec{u}_j = \vec{u}$ for all $j$ then we can write the quasi-Newton method as
+\begin{align*}
+\begin{bmatrix} \vec{q}^\top F_{n,k} \\ \vdots \\ \vec{q}^\top F_{n+k-1,k} \end{bmatrix} \vec{u} = & \begin{bmatrix} \vec{q}^\top \fxi{n} \\ \vdots \\ \vec{q}^\top \fxi{n+k-1} \end{bmatrix}, \\
+\vec{\hat{x}}_{n+1} = & \vec{x}_n - X_{n,k} \vec{u}.
+\end{align*}
+This gives as its solution
+\begin{equation}
+\vec{\hat{x}}_{n+1} = \frac{ \vmat{
+\vec{x}_n & \dots & \vec{x}_{n+k} \\
+\vec{q}^\top \fxi{n} & \dots & \vec{q}^\top \fxi{n+k} \\
+\vdots & & \vdots \\
+\vec{q}^\top \fxi{n+k-1} & \dots & \vec{q}^\top \fxi{n+2k-1}
+}}{ \vmat{
+1 & \dots & 1 \\
+\vec{q}^\top \fxi{n} & \dots & \vec{q}^\top \fxi{n+k} \\
+\vdots & & \vdots \\
+\vec{q}^\top \fxi{n+k-1} & \dots & \vec{q}^\top \fxi{n+2k-1}}}
+\end{equation}
+which is exactly TEA when replacing $\fxi{n+j}$ with $\vec{x}_{n+j+1} - \vec{x}_{n+j} = \vec{r}(\vec{x}_{n+j})$.
+
 \section{Generalized Shanks Transformation}
 
 Suppose that we have a sequence $\set{x_n}$ that tends towards $s$ as