Question

Find a formula for the least squares solution of ax=b when the columns of a are orthonormal1 .

Answer 1

Let $\mathbf A$ be a rectangular $m\times n$ matrix with column vectors $\mathbf a_1,\ldots,\mathbf a_n$, i.e.

$\mathbf A=\begin{bmatrix}\mathbf a_1&\cdots&\mathbf a_n\end{bmatrix}$

Then we have

$\mathbf A^\top=\begin{bmatrix}\mathbf a_1&\cdots&\mathbf a_n\end{bmatrix}^\top$

and the product of the two is

$\mathbf A^\top\mathbf A=\begin{bmatrix}\mathbf a_1\cdot\mathbf a_1&\mathbf a_1\cdot\mathbf a_2&\cdots&\mathbf a_1\cdot\mathbf a_n\\\mathbf a_2\cdot\mathbf a_1&\mathbf a_2\cdot\mathbf a_2&\cdots&\mathbf a_2\cdot\mathbf a_n\\\vdots&\vdots&\ddots&\vdots\\\mathbf a_n\cdot\mathbf a_1&\mathbf a_n\cdot\mathbf a_2&\cdots&\mathbf a_n\cdot\mathbf a_n\end{bmatrix}$

Because the columns of $\mathbf A$ are orthonormal, we have

$\mathbf a_i\cdot\mathbf a_j=\begin{cases}1&\text{for }i=j\\0&\text{for }i\neq j\end{cases}$

which means $\mathbf A^\top\mathbf A$ reduces to an $n\times n$ matrix with ones along the diagonal and zero everywhere else, i.e.

$\mathbf A^\top\mathbf A=\begin{bmatrix}1&0&\cdots&0\\0&1&\cdots&0\\\vdots&\vdots&\ddots&\vdots\\0&0&\cdots&1\end{bmatrix}=\mathbf I_n$

where $\mathbf I$ denotes the identity matrix. This means the solution to $\mathbf{Ax}=\mathbf b$ is given by

$\mathbf A^\top(\mathbf{Ax})=\mathbf A^\top\mathbf b\implies(\underbrace{\mathbf A^\top\mathbf A}_{\mathbf I})\mathbf x=\mathbf A^\top\mathbf b\implies\mathbf x=\mathbf A^\top\mathbf b$