Homework 2

In class, we stated the following theorem.

Theorem (Projection Theorem) Let $H$ be a Hilbert space and $B$ a subspace of $H$. Consider the problem:

$\inf_{m \in B} \|x - m\|$

(i) A necessary and sufficient condition for $m^* \in B$ to be the minimizing element in $B$ so that

$\inf_{m \in B} \|x - m\| = \|x - m^*\|$

is that, $x - m^*$ be orthogonal $B$; that is

$\|x - m^*\| \le \|x - y\|, \qquad \forall y \in B.$

If exists, such an $m^*$ is unique.

(ii) Let $H$ be a Hilbert space and $B$ a closed subspace of $H$. For any vector $x \in H$, there is a unique vector $m^* \in B$ satisfying (1).

Prove the theorem by visiting your class notes or the online notes.

Recall a problem that we solved in class on projection to a subspace. We now modify that problem with the minimization of

$\int_{-1}^{1} (t^n - m(t))^2 dt,$

for some $n \in \mathbb{Z}_+$, over all $m$ such that

$m \in M := \{f : f \in L_2([-1,1]; \mathbb{R}), f(t) = \alpha + \beta t + \gamma t^2; \alpha, \beta, \gamma \in \mathbb{R}\}$

a) State the problem as a Projection problem by identifying the Hilbert space, and the projected subspace.

b) Compute the solution, that is find the minimizing $m$.

Hint: Note here that the projected subspace is different.

Show that the family of complex exponentials in $L_2([0, 2\pi]; \mathbb{C})$:

$\{e_n(t)\} = \left\{\frac{1}{\sqrt{2\pi}} e^{int}, \quad n \in \mathbb{Z}\right\}$

forms an orthonormal sequence. This sequence is used for the Fourier expansion of functions in $L_2([0, 2\pi]; \mathbb{C})$.

Let $\{e_i\}$ be a sequence of orthonormal vectors in a Hilbert space $H$. Let $\{x_n = \sum_{i=1}^{n} \xi_i e_i\}$ be a sequence of vectors in $H$. Show that this sequence converges to a vector $x$ if and only if

$\sum_{i=1}^{\infty} |\xi_i|^2 < \infty.$

Let $H$ be a Hilbert space, and $\{e_i\}$ a complete orthonormal sequence in $H$. That is, the only element in the Hilbert space which is orthogonal to each of the $e_i$ vectors is the null vector.

Show that there is a unique representation for every $h \in H$ in terms of linear expansions involving the sequence $\{e_i\}$.

Show that $l_2(\mathbb{Z}_+; \mathbb{R})$ is separable.

Theorem: Let $\mathcal{M}$ be a closed subspace of a Hilbert space $H$. Let $x$ be a fixed element in $H$ and let $V \subset H$ be a subset such that $V = \{v : x + y, y \in \mathcal{M}\}$ (also called a linear variety of $\mathcal{M}$). Then there is a unique vector $m_0 \in V$ of minimum norm. Furthermore, $m_0$ is orthogonal to $\mathcal{M}$.

The following is an application of this result. Let $x \in \mathbb{R}^n$. Consider the following optimization problem:

$\min\, x^T Q x,$

such that

$Ax = b,$

where $A$ is an $m \times n$ matrix $m \le n$, with rank $m$ and $Q$ a symmetric, positive definite matrix.

Show that, using the Projection Theorem, the optimal $x^*$ minimizing $x^T Q x$ is given by:

$x^* = Q^{-1}A^T(AQ^{-1}A^T)^{-1}b$

Using Matlab (or any other program), generate a (function) code named GramSchmidtProcedure, which takes as input a fixed number of linearly independent vectors and generates a family of orthonormal vectors as a result. Apply your code to the following problem:

Let

$x_1 = \begin{bmatrix} -4 \\ -4 \\ 2 \\ 1 \\ 0 \end{bmatrix}, \quad x_2 = \begin{bmatrix} 0 \\ 1 \\ 1 \\ 3 \\ 0 \end{bmatrix}, \quad x_3 = \begin{bmatrix} 0 \\ 4 \\ 0 \\ 2 \\ 1 \end{bmatrix}, \quad x_4 = \begin{bmatrix} 0 \\ 0 \\ 0 \\ -2 \\ 1 \end{bmatrix}$

Generate a sequence of orthonormal vectors $\{e_1, e_2, e_3, e_4\}$ which span the same space that is spanned by $\{x_1, x_2, x_3, x_4\}$.